Proximal Point Imitation Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
This work develops new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning (IL) with linear function approximation withoutrestrictive coherence assumptions.
We begin with the minimax formulation of theproblem and then outline how to leverage classical tools from optimization, inparticular, the proximal-point method (PPM) and dual smoothing, for online andoffline IL, respectively.
Thanks to PPM, we avoid nested policy evaluation and costupdates for online IL appearing in the prior literature.
In particular, we do awaywith the conventional alternating updates by the optimization of a single convexand smooth objective over both cost and Q -functions.
When solved inexactly, werelate the optimization errors to the suboptimality of the recovered policy.
As anadded bonus, by re-interpreting PPM as dual smoothing with the expert policy as acenter point, we also obtain an offline IL algorithm enjoying theoretical guaranteesin terms of required expert trajectories.
Finally, we achieve convincing empiricalperformance for both linear and neural network function approximation.
Introduction
This work is concerned with the prototypical setting of imitation learning (IL) where
An expert provides demonstrations of state-action pairs in an environment.
The expert could
be optimal or suboptimal with respect to an unknown cost/reward function.
The learner chooses distance measure between its policy to be learned and the expert
empirical distribution estimated from demonstrations.
The learner employs an algorithm, which additionally may or may not use interactions withthe environment, to minimize the chosen distance.
In IL, the central goal of the learner is to recover a policy competitive with expert with respectto the underlying unknown cost function.
IL is important for several real world applications likedriving [1], robotics [2], and economics/finance [3] at the expense of following resources: (R1)expert demonstrations, (R2) (optional) interactions with the environment where the expert collectedthe demonstrations, and (R3) computational resources for solving the problem template.
Interestingly, while there is a vast amount of literature using optimization ideas on the IL problemtemplate, i.e. Lagrangian duality [4–8], resource guarantees are still widely missing since theoptimization literature focuses on the resource (R3) where IL literature mainly focuses on the firsttwo resources (R1) and (R2).
Our work leverages deeper connections between optimization tools and
IL by showing how classical optimization tools can be applied in a linear programming formulationof IL problem guaranteeing efficiency in all (R1), (R2), (R3).
Our contributions: This work aims at designing an algorithm enjoying both theoretical guaranteesand convincing empirical performance. Our methodology is rooted in classical optimization toolsand the LP approach to MDPs.
More precisely, the method uses the recently repopularized overparameterization technique to obtain the Q-function as a Lagrangian multiplier [9, 10] and solves theassociated program using a PPM update with appropriately chosen Bregman divergences. This resultsto an actor-critic algorithm, with the key feature that the policy evaluation step involves optimizationof a single concave and smooth objective over both cost and Q -functions. In this way, we avoidinstability or poor convergence due to adversarial training [4, 11–13], and can also recover an explicitcost along with Q-function.
We further account for potential optimization errors, presenting an errorpropagation analysis that leads to rigorous guarantees for both online and offline setting.
For thecontext of linear MDPs [10, 14–19], we provide explicit convergence rates and error bounds for thesuboptimality of the learned policy, under mild assumptions, significantly weaker than those found inthe literature until now. To our knowledge, such guarantees in this setting are provided for the firsttime. Finally, we demonstrate that our approach achieves convincing empirical performance for bothlinear and neural network function approximation.
Related Literature.
The first algorithm addressing the imitation learning problem is behavioralcloning [20].
Due to the covariate shift problem [21, 22], it has low efficiency in terms of experttrajectories (R1).
To address this issue, [23–32] proposed to cast the problem as inverse reinforcementlearning (IRL).
IRL improves the efficiency in terms of expert trajectories, at the cost of introducingthe need of running reinforcement learning (RL) repetitively, which can be prohibitive in termsof environment samples (R2) and computation (R3).
A successive line of work started with [33]highlights that repeated calls to an RL routine can be avoided.
This work inspired generativeadversarial imitation learning (GAIL) [4] and other follow-up works [5–8] that leveraged optimizationtools like primal-dual algorithms but did not try to deepen the optimization connections to deriveefficiency guarantees in terms of all (R1),(R2),(R3).
Finally, a recent line of work [34, 35] in ILbypasses the need of optimizing over cost functions and thus avoids instability due to adversarialtraining.
Although these algorithms achieve impressive empirical performance in challenging highdimensional benchmark tasks, they are hampered by limited theoretical understanding. This is thefundamental difference from our work, which enjoys both favorable practical performance and strongtheoretical guarantees.
Existing model-free IL theoretical papers with global convergence guarantees assume either afinite horizon episodic MDP setting [12], or tabular MDPs [13], or the infinite horizon case butwith restrictive assumptions, such as linear quadratic regulator setting [36], continuous kernelizednonlinear regulator [37, 38], generative model, coherence assumption [39, 10], or a linear transitionlaw that can be completely specified by a finite-dimensional matrix [11, 12]. On the other hand,we provide convergence guarantees and error bounds for the context of linear MDPs [10, 14–19]. Despite being linear, the transition law can still have infinite degrees of freedom. To our knowledge,such guarantees in this setting are provided for the first time.
Our work applies the technique known as regularization in the online learning literature [40,
41]and Bregman proximal-point or smoothing in optimization literature [42, 43] to the LP formulationfor MDP [44–62, 15, 63].
From this perspective, we can see Deep Inverse Q-Learning [35] and
IQ-Learn [34] that consider entropy regularization in the objective as smoothing using uniformdistribution as center point.
In our case, we instead use as center point the previous iteration of thealgorithm (for the online case) or the expert (for the offline case).
From the technical point of view, the most important related works are the analysis of REPS/QREPS [64, 10, 65] and O-REPS [66] that first pointed out the connection between REPS and PPM.
Webuild on their techniques with some important differences.
In particular, while in the LP formulationof RL, PPM and mirror descent [67, 68] are equivalent, recognizing that they are not equivalent in ILis critical for stronger empirical performance.
As an independent interest, our techniques can be usedto improve upon the best rate for REPS in the tabular setting [65] and to extend the guarantees tolinear MDPs.
In order to discuss in more detail our research questions and situate them among priorrelated theoretical and practical works, we provide in Appendix A an extended literature review.
Backgroud2.1 Markov Decision Processes
The RL environment and its underlying dynamics are typically abstracted as an MDP given by a tuple( S , A , P, ν 0 , c , γ ) , where S is the state space, A is the action space, P : S×A → ∆ S is the transitionlaw, ν 0 ∈ ∆ S is the initial state distribution, c ∈ [0 , 1] |S||A| is the cost, and γ ∈ (0 , 1) is the discountfactor.
For simplicity, we focus on problems where S and A are finite but too large to be enumerated.
A stationary Markov policy π : S → ∆ A interacts with the environment iteratively, starting with aninitial state s 0 ∼ ν 0 .
At round t , if the system is at state s t , an action a t ∼ π ( ·| s t ) is sampled andapplied to the environment.
Then a cost c ( s, a ) is incurred, and the system transitions to the nextstate s t
+1 ∼ P ( ·| s, a ) .
The goal of RL is to solve the optimal control problem ρ ⋆ c ≜ min π ρ c ( π ) ,where ρ c ( π ) ≜ (1 − γ ) ⟨ ν 0 , V π c ⟩ is the normalized total discounted expected cost of π .
The state value function V π c ∈
R |S| of π , given cost c , is defined by V π c ( s ) ≜
E πs (cid:104) (cid:80) ∞ t =
γ t c ( s t , a t ) (cid:105) , where E πs denotes the expectation with respect to the trajectories generated by π starting from s 0 = s .
The optimal value function V ⋆ c ∈ R |S| is defined by
V ⋆ c ( s ) ≜ min π V π c ( s ) .
The optimal state-action value function Q ⋆ c ∈ R |S||A| , given by Q ⋆ c ( s, a ) ≜c ( s, a ) + γ (cid:80) s ′
V ⋆ c ( s ′ )
P ( s ′ | s, a ) , is known to characterize optimal behaviors.
Indeed V ⋆ c is theunique solution to the Bellman optimality equation V ⋆ c ( s ) =
min a Q ⋆ c ( s, a ) .
In addition, any deterministic policy π ⋆ c ( s ) = arg min a Q ⋆ c ( s, a ) is known to be optimal.
For every policy π , we define the normalized state-action occupancy measure µ π ∈ ∆ S×A , byµ π ( s, a ) ≜ (1 − γ ) (cid:80) ∞ t =
γ t P π ν
[ s t
=
s, a t = a ] , where P π ν 0 [ · ] denotes the probability of an eventwhen following π starting from s 0 ∼ ν 0 .
The occupancy measure can be interpreted as the discountedvisitation frequency of state-action pairs.
This allows us to write ρ c ( π ) = ⟨ µ π , c ⟩ .
2.2 Imitation Learning
Similarly to RL, the IL problem is posed in the MDP formalism, with the critical difference that thetrue cost c true is unknown.
Instead, we have access to a finite set of truncated trajectories sampled i.i.d.by executing an expert policy π E in the environment.
The goal is to learn a policy that performs betterthan π E with respect to the unknown c true .
To this end, we adopt the AL formalism [25, 33, 69, 4, 13],which carries the assumption that c true belongs to a class of cost functions C .
We then seek an apprentice policy π A that outperforms the expert across C by solving the following optimization problem
[Equation]
(1) where d C ( π, π E ) ≜ max c ∈C (cid:0) ρ c ( π ) − ρ c ( π E ) (cid:1) defines the C -distance between π and π E [4, 70, 11, 12].
Then, π A satisfies the goal of IL, since it holds that ρ c true ( π A ) − ρ c true ( π E ) ≤ ζ ⋆ ≤ 0 .
Intuitively, thecost class C distinguishes the expert from other policies.
The maximization in (1) assigns high totalcost to non-expert policies and low total cost to π E [4], while the minimization aims to find the policythat matches the expert as close as possible with respect to d C .
By writing d C in its dual form ¯ d C ( µ π , µ π E ) ≜ max c ∈C (cid:0) ⟨ µ π , c ⟩ − (cid:10) µ π E , c (cid:11) (cid:1) , it can be interpretedas an integral probability metric [71, 72] between the occupancy measures µ π and µ π E .
Dependingon how C is chosen, d C turns to a different metric of probability measures like the 1 -Wassersteindistance [73, 74] for C
= Lip 1 ( S × A ) , the total variation for C =
{ c | ∥ c ∥ ∞ ≤ 1 } , or the maximummean discrepancy for C
= { c | ∥ c
∥ H ≤ 1 } , where Lip 1 ( S × A ) denotes the space of 1 -Lipschitzfunctions on S × A , and ∥·∥ H denotes the norm of a reproducing kernel Hilbert space H [75].
In our theoretical analysis, we focus on linearly parameterized cost classes [27, 33, 4, 12, 13] ofthe form C ≜ { c w ≜ (cid:80) m i =
1 w i ϕ i | w ∈ W} , where { ϕ i } {+m i =
+} ⊂ R |S||A| + are fixed feature vectors,such that ∥ ϕ i ∥ 1 ≤ 1 for all i ∈ [ m ] , and W is a a convex constraint set for the cost weights w .
Thisassumption is not necessarily restrictive as usually in practice the true cost depends on just a few keyproperties, but the desirable weighting that specifies how different desiderata should be traded-off isunknown [25].
Moreover, the cost features can be complex nonlinear functions that can be obtained via unsupervised learning from raw state observations [76, 77].
The matrix Φ ≜ [ ϕ. . .
ϕ m ]gives rise a feature expectation vector (FEV) ρ Φ ( π ) ≜ ( ρ ϕ 1 ( π E ) , . . .
, ρ ϕ m ( π E )) ⊺ ∈ R m of a policyπ .
Then, by choosing W to be the ℓ 2 unit ball B m 1 ≜ { w ∈ R m | ∥ w ∥ 2 ≤ 1 } [25], we get a feature expectation matching objective d C ( π, π π E ) =
∥ ρ Φ ( π ) −
ρ Φ ( π E ) ∥ 2 , while for W being the probabilitysimplex ∆ [ m ] [27, 33] we have a worst-case excess cost objective d C ( π, π π E ) =
max i ∈ [ m ] (cid:0)
ρ ϕ i ( π ) −ρ ϕ i ( π E ) (cid:1) .
For clarity, we will replace c by w in the notation of the quantities defined in Section 2.1.
A Q -Convex-Analytic Viewpoint
Our methodology builds upon the convex-analytic approach to AL, first introduced by [33], with thekey difference that we consider a different convex formulation that introduces Q -functions as slackvariables.
This allows to design a practical scalable model-free algorithm with theoretical guarantees.
Let F ≜ { µ ∈ R |S||A| | ( B − γ P ) ⊺ µ = (1 − γ ) ν 0 , µ ≥ 0 } be the state-action polytope , where Pis the vector form of P , i.e., P ( s,a ) ,s ′ ≜ P ( s ′ | s, a ) , and B is a binary matrix defined by B ( s,a ) ,s ′ ≜ 1if s = s ′ , and B ( s,a ) ,s ′ ≜ 0 otherwise.
The linear constraints that define the set F , also known as
Bellman flow constraints , precisely characterize the set of state-action occupancy measures.
Proposition 1 (78) .
We have that µ ∈ F if and only if there exists a unique stationary Markov policyπ such that µ = µ π .
If µ ∈ F then the policy π µ ( a | x ) ≜ µ ( x,a ) (cid:80) a ′∈A µ ( x,a ′ ) has occupancy measure µ .
Using Proposition 1 and the dual form of the C -distance ¯ d C ( µ , µ π E ) = max w ∈W (cid:10) µ − µ π E , c w (cid:11) ,it follows that (1) is equivalent to the primal convex program ζ
⋆ = min µ { ¯ d C ( µ , µ π E ) | µ ∈ F } .
In particular for W = ∆ [ m ] and by using an epigraphic transformation, we end up with an LPprogram [33, 39], while for W =
B m 1 we get a quadratic objective with linear constraints [25].
A slight variation of the above reasoning is to introduce a mirror variable d and split the Bellmanflow constraints in the definition of F .
We then get the primal convex program
[Equation]
Primal [Equation] where the new polytope is given by M ≜ { ( µ , d ) | B ⊺ d = γ P ⊺ µ + (1 − γ ) ν 0 , µ = d , d ≥ 0 } .
This overparameterization trick has been first introduced by Mehta and Meyn [79] and has beenrecently revisited by [10, 19, 60, 80, 9, 81].
A salient feature of this equivalent formulation is that itintroduces a Q -function as Lagrange multiplier to the equality constraint d = µ , and so lends itselfto data-driven algorithms.
To motivate further this new formulation, in Appendix C, we shed light toits dual and provide an interpretation of the dual optimizers.
In particular, when W = B m 1 , we showthat ( V ⋆ w true , Q ⋆ w true , w true ) is a dual optimizer.
For our theoretical analysis we focus on the linear MDP setting [15], i.e., we assume that the transitionlaw is linear in the feature mapping.
We denote by ϕ ( s, a ) the ( s, a ) -th row of Φ .
Assumption 1 (Linear MDP) .
There exists a collection of m signed measures ω
= ( ω 1 , . .
ω m ) on
S , such that P ( ·| s, a ) = ⟨ ω ( · ) , ϕ ( s, a ) ⟩ , for all ( s, a ) .
Moreover ϕ
( s, a ) ∈ ∆ [ m ] , for all ( s, a ) .
Assumption 1 essentialy says that the transition matrix P has rank at most m , and P = ΦM forsome matrix M ∈
R m ×|S| .
It is worth noting that in the case of continuous MDPs, despite beinglinear, the transition law P ( ·| s, a ) can still have infinite degrees of freedom.
This is a substantialdifference from the recent theoretical works on IL [11–13] which consider either a linear quadraticregulator, or a transition law that can be completely specified by a finite-dimensional matrix suchthat the degrees of freedom are bounded.
Assumption 1 enables us to consider a relaxation of (Primal).
In particular, we aggregate theconstraints µ = d by imposing Φ ⊺ µ = Φ ⊺ d instead, and introduce a variable λ =
Φ ⊺ µ .
It followsthat λ
lies in the m -dimensional simplex ∆ [ m ] .
Then, we get the following convex program
[Equation]
where M Φ ≜ { ( λ , d ) | B ⊺ d = γ M ⊺ λ + (1 − γ )
ν 0 , λ =
Φ ⊺ d , λ ∈ ∆ [ m ] , d ∈ ∆ S×A } .
As shown in [19, 10, 80], for linear MDPs, the set of occupancy measures F can be completelycharacterized by the set M Φ (c.f., Proposition 2).
While the number of constraints and variablesin (Primal ′ ) is intractable for large scale MDPs, in the next paragraph, we show how this problemcan be solved using a proximal point scheme.
Proximal Point Imitation Learning
By using a Lagrangian decomposition, we have that (Primal ′ ) is equivalent to the following bilinearsaddle-point problem
[Equation]
where A ∈ R (2 m + |S| ) × ( m + |S||A| ) , and b ∈ R ( m + |S| + |S||A| ) are appropriately defined (see Appendix D), x ≜ [ λ ⊺ , d ⊺ ] ⊺ , y ≜ [ w ⊺ , V ⊺ , θ ⊺ ] ⊺ , X ≜ ∆ [ m ] × ∆ S×A , and Y ≜ W × R |S| × R m .
Since in practice we do not have access to the whole policy π E , but instead can observe a finiteset of i.i.d.
sample trajectories D E ≜ { ( x ( l ) 0 , a ( l ) 0 , x ( l ) 1 , a ( l ) 1 , . . .
, x ( l ) H , a ( l ) H ) } n E l =1 ∼ π E , we define thevector (cid:98) b by replacing ρ Φ ( π E ) with its empirical counterpart
ρ Φ ( (cid:99) π E ) (by taking sample averages) inthe definition of b .
We then consider the empirical objective f ( x ) ≜ max y ∈Y (cid:68) y , Ax + (cid:98) b (cid:69) andapply PPM on the decision variable x .
For the λ -variable we use the relative entropy D ( λ || λ ′ )
≜(cid:80) mi =1 λ
( i ) log λ ( i )
λ ′ ( i ) , while for the occupancy measure d we use the conditional relative entropy
H ( d || d ′ ) ≜ (cid:80) s,a d ( s, a ) log π d ( a | s ) π d ′ ( a | s ) .
With this choice we can rewrite the PPM update as( λ k +1 , d k +1 ) =
[Equation]
where we used primal feasibility to replace λ k with Φ ⊺ d k as the center point of the relative entropy.
PPM is implicit, meaning that it requires the evaluation of the gradient at the next iterate x k +1 .
Sucha requirement makes it not implementable in general.
However, in the following, we describe a procedure to apply proximal point to our specific f ( x ) .
The following Proposition summarizes the result.
Proposition 2.
For a parameter θ ∈ R m , we define the logistic state-action value function Q θ ∈
R |S||A| by Q θ ≜ Φ θ , and the k -step logistic state value function V k θ ∈ R |S| by
[Equation]
Moreover, we define the k -step reduced Bellman error function δ k w , θ ∈ R m by δ k w , θ ≜ w + γ MV k θ −θ .
Then, the PPM update ( λ ⋆k , d ⋆k ) in 2 is given by
[Equation]
where ( w ⋆k , θ ⋆k ) is the maximizer over W × R m of the k -step logistic policy evaluation objective
[Equation]
Moreover, it holds that G k ( w ⋆k
[Equation] +1 α H
( d ⋆k || d k − 1 ) .
If in addition Assumption 1 holds, then d ⋆k is a valid occupancy measure, i.e.,d ⋆k ∈ F and so d ⋆k = µ π d ⋆k .
The proof of Proposition 2 is broken down into a sequence of lemmas and is presented in Appendix E.
It employs an analytical-oracle g : Y
→ X given by
[Equation]
and a max-oracle h : X → Y given by h ( x ) ≜ arg max y ∈Y ⟨ y , Ag ( y ; x ) ⟩ + 1 τ D Ω ( g ( y ; x ) || x ) .where
we used D Ω to compact the two divergences.
By noting that the PPM update Equation (2) can berewritten as x k +
=
g ( h ( x k ); x k ) , its analytical computation is reduced to the characterization of thetwo aforementioned oracles.
In particular, the updates (3)–(4) come from the analytical-oraclewhile (5) is the objective of the max-oracle .
The choice of conditional entropy as Bregman divergence for the λ variable living in the probabilitysimplex is standard in the optimization literature and is known to mitigate the effect of dimension.
Inparticular, as noted in [28], the classic REPS algorithm [64] can be seen as mirror descent with relativeentropy regularization.
On the other hand, the choice of conditional entropy as Bregman divergencefor the d variable is less standard and has been popularized by Q-REPS [10].
Such particulardivergence leads to an actor-critic algorithm that comes with several merits.
By Proposition 2, it isapparent that we get analytical softmin updates for the policy π d rather than the occupancy measure d .
Moreover, these softmin updates are expressed in terms of the logistic Q -function and do not involvethe unknown transition matrix P .
Consequently, we avoid the problematic occupancy measureapproximation and the restrictive coherence assumption on the choice of features needed in [61, 39],as well as the biased policy updates appearing in REPS [64, 65].
In addition, the newly introducedlogistic policy evaluation objective G k ( w , θ ) has several desired properties.
It is concave and smoothin ( w , θ ) and has bounded gradients.
Therefore, it does not suffer from the pathologies of the squared
Bellman error [82] and does not require heuristic gradient clipping techniques.
Moreover, unlike [39]it allows a model-free implementation without the need for a generative model (see Section 4.1)
We stress the fact that the max-oracle of our proximal point scheme performs the cost update andpolicy evaluation phases jointly.
This is a rather novel feature of our algorithm that differs fromthe separate cost update and policy evaluation step used in recent theoretical imitation learningworks [11, 13, 12].
Our joint optimization over cost and Q -functions avoids instability due toadversarial training and can also recover an explicit cost along with the Q -function without requiringknowledge or additional interaction with the environment (see Section 5).
It is worth noting thatapplication of primal-dual mirror descent to (SPP) does not have this favorable property.
While inthe standard MDP setting, proximal point and mirror descent coincide because of the linear objective,in imitation learning proximal point optimization makes a difference.
In Appendix K, we includea more detailed discussion and numerical comparison between PPM and mirror descent updates.
4.1 Practical Implementation
Exact optimization of the logistic policy evaluation objective is infeasible in practical scenarios, dueto unknown dynamics and limited computation power.
In this section, we design a practical algorithmthat uses only sample transitions by obtaining stochastic (albeit biased) gradient estimators.
Proposition 2 gives rise to Proximal Point Imitation Learning ( P 2 IL ), a model-free actor-critic IRLalgorithm described in Algorithm 1.
The key feature of P 2 IL is that the policy evaluation stepinvolves optimization of a single smooth and concave objective over both cost and state-action valuefunction parameters.
In this way, we avoid instability or poor convergence in optimization due tonested policy evaluation and cost updates, as well as the undesirable properties of the widely usedsquared Bellman error.
In particular, the k th iteration of P 2 IL consists of the following two steps : (i)( Critic Step ) Computation of an approximate maximizer ( w k , θ k )
≈ arg max w , θ G k ( w , θ ) of theconcave logistic policy evaluation objective, by using a biased stochastic gradient ascent subroutine;(ii)
( Actor Step ) Soft-min policy update π k ( a | s ) ∝ π k − 1 ( a | s ) e − αQ θ k ( s,a ) expressed in terms of thelogistic Q -function.
The domain Θ in Algorithm 1 is the ℓ ∞ -ball with appropriately chosen radius D .
Moreover, Π Θ ( x ) ≜arg min y ∈ Θ ∥ x − y ∥ 2 (resp.
Π W ( w ) ) denotes the Euclidean projection of x (resp.
w ) onto Θ (resp.
W ).
The following proposition ensures that max w , [Equation]
Therefore, this constraint does not change the problem optimality, but will considerably acceleratethe convergence of the algorithm by considering smaller domains.
Proposition 3.
It holds that ∥ θ ⋆k ∥ ∞ ≤ M 1+ | log β | 1 − γ ≜ D , for some constant M ≥ 1 .
Input: Feature matrix Φ , expert demonstrations D E , number of iterations K , step sizes η and α
Input: Number of SGD iterations T, SGD learning rates β
= { β t } T − 1 t =0 , number-of-samples function n : N → N
Initialize π 0 as uniform distribution over A
Compute the empirical FEV ρ Φ ( (cid:99) π E ) using expert demonstrations D E for k = 1 , . . .
K do //
Critic-step (policy evaluation)
Initialize θ k, 0 = 0 and w k, 0 = 0 for t = 0 , . . .
T − 1 do
Run π k − 1 and collect i.i.d.
samples { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′ ( n ) k − 1 ) } n ( t )+1 n =
Compute biased stochastic gradient estimators ( (cid:98) ∇ w G k ( w k , θ k ) , (cid:98) ∇ θ G k ( w k , θ k ))
=
BSGE ( k, w t , θ t , n ( t ))
w k,t +1 =
Π W ( w k,t + β t (cid:98) ∇ w G k ( w k , θ k ))
θ k,t +1 = Π Θ ( θ k,t + β t (cid:98) ∇ θ G k ( w k , θ k )) end for ( w k , θ k ) =
( 1 T (cid:80)
Tt =1 w k,t , 1 T (cid:80)
Tt
=1 θ k,t ) //
Actor-step (policy update)
Policy update: π k ( a | s ) ∝ π k − 1 ( a | s ) e − αQ θ k
( s,a ) end for
Output: Mixed policy (cid:98) π K of { π k } k ∈ [ K ]
[Figure]
In order to estimate the gradients ∇ θ G k ( w , θ ) and ∇ w G k ( w , θ ) we invoke the Biased Stochastic
Gradient Estimator subroutine (BSGE) (Algorithm 2) given in Appendix H. By using the linear
MDP Assumption 1 and leveraging ridge regression and plug-in estimators, the proposed stochasticgradients can be computed via simple linear algebra with computational complexity poly ( m, n ( t )) ,independent of the size of the state space.
4.2 Theoretical Analysis
The first step in our theoretical analysis is to study the propagation of optimization errors madeby the algorithm on the true policy evaluation objective.
In particular at each iteration step k , theideal policy evaluation update ( w ⋆k , θ ⋆k ) and the ideal policy update π ⋆k are given by ( w ⋆k , θ ⋆k ) =
arg max w , θ G k ( w , θ ) , and π ⋆k ( a | s ) =
π k − 1 ( a | s ) e − α
( Q θ ⋆k ( s,a ) − V k θ ⋆k ( s )) .
On the other hand, considerthe realised policy evaluation update ( w k , θ k ) such that G k ( w ⋆ k , θ ⋆ k ) − G k ( w k , θ k ) =
ϵ k , the corresponding policy π k given by π k = π k − 1 ( a | s ) e − α
( Q θ k ( s,a ) − V k θ k ( s )) , and let d k ≜ µ π k .
We denote by(cid:98)
π K the extracted mixed policy of { π k } Kk =1 .
We are interested in upper-bounding the suboptimalitygap d C ( (cid:98) π K , π E ) of Algorithm 1 as a function of ε k .
To this end, we need the following assumption.
Assumption 2.
It holds that λ min ( E ( s,a ) ∼ d k ϕ ( s, a ) ϕ
( s, a ) T ) ≥ β , for all k ∈ [ K ] .
[Figure]
2d
Assumption 2 states that every occupancy measure d k induces a positive definite feature covariancematrix, and so every policy π k explores uniformly well in the feature space.
This assumption iscommon in the RL theory literature [83–88].
It is also related to the condition of persistent excitationfrom the control literature [89].
We can now state our error propagation theorem.
Theorem 1.
Let (cid:98) π K be the output of running Algorithm 1 for K iterations, with n E ≥ 2log( 2 m δ ) εexperttrajectories of length H ≥ 11 − γ log( 1 ε ) .
Let C ≜ 1 βη
(cid:0)(cid:113) 2 α 1 − γ + √ 8 η (cid:1) + (cid:113) 18 α 1 − γ .
Then, with probabilityat least 1 − δ , it holds that d C ( (cid:98) π K , π E ) ≤ 1 K (cid:16) D ( λ ∗ ||
Φ T d 0 )
η
+ H (
[Equation]
By Theorem 1, whenever the policy evaluation errors ε k , as well as the estimation error ε can bekept small, Algorithm 1 ouputs a policy (cid:98) π K with small suboptimality gap ρ c true ( (cid:98) π K ) − ρ c true ( π E ) .
Notably, there is no direct dependence on the size of the state space or the dimension of the featurespace.
In the ideal case, where ε k = 0 for all k , the convergence rate is O (1 /K ) .
The provided errorpropagation analysis still holds with general function approximation, i.e., in the context of deep RL.
Indeed, by choosing Φ =
I , Assumption 1 is trivially satisfied and the θ variable in the objective G kis replaced by a Q -function.
In practice, the estimation error ε can be made arbitrary small, by increasing the number of expertdemonstrations n E .
Moreover, the next theorem ensures that under Assumptions 1 and 2 the biasedstochastic gradient ascent (BSGA) subroutine has sublinear convergence rate.
Theorem 2.
Let ( w k , θ k ) be the output of the BSGA subroutine in Algorithm 2 for T iterations, withn (
[Equation]
sample transitions, and learning ratesβ t
= O ( 1 √ t ) .
Then, ϵ k = G k ( w ⋆k , θ ⋆k ) − G k ( w k , θ k ) ≤
O ( max { η, 1 } mD β √ T ) , with probability 1 − δ .
Corollary 1 (Resource guarantees) .
Choose η = α = 1 and let K
= Ω (cid:0) ϵ − 1 (cid:1) , T = Ω (cid:0) ϵ − 4 (cid:1) .
Thenfor Ω ( KT ) =
Ω (cid:0) ϵ − 5 (cid:1) sample transitions, Ω (cid:0) ε − 2 (cid:1) expert trajectories and approximately solvingΩ (cid:0) ϵ − 1 (cid:1) concave maximization problems, we can ensure d C ( (cid:98) π, π E ) ≤
O ( ϵ + ε ) , with high probability.
Offline Setting.
Finally, we notice that using Φ ⊺ µ π E as the reference distribution for the relativeentropy we can obtain an offline algorithm that does not require environment interactions.
Byreinterpreting smoothing [43] as one step of proximal point, and using similar arguments as in theproof of Theorem 1, we can provide similar theoretical guarantees for the offline setting.
The detailsas well as the optimization of the empirical policy evaluation objective are presented in Appendix J(see Theorems 4 and 6).
Experiments
In this section, we demonstrate that our approach achieves convincing empirical performance in bothonline and offline IL settings on several environments.
The precise setting is detailed in Appendix L.
Online Setting.
We first present results in various tabular environments where we can implement ouralgorithm without any practical relaxation outperforming GAIL [4], AIRL [5] and IQ-Learn [34].
Results are given in Figure 1.
Good performance but inferior to IQ-Learn is observed also for continuousstates environments (CartPole and Acrobot) where we used neural networks function approximation.
Offline Setting.
Figures 2a to 2c shows that our method is competitive with the state-of-the-artoffline IL methods IQLearn [34] and AVRIL [90] that recently showed performances superior toother methods like [91][8].
We also tried our algorithm in the complex image-based Pong task fromthe Atari suite.
Figure 2d shows that the algorithm reaches the expert level after observing 2 e 5 expertsamples.
We did not find AVRIL competitive in this setting, and skip it for brevity.
In these settings,we verified that the algorithmic performance is convincing even for costs parameterized by neuralnetworks.
Continuous control experiments.
We attain the expert performance also in 2 MuJoCo environments:
Ant , HalfCheetah , Hopper , and Walker (see Figures 2e to 2h).
The additional difficulty inimplementing the algorithm in continuous control experiments is that the analytical form of the policyimprovement step is no longer computationally tractable because this would require to compute anintegral over the continuous action space.
Therefore, we approximated this update using the Soft
Actor Critic (SAC) [92] algorithm.
SAC requires environment samples making the algorithm online.
The good empirical result opens the question of analyzing policy improvement errors as in [93].
Recovered Costs.
A unique algorithmic feature of the proposed methodology is that we can explicitlyrecover a cost along with the Q -function without requiring adversarial training.
In Figures 6 and 7, wevisualize our recovered costs in several simple tabular environments.
Most importantly, we verify thatthe recovered costs induce nearly optimal policies w.r.t.
the unknown true cost function.
Comparedto IQ-Learn, we do not require knowledge or further interaction with the environment.
Regarding thetransfer capability to new dynamics, we experimented on Gridworld (Figure 8) and noticed that therecovered cost induces an optimal policy for the new dynamics while the imitating policy fails.
Weelaborate details in Appendix M.
Discussion and Outlook
In this work, we studied a Proximal Point Imitation Learning ( P 2 IL ) algorithm with both theoreticalguarantees and convincing empirical performance.
Our methodology is rooted in classical optimization tools and the LP approach to MDPs.
The most significant merits of P 2 IL are the following: (i) Itoptimizes a convex and smooth logistic Bellman evaluation objective over both cost and Q-functions.
In particular, it avoids instability due to adversarial training and can also recover an explicit cost alongwith Q function; (ii)
In the context of linear MDPs, it comes with efficient resource guarantees anderror bounds for the suboptimality of the learned policy (Theorem 2 and Corollary 1).
In particular,given poly(1 /ε, log(1 /δ ) , m ) many samples , it recovers an ε -optimal policy, with probability 1 − δ .
Notably, the bound is independent of the size of the state-action space; (iii) Beyond the linear MDPsetting, it can be implemented in a model-free manner, for both online and offline setups, with generalfunction approximation without losing its theoretical specifications.
This is justified by providing anerror propagation analysis (Theorems 1 and 4), guaranteeing that small optimization errors lead tohigh-quality output policy; (iv) It enjoys not only theoretical guarantees but also favorable empiricalperformance.
At the same time, our newly introduced methods bring challenges and open questions.
One interestingquestion is whether one can accelerate the PPM updates and improve the convergence rate.
Anotherdirection for future work is to provide rigorous arguments for the near-optimality of the recovered costfunction.
On the practical side, we plan to conduct experiments in more challenging environmentsthan MuJoCo and Atari.
We hope our new techniques will be useful to future algorithm designersand lay the foundations for overcoming current limitations and challenges.
In Appendix B, we pointout in detail a few interesting future directions.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We reflect the contribution described in the introduction
with the theoretical results in Section 4, Section 4.2(b) Did you describe the limitations of your work?
[Yes] We state the assumptions neededfor the analysis in Sections 3 and 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work is mainly theoretical, we do not foresee potential negative impacts on the society.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We acknowledge habing read the review guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state theassumptions needed for the analysis in Section 3 Section 4.2(b)
Did you include complete proofs of all theoretical results?
[Yes] The main results arestated in Section 4.2 and the proofs are included as supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The code isincluded in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Training details are provided in the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] We averaged multiple seeds whenever possible
computationally.
We ran a single seed for the computational expensive Pong environ-
ment.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We specified the resource in the
Supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite [34] for using their codebase and their expert data.
(b) Did you mention the license of the assets?
[Yes] We mention the license in the
Supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We attach the code to the supplementary.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We discuss this in the supplemnetary material
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
We do not use personal data.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
We did not involve human partecipants.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
We did not involve human partecipants.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
We did not involve human partecipants.
[Table]
A Related Literature (Extended)
In order to state our research questions and situate them among prior related theoretical and practicalworks, we provide an extended literature review.
Theoretical Imitation Learning.
Our work is related to recent actor-critic IL schemes with theoreticalguarantees for different MDP models, and different policy evaluation objectives (e.g., minimizingthe squared Bellman error)
Contrary to these actor-critic schemes, in ourproximal-point imitation learning algorithm, the policy evaluation step involves optimization of asingle objective over both cost and Q -functions.
In this way, we avoid instability or poor convergencedue to nested policy evaluation and cost update steps [35] as well as the undesirable properties ofthe widely used squared Bellman error [82].
Moreover, for the context of linear MDPs [10, 14–19],we provide guarantees and convergence rates for the suboptimality of the learned policy, under mildassumptions, significantly weaker than those found in the literature until now.
To our knowledge,such guarantees in this setting are provided for the first time.
It is worth noting that in the case ofcontinuous MDPs, despite being linear, the transition law can still have infinite degrees of freedom.
This is a substantial difference from the recent theoretical works on IL [36, 11, 37, 12, 13] whichconsider either tabular MDPs [13], or a linear quadratic regulator [36], or a linear transition law thatcan be completely specified by a finite-dimensional matrix [11, 12].
In the last case, the degrees offreedom are bounded, and thus mitigate the challenges in estimating the transition model.
Indeed,the linear MDP setting studied in [11, 12] reduces the unknown dynamics problem to estimatingan unknown finite-dimensional matrix, which differs from our nonparametric approach.
We alsonote that [11, 94] require the restrictive assumption of bounded concentrability coefficients, whilethis is not the case for the analysis in this paper.
The convergence and generalization of actor-critic
IL schemes for general MDPs has been studied in [70].
However, the authors in [70] only providelocal optimality convergence guarantees, i.e., convergence to a stationary point.
On the contrary, ouralgorithm provides global convergence guarantees for the linear MDP setting.
Moreover, we accountfor potential policy evaluation errors , presenting an error propagation analysis that leads to rigorousguarantees for both online and offline setting, beyond the linear MDP assumption.
Indeed, it is worthnoting that the provided error propagation analysis justifies using our derived actor-critic schemewith general function approximation.
A scalable deep reinforcement learning implementation ispossible, without losing the theoretical guarantees of Theorem 1.
The work [37] studies offline IL forthe continuous kernelized nonlinear regulator and Gaussian process setting [38].
We notice that thissetting is different from the linear MDP model studied in this paper, and each one does not implythe other.
Finally, a recent theoretical IL work that is rooted in the LP approach to MDPs is [39].
The authors consider a Lagrangian reformulation of the problem and design a stochastic primal-dualalgorithm with explicit performance bounds on the quality of the extracted policy.
The most importantlimitations of the primal-dual algorithm [39] are (i) the need of a generative oracle, (ii) restrictedcoherence assumptions on the choice of features, as well as (iii) the problematic occupancy measureapproximation.
These limitations lead to poor practical performance for challenging high-dimensionaland model-free IL setups.
On the other hand, our algorithm overcomes these difficulties by applyinga proximal point update to an alternative Q -LP formulation [9, 80].
This results to a model-freeactor-critic scheme with explicit tractable softmax policy updates.
Compared with the setting in [39],where access to a generative-model oracle is assumed, we only have the ability to execute learnedpolicies in the underlying MDP to generate trajectories.
This assumption is considerably weaker thathaving a simulator-based MDP, however it is stronger than having ”irreversible experience”, wherethe learner must follow a single trajectory without having access to a reset action , that obtains a newtrajectory from the initial state distribution.
Most importantly our algorithm enjoys not only strongtheoretical guarantees, but also favorable practical performance.
Approximate Linear Programming.
There is an emerging body of literature [50, 55, 57, 95, 56,58, 96, 60, 97–99, 62, 15, 63, 10] that studies ALP for the forward RL.
While this approach datesback to 1960s [44], it has recently witnessed an interesting renaissance for its potential to provide asolid formal framework for newly derived methods, as well as a deeper understanding of existingempirically successful algorithms.
In this paper, we present scalable imitation learning algorithms with theoretical guarantees rooted in the LP approach, highlighting how historical key limitations havebeen eliminated.
Prior approximate linear programming (ALP) approaches developed algorithms forsolving large-scale and/or continuous MDPs on a low-dimensional subspace by reducing the numberof constraints (e.g., by constraint sampling)
However, these prior works either scale badlywith the size of the state-action spaces or require access to samples from a distribution that depends onthe optimal policy.
Moreover, they focus mainly on the approximation of the optimal value but not somuch on extracting a near optimal policy.
On the other hand, a recent line of works [57, 60, 96, 15, 63]solve the problem for large-scale MDPs by employing stochastic primal-dual methods, in light of
Lagrangian duality.
Although this approach achieves state-of-the-art sample complexity guarantees, itshows poor performance in practice.
First, current primal-dual algorithms need access to a simulator,mitigating implicitly the problem of exploration, Second, when dealing with linear relaxations of
MDPs [10, 39] one needs to impose a restrictive coherence assumptions to ensure that small dualitygap for the linearly relaxed LP implies small suboptimality gap for the extracted policy.
Finally, whiletheir is enough intuition behind the use of linear function approximation for value functions, this isnot the case for occupancy measure approximation.
A new breed of algorithms that seem to overcomethese difficulties is based on an alternative Q -LP formulation of RL.
This approach has been firstintroduced by Mehta and Meyn [79] and has been recently revisited by [10, 19, 60, 80, 9, 81].
Asalient feature of this equivalent formulation is that it introduces a Q -function as slack variables, andso lends itself to data-driven algorithms.
Our work is inspired by these line of works.
The most relatedworks are the analysis of REPS/Q-REPS [64, 10, 65] and O-REPS [66] that first pointed out theconnection between REPS and PPM.
We build on their techniques with some important differences.
In particular, while in the LP formulation of RL, PPM and mirror descent [67, 68] are equivalent,recognizing that they are not equivalent in IL is critical for stronger empirical performance.
Moreover,our techniques can be used to improve upon the best rate for REPS in the tabular setting [65] and toextend their guarantees to Linear MDPs.
State-of-the-art Imitation Learning.
Generative adversarial imitation learning (GAIL) [4] and otherfollow-up works [5–8] formulate the IL as a minimax adversarial problem similar to a GAN [100] andleverage primal-dual optimization tools.
In particular, GAIL solves IL with alternating updates of bothpolicy and cost functions.
On the other hand, a recent line of work [34, 101, 102] bypasses the need ofoptimizing over cost functions and thus avoids instability due to adversarial training.
Although thesealgorithms achieve impressive empirical performance in challenging high dimensional benchmarktasks, they are hampered by limited theoretical understanding.
This is the fundamental differencefrom our work, which enjoys both favorable practical performance and strong theoretical guarantees.
Moreover, a unique algorithmic feature of our proposed methodology is a convex and smooth logisticpolicy evaluation objective that optimizes jointly cost and Q -functions.
As a result, our algorithm hasthe additional practical benefit that can also recover an explicit cost along with the Q-function withoutrequiring knowledge or further interaction with the environment (as in [34, 101, 102]).
Therefore, therecovered cost functions show promising transfer capability to new dynamics.
In addition, unlike
IQ-Learn [34], in our online IL algorithm, instead of regularizing the IL objective, the key idea is topenalize the divergence between the current policy and the policy obtained at the previous iteration.
We do so by employing a Bregman proximal point update.
Most importantly, as we have alreadyhighlighted, the convergence properties of [34, 101, 102]) remain largely elusive in the functionapproximation and model-free regime.
It is unclear whether the sampling-based variants of theiralgorithms converge to a global optimum or if they converge at all, even for the simple tabular setting.
B Future directions
In this work, we studied a proximal point imitation learning algorithm with both theoretical guaranteesand convincing empirical performance in challenging benchmark tasks.
Our methodology is rootedin classical stochastic optimization tools and in the LP approach to MDPs.
We hope that our newtechniques will be useful for future algorithm designers and lay foundations for overcoming currentlimitations and challenges.
We point out a few interesting directions.
Accelerated proximal point.
An appealing possibility is to study an accelerated proximal pointscheme with inexact updates and achieve faster convergence rates.
While there has been an effortin this direction [103, 104], the acceleration relies on the triangle/quadrangle scaling propertyassumption [105] that does not hold for KL divergence over the simplex.
Understanding if it ispossible to accelerate PPM without such an assumption is an open question, whose solution has directapplication to the LP formulation of RL and imitation learning.
Primal-dual methods with conditional relative entropy.
Recent primal-dual RL algorithms rootedin the LP approach to MDPs achieve state-of-the-art sample complexity guarantees.
See, for example,[61] for exact gradients, [106, 15] for stochastic gradients, and [39] for the imitation learningproblem.
The most important disadvantages of primal-dual RL algorithms are (i) the need of agenerative oracle, (ii) restricted coherence assumptions on the choice of features, as well as (iii)the problematic occupancy measure approximation.
Unfortunately, these limitations lead to poorpractical performance for challenging high-dimensional and model-free RL and IL setups.
On theother hand, our algorithm overcomes these difficulties but requires to approximately solve a smalldimensional convex program repetitively.
It is also challenging to account for the biased gradientestimates beyond the linear MDP setting.
It is promising to investigate if by combining the alternative
Q -LP formulation and the conditional relative entropy as Bregman divergence in a primal-dual mirrordescent scheme
, one can avoid the current practical limitations of primal-dual RL methods.
It is alsointeresting that in this case, the action-value parameters will be updated by taking one gradient stepeach time, instead of solving a small-dimensional convex program.
Inexact policy improvement update.
The error propagation presented in this work accommodatesfor errors only in the policy evaluation phase, while it assumes that the policy improvement stepcan be implemented exactly.
This happens in other related works like [10, 107].
In contrast, theerror propagation analysis in [93] takes into account an error in the policy improvement step butunfortunately it does not provide a way to ensure that such an error is small.
Future research effortwill aim to include in our error propagation analysis a term given by inexact policy improvementsteps, ensure that such errors are small and characterizing the deterioration in the sample complexityunder policy improvement errors.
This kind of analysis would be important for continuous actionsenvironment where the softmax policy update can not be computed in closed-form.
C Dual Program Interpretation
To motivate further the Primal formulation, we shed light to its dual and provide an interpretationof the dual optimizers.
For brevity, we focus on the case W =
B
m 1 .
The proof can be found in
Appendix C.1 and is based on strong duality between the two convex programs.
Proposition 4.
The dual convex program is given by
[Equation]
Moreover, for W = B m 1 , a triple ( V A , Q A , w A ) is dual optimal if and only if (i) π E is optimal for the
RL problem with cost c = c w A , (ii) V A
= V ⋆ w A , (iii) Q A = Q ⋆ w A , and (iv) w A ∈ W .
In particular,( V ⋆ w true , Q ⋆ w true , w true ) is a dual optimizer.
Proposition 4 states that the set of dual optimal costs c w A is the set of costs in C for which the expert isoptimal.
In this case, the optimal V A coincides with the corresponding optimal value function 1 , whilethe optimal Q A coincides with the corresponding optimal state-action value function.
In particular,the true weights w true , the true optimal value function V ⋆ w true and the true optimal state-action valuefunction Q ⋆ w true are dual optimizers.
Therefore, the presented Q -convex approach allows to recoveran optimal solution to the original problem (1) from both the (Primal) and (Dual) formulations: it can be obtained either as the induced policy of a primal optimal occupancy measure or as a greedypolicy associated to a dual optimal Q -function.
In Section 4, we generalize the later observation toimplement PPM using softmin updates in terms of Q -functions.
C.1 Proof of Proposition 4
We recall the alternative Q -LP approach to MDPs [79, 9, 19, 80, 10].
Let c ∈ R
|S||A| be a costfunction.
The forward RL problem is equivalent to the following linear programs
[Equation]
We have that if π ⋆ is an optimal policy for the forward RL problem with cost c , then ( µ π ⋆ , µ π ⋆ ) isoptimal for (Primal Q -LP) and conversely if ( µ ⋆ , d ⋆ ) is optimal for (Primal Q -LP), then π µ ⋆ is anoptimal policy for the forward RL problem with cost c .
Moreover, ( V ⋆ c , Q ⋆ c ) is an optimal solutionto (Dual Q -LP) and it is the unique optimizer when ν 0 ∈ R |S| ++ .
For the following results, we willassume without loss of generality that ν 0 ∈ R |S| ++ .
Proof of Proposition 4.
We first derive the dual convex program.
We have,
[Equation]
where the second equality follows by Sion’s minimax theorem [108], since M is convex andcompact, W is convex and the objective is bilinear, the third equality follows by introducing Lagrangemultipliers V and Q , and the fifth equality follows by linear duality.
Note that the derivations holdfor any convex set W .
From now on we consider the case W =
B m 1 = { w ∈ R m | ∥ w ∥ 2 ≤ 1 } .
Then, the (Primal)program can be written in the form
[Equation]
Primal [Equation] where in the last equality we used that the ℓ 2 -norm is self-dual, that is, the dual norm of the ℓ 2 -norm isstill the ℓ 2 -norm.
Therefore, when W = B m 1 , we get a quadratic objective with linear constraints [25].
Assume first that ( V A , Q A , w A ) is optimal for (Dual).
Then,
[Equation] [Equation]
where (6) holds because ( V A , Q A , w A ) is feasible to (Dual), and (7) holds by optimality.
Therefore,( V A , Q A ) is feasible for (Dual Q -LP) with cost c
= c w A .
Moreover, ( µ π E , µ π E ) is feasible for (Primal
Q -LP) with cost c =
c w A .
Therefore,
[Equation]
However, by (7) we get that (1 − γ ) ⟨ ν 0 , V A ⟩ = (cid:10) µ π E , c
w A (cid:11) .
Thus, ( µ π E , µ π E ) is optimal for (Primal
Q -LP) with cost c = c w A and ( V A , Q A ) is optimal for (Dual Q -LP) with cost c
= c w A .
Thus π E isoptimal for the forward RL problem with cost c w A , V A
= V ⋆ c w A , and Q A
= Q ⋆ c w A
Conversely, assume that w A ∈ W , π E is optimal for c w A , V A
= V ⋆ w A , and Q A = Q ⋆ w A .
Then, wehave that ( µ π E , µ π E ) is optimal for (Primal Q -LP) with cost c w A , and ( V A , Q A ) is optimal for (Dual
Q -LP) with cost c w A .
By dual feasibility, we get
[Equation]
Moreover, by primal-dual optimality, we have
[Equation]
From (9), we get that ( V A , Q A , w A ) is feasible to (Dual).
Since ζ ⋆ = 0 , by (10), we conclude that( V A , Q A , w A ) is optimal for (Dual).
D Saddle-Point Formulation
By using a compact notation, we have that Primal ′ is equivalent to the following bilinear saddle-pointproblem
[Equation]
where
A ≜
[Equation]
x ≜ [ λ ⊺ , d ⊺ ] ⊺ , y ≜ [ w ⊺ , V ⊺ , θ ⊺ ] ⊺ , X ≜ ∆ [ m ] × ∆ S×A , and Y ≜ W × R |S| × R m .
E Proof of Proposition 2
Proof of Proposition 2.
We break the proof in three parts.
In the first two parts, we introduce andcompute the explicit forms of the oracles, while in the third part we derive the proximal point updates.
Analytical oracle.
We characterize the analytical-oracle by employing the first-order optimalityconditions for λ and d .
In particular, at each iteration step k , for any [ w ⊺ , V ⊺ , θ ⊺ ] ⊺ , we have that the
Lagrangian of the optimization problem in the definition of the analytical-oracle has the form
[Equation]
[Equation]
where we considered a Lagrangian multiplier τ for the simplex constraint (cid:80) i λ ( i )
= 1 .
Now takingthe derivatives with respect to to λ and d , we obtain the following first order optimality conditions:
[Equation]
Therefore
[Equation]
we obtain
[Equation]
where δ k w , θ ≜ w + γ MV k θ − θ .
In addition, the simplex constraint (cid:80) i λ ( i )
= 1 is satisfied bychoosing τ = τ k w , θ , where
[Equation]
Moreover, by setting Q θ =
Φ θ , we get
[Equation]
Equation (4) follows by noting that the constraint (cid:80) a π d ( a | x ) implies that V equals the logisticvalue function V k θ given in Proposition 2.
Finally, since ( λ k , d k ) are ideal updates, they are primalfeasible.
Hence, we can use the constraint λ k =
Φ ⊺ d k in Equation (11) to obtain Equation (3).
All in all, for any y
=
[ w ⊺ , V ⊺ , θ ⊺ ] ⊺ the analytical-oracle outputs g ( y ; x k ) =
[ λ ⊺ , d ⊺ ] ⊺ with
[Equation]
[Equation]
Note that the derivatives with respect to λ and d differ from the ones in Logistic Q-Learning [10].
Inour case, δ k w , θ depends on both cost weights w and logistic action-value parameters θ .
In addition,δ k w , θ is the reduced Bellman error in the feature space rather than in the high dimensional state-actionspace.
Max oracle.
Since the objective in (2) is convex in x and linear in y , X is convex and compact,and Y is convex, by virtue of Sion’s minimax theorem [108], we can exchange the min and max in
Equation (2).
We then have
[Equation]
Therefore, we get
[Equation]
Proximal point updates via max and analytical oracles.
It remains to prove the closed-formexpressions for π d ⋆ and λ ⋆ given in Equation (3) and Equation (4), respectively.
We start rewritingthe objective of the max-oracle as a function of λ and d .
In particular, we have
[Equation]
The minimizers of the previous expression are characterized via the analytical-oracle .
Inparticular, plugging in the analytical forms for λ , d and V , we obtain
[Equation]
This is the objective of the max-oracle in Proposition 2.
Given that the max-oracle returns( w ⋆ k , θ ⋆k ) , the corresponding primal variables ( d ⋆k , λ ⋆ k ) satisfy ( d ⋆k , λ ⋆ k ) =
g ([ w ⋆k , V θ ⋆k , θ ⋆k ]; x k − 1 ) .
This completes the proof of the first part of Proposition 2.
It remains to show the dual form of the max-oracle objective G k ( w , θ ) .
In particular, we will showthat
[Equation] [Equation]
We first recall that
[Equation]
Then, by taking the maximum over y =
[ w , V , θ ] on both sides and using Sion’s minimax theorem,we get
[Equation]
where in the last equality we used the definition of proximal point update in Equation (2).
Finally, by LP strong duality, we have that max w ⟨ λ k +1 , w ⟩ − ⟨ ρ Φ ( (cid:99) π E ) , w ⟩ =max y ∈Y (cid:68) y , A (cid:2) λ ⊺ k +1 , d ⊺ k +
[Equation]
(cid:3) ⊺ + (cid:98) b (cid:69) .
Hence, we conclude that (16) holds.
F Proof of Proposition 3
Proof of Proposition 3.
From first order optimality conditions for λ ⋆ , we get
[Equation]
all
[Equation] [Equation]
Furthermore, by strong duality, it holds that
[Equation]
By combining Equation (18) and Equation (17), we obtain
[Equation]
[Equation]
We define the regularized cost weights by (cid:101) w ⋆k ≜ w ⋆k + 1 η log λ ⋆ ( i )
λ
k − 1 ( i ) .
Then, we can rewrite
[Equation]
Furthermore, by noticing that 1 α H ( d ⋆k ||
d
k )
= (cid:68) d
⋆k , BV k θ ⋆k − Φ θ ⋆k (cid:69) , and ⟨ λ ⋆k , (cid:101) w ⋆k ⟩ = ⟨ d ⋆k , Φ (cid:101) w ⋆k ⟩
[Equation]
we obtain
[Equation]
We define the span norm as ∥ x ∥ sp = inf c ∈ R ∥ x − c 1 ∥ ∞ .
Then multiplying by Φ from the left, wehave that Φ (cid:101) w ⋆k + γ PV k θ ⋆k − Φ θ ⋆k = c 1 .
Moreover, we can write
V k
[Equation]
⋆k
[Equation]
log (cid:32)(cid:88)
[Equation]
We set ( T V k θ ⋆k )
( s ) ≜ − 1 α log (cid:18)(cid:80)
[Equation]
π
[Equation]
−
[Equation]
a | s ) e − α ( Φ (cid:101) w ⋆k + γ PV k θ ⋆k )( s,a ) (cid:19) .
Note that T is the softBellman operator [109, 93] that is a γ -contraction with respect to ∥·∥ ∞ -norm.
It follows that
[Equation]
Therefore,
[Equation]
k
[Equation]
Φ (cid:101)
[Equation]
≤ 1+log 1 β 1 − γ .
Moreover, using the relation (cid:101) w ⋆k + γ MV k θ ⋆k − θ ⋆k =
c 1 ,we have that
[Equation]
Finally, by equivalence of norms in finite-dimensional vector spaces, we get that there exists M > 0 ,such that ∥ θ ⋆k ∥ ∞ ≤ M ∥ θ ⋆k ∥ sp for all ∥ θ ⋆k ∥ sp ̸ = 0 , that is we exclude the vectors θ ⋆k whose eachelement is equal to a constant.
Note that the constant M is independent of k .
Moreover, it is problemindependent, i.e., it does not depend on the specific MDP instance.
Finally, since ∥ x ∥ sp ≤ ∥ x ∥ ∞ , forany vector x , we conclude that necessarily M ≥ 1 .
G Proof of Theorem 1
We will analyze the proximal point method applied to SPP.
We use a similar error propagationanalysis as in [10].
By Proposition 2, the ideal updates ( θ ⋆k , w ⋆k , π ⋆k , λ ⋆k , d ⋆k ) are given by
[Equation]
where τ k θ ⋆k , w ⋆k is a normalization constant.
By feasibility of the ideal updates we also have λ ⋆k =
Φ ⊺ d ⋆k
On the other hand, the realized updates ( θ k , w k , π k , λ k , d k ) are given by
[Equation]
where τ k θ k , w k is a normalization constant, and the notation ( w k , θ k )
=
arg max w , θ G ϵ k k ( w , θ )
meansthat G k ( w ⋆k , θ ⋆k )
− G k ( w k , θ k ) =
ϵ k .
We start by introducing some auxiliary results
Lemma 1.
For any occupancy measures d 1 , d 2 ∈ F , and for any cost vectors c , c ′ ∈ C , we have:
[Equation]
Proof.
We have that
[Equation]
Corollary 2.
Let d ⋆ = argmin d ∈ F max c ∈C ⟨ d , c ⟩−⟨ µ π E , c ⟩ .
Setting c
=
Φw k , d 1 = d ⋆ , d 2 =
d k ,we ge that ⟨ µ π E − d ⋆ , Φw
[Equation]
⟩ − [Equation] ∈C ⟨ µ
[Equation]
− d
[Equation]
( π E
[Equation]
π d k
[Equation]
− d C ( π E , π d ⋆ ) .
Lemma 2.
It holds that D ( λ ⋆k ||
λ k )
η
[Equation]
Proof.
The proof is analogous to Lemma 1 in [10].
Lemma 3 (First order optimality conditions for G k ) .
For all k ∈ [ K ] , it holds that
[Equation]
Proof.
We start by taking the gradient of G k ( w k , θ k ) with respect to w .
In particular, the partialderivative with respect to the i th component is given by
[Equation]
Therefore,
[Equation]
Then, by using the first-order optimality conditions for a concave function, we have
[Equation]
By replacing the expression for ∇ w G k
w ⋆k , θ ⋆k ) , we obtain
[Equation]
We also need the following auxiliary result.
Lemma 4.
For all k ∈ [ K ] , it holds that
[Equation]
Proof.
By introducing ¯ w ⋆k = arg min w ∈W ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⟩ , and applying triangular inequality,we obtain
[Equation]
Moreover
[Equation]
we have
[Equation]
The first equality holds because the term in w ⋆k is a constant wrt w , the variable of the max.
In thelast inequality follows from (A) being zero as we show next:
[Equation]
Lemma 5 (Lower Bound on feature expectation vectors) .
Let Assumption 2 hold.
We then have( Φ ⊺ d k )( j ) ≥ β for all j ∈ [ m ] .
Proof.
Let e j ∈ R m the vector with zeros everywhere but in position j where it takes the value of 1 .
Then, we observe that
[Equation]
Theorem 3 (Error propagation with empirical expert feature expectation vector) .
Let d ⋆ =argmin d ∈ F max c ∈C ⟨ d , c ⟩ − ⟨ µ π E , c ⟩ , and let λ ⋆ be any state-action occupancy measure suchthat ( λ ⋆ , d ⋆ ) ∈
M Φ .
Moreover, let C ≜ 1 βη
(cid:0)(cid:113) 2 α 1 − γ + √ 8 η (cid:1) + (cid:113) 18 α 1 − γ .
Then, we have that
[Equation]
Proof.
We have that
[Equation]
k = D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ γ M T λ ⋆ , V k θ k ⟩ + ητ k θ k , w k
=
D
( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ − η (1 − γ ) ⟨ ν 0 , V k θ k ⟩ + ητ k θ k , w k
=
D
( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ − η (1 − γ ) ⟨ ν 0 , V k θ k ⟩ + ητ k θ k , w k
=
D
( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ − η G k ( θ k , w k ) − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ − η G k
( θ ⋆k , w ⋆k )
+ ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k ⟩ − D ( λ ⋆k ||
Φ ⊺ d k − 1 ) − η H ( d ⋆k || d k − 1 ) α + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺
d
k − 1 )
+ η ⟨ λ ⋆ , w k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k ⟩ + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺ d k − 1 ) + η ⟨ d ⋆ , Φw k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⋆k ⟩ + η ⟨ d k − d ⋆k , Φw ⋆k ⟩ + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺ d k − 1 ) + η ⟨ d ⋆ , Φw k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⋆ k ⟩ + η ∥
d k − d ⋆k ∥ 1 + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ Using Lemma≤ D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ d ⋆ , Φw k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + min w ∈W
η ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⟩ + 3 η ∥
d k − d ⋆k ∥ 1 + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ .
Therefore
[Equation]
it follows that
[Equation]
Then, by using H ( d ⋆ || d k
[Equation]
= H ( d ⋆ || d k − 1 ) − α ⟨ d
⋆
, Φ θ k − BV k θ k ⟩
[Equation]
we obtain
[Equation]
Summing over iteration indices k and dividing by the total number of iterations K
[Equation]
we obtain
K
[Equation]
Moreover, by a telescoping sum, we get(cid:88)
[Equation]
η
Combining this derivation with (25)
[Equation]
we get
K (cid:88)
[Equation]
In order to bound the term D ( λ ⋆ || Φ ⊺ d k ) − D ( λ ⋆ || λ k )
[Equation]
we introduce the Bregman projection to the space of feature expectation vectors induced by valid occupancy measures ˜ λ k =arg min { λ = Φd | d ∈F} D ( λ || λ k ) .
We then have
[Equation]
where in the second inequality
[Equation]
we used Lemma 11.3 in [110]
[Equation]
Furthermore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
where we used max i λ ⋆ ( i ) Φ ⊺ d k ( i ) ≤ 1 β thanks to Lemma 5 while in the last line we use the fact that
H ( d ⋆k || d k ) is positive and the equality in Lemma 2.
To bound the ℓ 1 -norm, we apply Pinkser’s inequality and Lemma 2 in [10] to get that
[Equation]
Plugging the last derivation in Equation (26) gives
K
[Equation]
Finally, using Lemma 3 we have that the term ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k − w k ⟩ is non positive.
Therefore,
K (cid:88)
[Equation]
K , where C = 1 βη ( (cid:113) 2 α 1 − γ + √ 8 η ) + 3 (cid:113) 2 α 1 − γ .
Finally, we need a Lemma that provides a concentration for the estimated expert feature expectationvector.
Lemma 6 ([27]) .
Let D π E ≜ { ( s ℓ 0 , a ℓ 0 , s ℓ 1 , a ℓ 1 , . . .
, s ℓH , a ℓH ) } n E ℓ =1 ∼ π E be a finite set of i.i.d.
truncated sample trajectories.
We consider the empirical expert feature expectation vector ρ Φ ( (cid:99) π E ) by taking sample averages, i.e.,
[Equation]
Suppose the trajectory length is H ≥
− γ log( 1 ε ) , and the number of of expert trajectories isn E ≥ 2log( 2 mδ ) ε.
Then, with probability at least 1 − δ , it holds that ∥ ρ Φ ( π E ) − ρ Φ ( (cid:99) π E ) ∥ ∞ ≤ ε.
At this point, Theorem 1 is proven from the results of Theorem 3,Lemma 6 and Lemma 1.
H Biased Stochastic Gradients and their Properties
In order to estimate the gradient ∇ θ G ( w , θ ) , we define the policy π k [Equation] s [Equation]
s, [Equation] ) ,for all k ∈ N , and for all θ ∈ R m . Then, by standard computations we get that for all ( w , θ ) , and forall j ∈ [ m ] ,
[Equation]
θ , [Equation] w , θ )
[Equation]
a 0 ) (cid:105) , where B k w , θ ( i ) ≜ exp
[Equation]
[Equation]
Φ
[Equation]
π
[Equation]
[Equation]
i ) , and Γ k
[Equation]
i, j ) ≜ 1005 (cid:80) s ′ ,a ′ M i,s ′ π k − 1 , θ ( a ′ | s ′ )
ϕ
j ( s ′ , a ′ ) , .
Similarly, for the gradient ∇ w G ( w , θ ) , we can write
[Equation]
Note that the following estimators of ∇ θ G k ( w , θ ) and ∇ w G k ( w , θ ) are unbiased: Sample( s ′ , a ′ ) ∼ d k − 1 , i ′ ∼ ϕ
( s ′ , a ′ ) , s 0 ∼ ν 0 , and a 0 ∼ π k − 1 , θ ( ·| s 0 ) , then define
[Equation]
[Equation]
These expressions give rise to the Biased Stochastic Gradient Estimator subroutine (BSGE) givenin Algorithm 2, where we plug-in estimators (cid:98) B k w , θ ∈ R m and (cid:98) Γ k ∈ R m × m to Equations (28)and (29).
It remains to show how to maintain good estimators (cid:98) B k w , θ and (cid:98)
Γ k by using the linear MDP
Assumption 1.
While the estimator (cid:98) Γ k ∈ R m is a standard ridge regression estimator, the constructionof (cid:98) B k w , θ is more involved.
In particular, we first need to build an estimator for the product MV kθ viaridge regression.
Then, the estimator for (cid:98) B k w , θ is derived by plugging-in the estimator of MV kθ , andthe estimator for the feature expectation vector ρ Φ ( π k − 1 ) to equation B k w , θ ( i ) ≜ exp (cid:0) − η δ k w , θ ( i ) (cid:1)
Z k .
The reasoning and analysis is inspired by [111, 65].
H.1 Ridge estimators
This section leverages ridge regression [112] to build estimators (cid:98) B k w , θ and (cid:98) Γ k ∈ R m × m .
We workunder the Assumption 2 which ensures that every iterate covers the features space.
We recall that by
Lemma 5, Assumption 2 implies that Φ ⊺ d k ( s, a ) ≥ β , for all k ∈ [ K ] .
Input: Policy evaluation step k , reference points ( w , θ ) , number of samples N
Run π k − 1 and collect i.i.d.
samples { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′ ( n ) k − 1 ) }
Nn =1 such that ( s ( n ) k − 1 , a ( n ) k − 1 ) ∼ d k − 1 , and s ′ ( n ) k − 1 ∼ P ( ·| s ( n ) k − 1 , a ( n ) k − 1 )
Compute empirical estimators (cid:98) δ k w , θ ∈ R m , (cid:98) Γ k ∈ R m × m , ρ Φ ( (cid:91) π k − 1 ) ∈
R m for i = 1 , . . .
, m do
Compute (cid:98) B k w , θ ( i ) = exp (cid:0) − η (cid:98) δ k w , θ ( i ) (cid:1) (cid:98) Z k , Where (cid:98)
Z k = (cid:80) mi =1 exp (cid:0) − η (cid:98) δ k w , θ ( i ) (cid:1) ρ Φ ( (cid:91)
π k − 1 )( i ) end for
Sample ( s ( N +1)
k − 1 , a ( N +1)
k − 1 ) ∼ d k − 1 , i ( N +1)
k −∼ ϕ ( s ( N +1)
k − 1 , a ( N +1)
k − 1 ) , s (0)
k − 1 ∼ ν 0 , and a (0) k − 1 ∼ π k − 1 , θ ( ·| s 0 ) Compute (cid:98)
[Equation] [Equation]
[Equation] [Equation]
k
[Equation]
= j } (cid:98)
[Equation]
B k w , θ
( i ( N +1)
k − 1 ) (cid:104) γ (cid:98) Γ k
( i ( N +1)
k − 1 , j ) − 1 { i ( N +1) k −= j } (cid:105) + (1 − γ ) ϕ
j ( s (0) k − 1 , a 0 k − 1 )
Output: ( (cid:98) ∇ w G k ( w , θ ) , (cid:98) ∇ θ G k ( w , θ ))
H.1.1 Estimator for MV kθ
We first construct an estimator for M k V kθ .
We can start noticing that we can rewrite M k V kθ usingthe feature covariance matrix ¯ Λ k ≜
E
[Equation]
s,a
[Equation]
(cid:2) ϕ ( s, a ) ϕ ( s, a ) T (cid:3) as showed by the next lemma.
Lemma 7.
It holds that MV kθ = ¯ Λ − 1 k
E
[Equation]
s,
[Equation] [Equation]
(cid:2) ϕ
( s,
[Equation]
s
[Equation]
) (cid:3) .
Proof.
[Equation]
It follows that MV kθ = arg min
z
[Equation]
Now, we move to the problem of estimating (cid:92) MV kθ with a finite amount of environment interactionssampled i.i.d from d k − 1 .
We define
[Equation]
By optimality conditions, we can obtain a closed-form expression for (cid:92) MV kθ .
Lemma 8.
It holds that
[Equation]
where Λ k,N ≜ 1 N (cid:80) Nn =1 ϕ ( s ( n ) k − 1 , a ( n ) k − 1 ) ϕ
( s ( n ) k − 1 , a ( n ) k − 1 ) T is the empirical covariance matrix.
Proof.
Let L ( z ) ≜ 1 N (cid:80)
Nn =1 (cid:16) ϕ
( s ( n ) k − 1 , a ( n ) k − 1 )
T z − V kθ ( s ′ ( n ) k − 1 ) (cid:17) 2 + χ ∥ z ∥ 22 .
The first derivative isgiven by
[Equation]
[Equation]
Since L ( · ) is convex in z , by first-order optimality conditions, we get
[Equation]
The statement follows from rearranging the terms.
Remark 1.
Note that when χ = 0 , and ϕ ( s, a ) is one-hot vector for every ( s, a ) , then we obtain thetabular estimators W v proposed in [65].
We invoke Theorem 2 in [112] to derive an upper bound for (cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2 ¯ Λ k .
Lemma 9.
Fix some χ > 0 and take N ≥ O
( log( mδ ) χβ ) .
Then, with probability at least 1 − δ , we have
[Equation]
N
[Equation]
(cid:18)
[Equation]
δ (cid:19)(cid:19) , where D ≜ M 1+log ( 1 β ) 1 − γ ≥ 1 is the upper bound of (cid:13)(cid:13) V kθ (cid:13)(cid:13) ∞ derived in Proposition 3.
Proof.
We introduce the following auxiliary quantities:
[Equation]
and the conditional expectation¯
MV
[Equation]
N
[Equation]
N
[Equation]
N (cid:88) n
[Equation]
E [Equation] with F n being the filtration F n = { s ( i ) k − 1 , a ( i ) k − 1 } ni =0 .
Then applying the general random designdecomposition in ([112], Proposition 3) we obtain:
[Equation]
where similarly to [112], we define ϵ rg as the ridge error, ϵ bs the ridge estimator bias and with ϵ vr theridge estimator variance.
By choosing N ≥ O (6 ρ 2 χ d 1 ,χ (log max (1 , d 1 ,χ )+log 1 δ ))
=
O ( 1 βχ log mδ ) ,we ensure that the conditions in Theorem 2 in [112] are satisfied.
We next bound each term separately.
Ridge error.
In [112], the bound derived for the ridge error is a function of the regularizationparameter χ , the eigenvalues of the covariance matrix ¯ Λ k denoted as { σ j } mj =1 and the correspondingeigenvectors { v j } mj =1 .
In particular, we have
[Equation]
where in the first inequality we used bullet (3) of Theorem 2 in [112].
Bias.
It holds that
[Equation]
where we used the notation
[Equation]
[Equation]
a
[Equation]
] ≜
[Equation]
Moreover,
[Equation]
Finally, according to Remark 2 in [112], we have that ρ χ is bounded as follows
[Equation]
where the last inequality follows from noticing that d 1 ,χ ≥ βm 1+ χ .
Therefore, we can conclude that:
[Equation]
N
[Equation]
Variance.
From the bullet (5) in [112] it follows that
[Equation]
δ
[Equation]
We have Var (cid:2)
V kθ ( s ′ ) | s, a (cid:3) ≤ (cid:13)(cid:13)
V kθ (cid:13)(cid:13) 2 ∞ ≤ D 2 .
Finally, bounding d 2 ,χ we obtain that
[Equation]
Hence we can conclude
[Equation]
Final bound.
By combining the above bounds with Equation (31), we get the final bound
[Equation]
The bound above is minimized by choosing χ as small as allowed.
This is made precise in the nextcorollary.
Corollary 3.
Let χ =
O ( log mδ βN ) .
With probability at least 1 − δ , it holds that
[Equation]
log (cid:16) m δ (cid:17)(cid:17) 2 + mD
N log (cid:18)δ (cid:19)(cid:19) .
In order to upper bound (cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2
we need the next lemma.
Hence, to bound(cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2 2 , we can directly apply Theorem 2 in [112] that leads to the following lemma.
Lemma 10.
Given a matrix A ∈ R m × m and a vector x ∈ R m , we have that ∥ x ∥
A ≥ λ min ( A ) ∥ x ∥ 2 .
Proof.
We have that A − λ min ( A ) I ≥ 0 that implies x ⊺ Ax ≥ λ min ( A ) x ⊺ x .
Corollary 4.
Let χ =
O ( log mδ βN ) .
With probability at least 1 − δ , it holds that
[Equation]
Corollary 5.
Let χ = O (cid:16) log mδ βN (cid:17) , and N ≥ max (cid:16) γ 2 mD 2 βϵlog(1 /δ ) , γ √ mDβ 3 ϵ log( m/δ ) (cid:17) .
Then, withprobability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) MV kθ − (cid:92)
[Equation]
H.1.2 Estimators for Γ k
Recall that we introduced Γ k ( i, j ) ≜ (cid:80) s ′ ,a ′ M i,s ′
π k − 1 , θ ( a ′ | s ′ )
ϕ
j ( s ′ , a ′ ) .
We can equivalentlyrewrite it as
[Equation]
where the last equality is obtained with manipulations analogous to Lemma 7.
Similarly, We can estimate Γ k
( i, j ) with a finite amount of environment interactions sampled i.i.d.from d k − 1 , by solving the following ridge regression problem:
[Equation]
Lemma 11.
By optimality conditions, we can obtain a closed form for (cid:98) Γ k as
[Equation]
By noting that ∥ h k − 1 ,j ∥ ∞ ≤ 1 for any k , it follows that
Corollary 6.
For χ = O ( log mδ βN ) , with probability at least 1 − δ , it holds that
[Equation]
Corollary 7.
For χ = O ( log mδ βN ) , and N ≥ max (cid:16) O (cid:16) mβϵ 2 log(1 /δ )
(cid:17) , O (cid:16) √ mβ 3 ϵ log( m/δ ) (cid:17)(cid:17) , withprobability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) Γ
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
H.1.3 Estimator for feature expectation vector ρ Φ ( π k − 1 )
The goal is to estimate ρ Φ ( π k − 1 ) .
Consider the sample transitions { s ( n ) k − 1 , a ( n ) k − 1 } Nn =1 ∼ d
Nk − 1 .
Thenwe estimate ρ Φ ( π k − 1 ) =
Φ ⊺ d k − 1 by ρ Φ ( (cid:91)
π k − 1 ) ≜ 1 N (cid:80) Nn
=1 ϕ ( s ( n ) k − 1 , a ( n ) k − 1 ) .
In the next lemma, we provide a useful concentration result.
Lemma 12.
With probability at least 1 − δ , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵ, and for all i ∈ [ m ]simultaneously, it holds that
[Equation]
Proof.
Consider the martingale difference sequence Z i ( n ) = ϕ
[Equation]
s ( n ) k − 1 , a ( n ) k − 1 ) − ρ Φ ( π k − 1 )( i ) withthe variance process V i ( n ) =
(cid:80) nj =1 E (cid:2)
Z 2 i ( j ) |F
j − 1 (cid:3) , where F j − 1 being the filtration up to the stateaction pair ( s ( j ) k − 1 , a ( j ) k − 1 ) .
We have,
V i ( n
[Equation]
The martingale difference sequence Z i ( j ) satisfies the sub- ψ P condition of [111] (see Bennet case intheir Table 3) with constant c = 2 .
Therefore, by Lemma 13 in [65] with m = ρ Φ ( π k − 1 )( i ) , with probability at least 1 − δ 2 m , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵsimultaneously, it holds that
[Equation]
Similarly, with probability at least 1 − δ 2 m , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵsimultaneously,it holds that ρ Φ ( (cid:91) π k − 1 )( i ) ≤ ρ Φ ( π k − 1 )( i ) + 2 .
26 ρ Φ ( π k − 1 )
( i ) Nϵ .
A union bound concludes theproof.
H.1.4 Estimators for (cid:98) B k w , θ
We can directly invoke Lemma 17 in [65] to get guarantees for the estimator (cid:98) B k w , θ ( i ) .
In particular,we obtain the following result.
Lemma 13.
Let (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ
(cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ and (cid:12)(cid:12) ρ Φ ( (cid:91)
π k − 1 )( i ) − ρ Φ ( π k − 1 )
( i ) (cid:12)(cid:12) ≤2 .
26 ϵ ρ Φ ( (cid:91)
π k − 1 )( i ) .
Then, it holds that (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ B k w , θ ( i ) ≤ 38 ηϵβ .
Proof.
First, we notice that (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ implies that (cid:98) δ k w , θ ( i ) − δ k w , θ ( i ) ≤ ϵ .
Therefore,by Lemma 17 in [65] we get (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ B k w , θ ( i ) .
Moreover, it holds that
[Equation]
Therefore,(cid:12)(cid:12)(cid:12)(cid:98)
B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ β , and (cid:98) B k w , θ
( i ) ≤ B k w , θ ( i ) (1 + 38 ηϵ ) ≤ 1 β (1 + 38 ηϵ ) .
Corollary 8.
Let N 1 ≥ max (cid:16) O (cid:16) γ 2 mD 2 βϵlog(2 /δ )
(cid:17) , O (cid:16) γ √ mDβ 3 ϵ log(2 m/δ ) (cid:17)(cid:17) and N 2 ≥1 .
4loglog(2 N 2 ) +log 20 .
8 m δ βϵ.
Then, for χ = O (cid:16) log 2 mδ βN (cid:17) , and for N ≥ max ( N 1 , N 2 ) , with probabilityat least 1 − δ , it holds that (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵβ , for all i ∈ [ m ] .
Proof.
By Corollary 5, we have that with N ≥ N 1 it holds that (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ , withprobability 1 − δ/ 2 .
Furthermore, Lemma 12 gives that for N ≥ 1 . 4loglog(2 N )+log 20 .
8 m δ βϵ, it holdswith probability 1 − δ/ 2 that (cid:12) (cid:12) ρ Φ ( (cid:91)
π k − 1 )( i ) − ρ Φ ( π k − 1 )( i ) (cid:12) (cid:12) ≤ 2 .
26 ϵ ρ Φ ( (cid:91)
π k − 1 )( i ) , for all i ∈ [ m ]simultaneously.
Therefore, a union bound gives that for N ≥ max ( N 1 , N 2 ) , with probability 1 − δ , we have that(cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ , and (cid:12)(cid:12) ρ Φ ( (cid:91)
π k − 1 )( i ) − ρ Φ ( π k − 1 )
( i ) (cid:12)(cid:12) ≤ 2 .
26 ϵ ρ Φ ( (cid:91)
π k − 1 )( i ) , for all i ∈ [ m ] .
An application of Lemma 13 concludes the proof.
H.1.5 Estimators for B k w , θ ( i ) Γ k ( i, j )
We obtain an estimator for B k w , θ ( i ) Γ k
( i, j ) simply as (cid:98) B k w , θ ( i ) (cid:98)
Γ k
( i, j ) .
The next lemma givesguarantees for such an estimator.
Lemma 14.
Assume that for any ( i, j ) ∈ [ m ] 2 , it holds that (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵβ and(cid:12)(cid:12)(cid:12)(cid:98) Γ k
( i, j ) − Γ k
( i, j ) (cid:12)(cid:12)(cid:12) ≤ ϵ .
Then, (cid:12)(cid:12)(cid:12) B k w , θ ( i ) Γ k ( i, j ) − (cid:98) B k w , θ ( i ) (cid:98)
Γ k
( i, j ) (cid:12)(cid:12)(cid:12)
≤ ϵβ (1 + (1 + ϵ )38 η ) , for all( i, j ) ∈ [ m ] 2 .
Proof.
We have that(cid:12)(cid:12)(cid:12) B k w , θ ( i ) Γ k
( i, j ) − (cid:98)
B k w , θ
( i ) (cid:98) Γ k
( i, j ) (cid:12)(cid:12)(cid:12) ≤ B k w , θ ( i ) (cid:12)(cid:12)(cid:12)(cid:98) Γ k
( i, j ) − Γ k
( i, j ) (cid:12)(cid:12)(cid:12) + (cid:98) Γ k
( i, j ) (cid:12)(cid:12)(cid:12)(cid:98)
B k w
[Equation]
where we used the bound ˆ Γ k
( i, j ) ≤ Γ k
( i, j ) + ϵ ≤ 1 + ϵ .
Lemma 15.
For χ = O (cid:16) log mδ βN (cid:17) , choose N ≥ max
[Equation]
with N 1 and N 2 as defined in Corollary 8, then with probability 1 − δ , for all ( i, j ) ∈ [ m ] 2simultaneously:
[Equation]
Proof.
By Corollary 8, when χ = O (cid:16) log mδ βN (cid:17) , and N ≥ max ( N 1 , N 2 ) , it holds with probability atleast 1 − δ that
[Equation]
B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ β , for all i ∈ [ m ] .
Moreover, when N ≥ max (cid:16) O (cid:16) mβϵ 2 log( m/δ ) (cid:17) , O (cid:16) √ mβ 3 ϵ log( m 2 /δ ) (cid:17)(cid:17) , by Corollary 7, with probability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) Γ k ( · , j ) − (cid:98) Γ k ( · , j ) (cid:13)(cid:13)(cid:13) 2 ≤ ϵ , for all j ∈ [ m ] simultaneously.
Finally, a union bound and Lemma 14 give that with probability at least 1 − δ , it holds that(cid:12)(cid:12)(cid:12) B k w , θ ( i ) Γ [Equation]
( i, j ) − (cid:98) B
[Equation]
w , [Equation]
( i ) (cid:98) Γ k
( i, j ) (cid:12)(cid:12)(cid:12)
≤ ϵβ (1 + (1 + ϵ )38 η ) .
H.2 Properties of Stochastic gradients
Lemma 16.
Let N ≥ max (cid:16) N 1 , N 2 , O (cid:16) mβϵ 2 log( m/δ ) (cid:17) , O (cid:16) √ mβ 3 ϵ log( m 2 /δ )
(cid:17)(cid:17) with N 1 and N 2with N 1 and N 2 as defined in Corollary 8.
Then, with probability 1 − δ , the following bounds on thestochastic gradient variance hold simultaneously:
[Equation]
θ
[Equation]
∞
[Equation]
G k
[Equation]
i
[Equation]
N
[Equation]
[Equation]
(cid:104)(cid:98)
[Equation]
(cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Furthermore, with probability at least 1 − δ , the following bounds on the stochastic gradient biashold simultaneously:
[Equation]
Proof.
Variance for gradient wrt θ .
Recall that by definition of the stochastic gradient we have that
[Equation]
It then follows that
[Equation]
Invoking Lemma 15, we have that if N ≥ max (cid:16) N
[Equation]
N
[Equation]
√
[Equation]
ϵ log
[Equation]
m
[Equation]
/δ ) (cid:17)(cid:17) with N 1 and N 2 as defined in Corollary 8, then with probability 1 − δ ,
[Equation]
Similarly, by Corollary 8, for N ≥ max ( N 1 , N 2 ) , we have that with probability 1 − δ ,
[Equation]
Hence, a union bound gives that with probability 1 − δ ,
[Equation]
This implies that
[Equation]
Therefore, by introducing a filtration F N = σ (cid:16) { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′
[Equation]
n
[Equation]
k −
[Equation]
} Nn =1 (cid:17) , and noticing that (cid:98) B k w , θand (cid:98) Γ k are F N -measurable, we get
[Equation] [Equation]
At this point, we can simply notice that
[Equation]
Therefore, with probability 1 − δ , it holds that
[Equation]
Variance for gradient wrt w .
Similarly with Corollary 8, we obtain that if N ≥ max ( N 1 , N 2 ) ,then with probability at least 1 − δ ,
[Equation]
This implies that
[Equation]
Bias for gradient wrt θ .
By using the unbiased estimator (cid:101) ∇ θ ,j G k ( w , θ ) in ??
, we get
[Equation]
∇
[Equation]
By choosing χ and N as in Lemma 15 and Corollary 8, and by a union bound, we have that withprobability 1 − δ ,
[Equation]
Using that (cid:101) ∇ θ ,j G k ( w , θ ) is an unbiased estimator of ∇ θ ,j G k ( w , θ ) , we get
[Equation]
Hence, we have that (cid:13)(cid:13)(cid:13)
[Equation]
∇
[Equation]
G
[Equation]
∇
[Equation]
ϵ ))) .
Bias bound for the gradient wrt w .
Similarly, we can notice that with probability at least 1 − δ , itholds that
[Equation]
Since we have only one non-zero element, and by the unbiasedness of (cid:101) ∇ w ,j G k ( w , θ ) , we get
[Equation]
I Proof of Theorem
We first prove a generalization of the Azuma-Hoeffding inequality (Theorem 3.14 in [113]) that holdswhen the martingale difference sequence is bounded with high probability but not almost surely.
Lemma 17 (Modified Azuma-Hoeffding) .
Let { Y i } ni be a martingale difference sequence adaptedto F i , such that for each i , | Y i | ≤
c i with probability at least 1 − δ 2 .
Then, it holds that
[Equation]
Proof.
Define the events E i =
{ Y i ≤ c i } and the intersection E = ∩ ni =1 { E i } , and notice that 1168 P
[Equation]
We then have the following decomposition:
[Equation]
where in the last step we noticed that under the event E , the martingale difference sequence isbounded almost surely, therefore we can apply the standard Azuma-Hoeffding inequality.
Corollary 9.
Let { Y i } ni be a martingale difference sequence adapted to F i , such that for each i ,
| Y i | ≤ c i with probability at least 1 −
δ 2 .
Then, with probability 1 − δ 1 (with δ 1 > nδ 2 ), it holds that
[Equation]
Proof of Theorem 2.
We fix a policy evaluation step k ∈ [ K ] , i.e., we study the k -th iteration of theouter loop of Algorithm 1.
Similarly to the proof of Lemma 19 in [65], the biased SGD subroutinecan be seen as an inexact gradient ascent scheme with updates [Equation]
[Equation]
with
[Equation] [Equation]
By
Lemma 16, and a union bound, we get that for n ( t ) ≥
[Equation]
log
[Equation] [Equation]
log
[Equation]
) , (cid:17)(cid:111) , with probability at least 1 − δ/ 2 , forall t = 1 , . .
, T simultaneously, it holds that
[Equation] [Equation]
where we used that { ξ t } Tt =1 ∪ { γ } ⊂ (0 , 1) .
Moreover, by Hölder’s inequality, we get
[Equation]
where we used that by the triangle inequality and Proposition 3, it holds that ∥ θ kt − θ ⋆k ∥ ∞ ≤2 M [-
 + [Equation]-]1+ | log β |
[Equation]
− γ ≜ 2 D .
We recall that D ≜ M 1+log ( 1 β ) 1 − γ ≥ 1 .
Since (cid:110)
X k θ ,t ≜ (cid:68)
ϵ k θ ,t , θ kt − θ ⋆k (cid:69) (cid:111) ∞
t =1 and (cid:110)
X k w ,t ≜ (cid:10) ϵ k w ,t , w kt − w ⋆k (cid:11) (cid:111) ∞
t =1 are martingale differences, by using Corollary 9 and a simple union bound, we get that with probability at least 1 − δ/ 2 ,
[Equation]
[Equation]
Furthermore, note that G k is η + α -smooth with respect to the ∥·∥ ∞ -norm, and so by Lemma 12in [65], we can bound the ∥·∥ 1 -norm of its gradients.
In particular, we have
[Equation] [Equation]
This in turn implies that
[Equation]
By smoothness and concavity of the objective G k , we can apply Lemma 9 in [65].
In particular, by
Equations (42)–(51), a union bound, and by summing over t in the bound of Lemma 9 in [65], wehave the following guarantee for our inexact gradient scheme: [Equation] t
[Equation]
( Tmδ ) , (cid:17)(cid:111) , and β t ≤ 2 α + η , then with probabilityat least 1 − δ , it holds that
[Equation]
where
[Equation]
We choose β t =
L √ t , for some constant L .
Then a telescoping sum gives
T (cid:88)
[Equation]
Moreover, (cid:80) T t =1 β t L 1 ≤ 2 L 1 L √ T .
By combining this inequality with Equations (59) and (65), weget that
[Equation]
The optimal choice for L is L = √
+ D 2 2 √
L 1 .
In addition, by setting ξ t = (cid:113) L 1 t , we conclude that
[Equation]
+ D
[Equation]
DL [Equation] (cid:111)(cid:112) L
[Equation]
+
[Equation]
(cid:114)
T log
[Equation]
δ ) . Therefore, by combining Equations (60)–(63) and Equation (68), and by Jensen’s inequality, weget that if n ( t ) ≥ max (cid:16) O (cid:16) γ 2 mDt ( η + α ) 2 β log Tmδ (cid:17) , O (cid:16) mtβ log Tmδ (cid:17)(cid:17) , and β t =
O ( 1 √ t ) , then, withprobability at least 1 − δ , it holds that G k ( w ⋆k , θ ⋆k ) − G k ( w k , θ k ) ≤ O (cid:16) max { η, 1 } mD β √ T (cid:17) .
J Offline imitation learning version
Inspecting Equation (5), one can notice that estimating the empirical logistic Bellman evaluationobjective G k or its gradients requires sampling from d k − 1 .
Hence, the algorithm needs interactionswith the environment at every iteration k .
It is possible to alleviate this requirement, changing thecenter point for the relative entropy.
This is akin to smoothing [43] choosing a convenient centerpoint.
In particular, we replace Equation (2) with the following update:
[Equation]
Note that we have removed the iteration index k , since the offline version does not require to iterativelycollect new samples from the environment.
Changing the reference distribution from Φ ⊺ d k to Φ ⊺ µ π Egives Algorithm 3.
In this case, the logistic Bellman evaluation objective takes the form
[Equation]
Input:
[Table]
[Table]
empirical
{ ( s ( n ) , a ( n ) , s ′ ( n ) ) } Nn =1 with s ( n ) , a ( n ) sampled i.i
d. from µ π E and s ′ ( n )
P ( ·| s ( n ) , a ( n ) ) and compute the empirical offline logistic Bellman error by
(cid:98)
[Equation]
(cid:99)
[Equation]
[Table]
[Equation] [Equation]
[Table]
[Equation]
[Table]
// policy evaluation & cost update
◦ Find an approximate maximizer of the negative empirical logistic Bellman error
[Equation]
policy improvement
Policy update:
[Equation]
Output: Policy π d
The difference with the online variant is that in the first term we have the expert occupancy measureinstead of the occupancy measure induced by the current policy.
We describe the correspondingempirical estimate in Algorithm 3.
Furthermore, we suppress the index k , since the offline algorithmdoes not require multiple iterations.
J.1 Theoretical guarantees for the offline case
With minor modifications of the error propagation analysis given in Theorem 1, one can prove thefollowing result.
Theorem 4.
Under the same assumptions as in Theorem 1, and by choosing α = 1220 (cid:18)
[Equation]
|
[Equation]
(cid:113) 1 − γ 2 ϵ
[Equation]
2 / 3 , we obtain
[Equation] [Equation]
where ϵ 1 is the error in the maximization of the logistic Bellman error, i.e. ϵ = max w ∈W , θ G ( w , θ ) −
G ( w 1 , θ 1 ) and ε is the error in estimating the expert feature expectation vector as in Lemma 6.
Proof.
Following exactly the same steps in the proof of Theorem 1 for the special case of K = 1 , weget
[Equation]
By using the bound ∥ d 1 − d ⋆ 1 ∥ 1 ≤ (cid:113) 2 αϵ 1 1 − γ , we have
[Equation]
Therefore, by choosing α as stated in the theorem we conclude the proof.
Notice that if the expert is nearly optimal, the step size η can be taken small, ensuring low bias in thegradients.
This allows to use the original empirical logistic Bellman error analysis, proposed in [10],where one can control the bias by choosing η appropriately small.
To this end, we need to relate thelogistic bellman error in the feature space to the one in the state-action space.
As we will show, thisintroduces an additional bias of order O ( η ) .
The statement is made precise in Theorem 5.
Thanks tothis result and Theorem 2 in [10], we have that ϵ ≤ (8 + e )
[Equation] [Equation]
where N isthe number of expert transitions in the dataset.
We have the following result.
Corollary 10.
Let C 1 = (cid:16)
[Equation]
and N =˜
O ( mϵ − 6 log (1 /δ )) .
Then, with probability 1 − δ , it holds that
[Equation]
Remark 2.
We notice that the optimal choice of η is smaller as the expert is closely optimal, i.e.
D ( λ ⋆ ||
Φ ⊺ µ π E ) is small.
In this condition, we can use the empirical objective estimator proposedin[10] ensuring small bias.
This means that estimating the objective from sample is feasible in theoffline setting.
It is still an open question if this is viable for the online setting improving the errorpropagation analysis.
Next, we present an important result showing that it is possible to replace the minimization of G , withits counterpart in the state-action space defined as
[Equation]
where we introduced δ S , A w , θ = Φ δ w , θ .
Theorem 5.
Let B ≜ 1 + 2 M 1+ | log β | 1 − γ .
Suppose η is chosen such that ηB ≤ 1 .
Then, it holds that
[Equation]
Proof.
From Proposition 3, we have that ∥ θ ∥ ∞ ≤ M 1+ | log β | 1 − γ and ∥ V θ ∥ ∞ ≤ M 1+ | log β | 1 − γ , for allθ ∈ R m .
It follows that for any ( w , θ ) ∈
W× R m , it holds that ∥ δ θ , w ∥ ∞ =
∥ w + γ MV θ − θ ∥ ∞
≤1 + 2 M 1+ | log β | 1 − γ = B .
Hence, it holds that η ∥ δ θ , w ∥ ∞ ≤ ηB ≤ 1 .
First, we recall the assumptionthat the rows of Φ are probability distributions, i.e., ϕ ( s, a ) ∈ ∆ [ m ] , for all ( s, a ) .
We then have
[Equation]
Moreover, we have
[Equation]
We can then lower bound W as
[Equation]
where the inequality follows by Jensen’s inequality for expectations.
We will now upper bound the term W .
Thanks to the choice of η such that ηB ≤ 1 , we havethat η ≤ 1 B ≤ 1 | δ w , θ ( i ) | , for all i .
Therefore, we can apply the inequality e x ≤ 1 + x + x 2 forx =
− η δ SA w , θ ( i ) ≤ 1 and obtain
[Equation]
where in the third line we used Equation (76), and in the last line we used the inequality 1 − x ≤ e − xfor x = δ SA w , θ ( s, a ) .
By taking expectations with respect to µ π E and logarithms on both sides, we get
[Equation]
Subtracting W yields
[Equation]
where in the third line we used the inequality log(1 + x ) ≤ x for x =
[Equation]
ηB
[Equation]
E ( s,a
[Equation]
∼
[Equation]
πE
[Equation]
e − η δ SA w , θ ( s,a ) (cid:21) , whilein the last line we used that ηB ≤ 1 .
This concludes the proof.
After having established with Theorem 5 that G S , A can be used as biased estimate of G , we canproceed as in [10].
In particular, we maximize the empirical objective (cid:98) G (see Algorithm 3) that is abiased estimate of G S , A ([10, Theorem 2]).
Then, we compute unbiased gradients of (cid:98) G , recurring tothe Donsker-Varadhan formula [114, Corollary 4.15] that implies the following result.
Theorem 6.
Given a batch of expert data { (cid:101) S n , (cid:101) A n , (cid:101) S ′ n } Nn =1 ∼ µ π E × P , the following is true:
[Equation]
77) with:
[Equation]
and the minimum attained at z ⋆ ∝ 1 N e − η
[Equation]
Hence, in the deep learning implementation we update the cost and the value networks backpropagating through S ( θ , w , z ⋆ ) .
J.2 Practical implementation
We test a practical relaxation of Algorithm 3 that uses two separate neural networks for cost andvalue function approximation.
We use a two layers neural network with 128 units per layer with ReLuactivation for the CartPole-v1 environment.
Whereas, for Acrobot-v1 and LunarLander-v2 weused a 3 layers architecture with 64 units per layer.
K Mirror Descent versus Proximal Point
To highlight an important message of our work, in this section, we briefly discuss a mirror descentscheme with alternating updates, and we compare it to our proximal point algorithm in Figure 4.
Notethat in contrast to the classical RL setting, where proximal point and mirror descent coincide becauseof the linear objective, in imitation learning this is not the case.
The updates for the mirror descent scheme involve alternation between updating the occupancymeasure d k and the feature expectation vector λ k in one stage and the cost weights in a second stage.
That is,
[Equation] [Equation]
One can notice that the update in Equation (81) corresponds to one update of Logistic Q -Learning[61].
Therefore, it can be implemented by maximizing the negative logistic Bellman error that isnow a function only of the variable θ and not of both ( θ , w ) as in PPM.
The next proposition is thecounterpart of Proposition 2 for the mirror descent scheme.
Proposition 5.
For a parameter θ ∈ R m , we define the state-action logistic value function Q θ ∈
R |S||A| by Q θ ≜ Φ θ , and the k -step state logistic value function V k θ ∈ R |S| by
[Equation]
Moreover, for a fixed cost c = c w , we define the k -step Bellman error function δ k θ , w by δ k θ , w ≜w + γ MV k θ − θ .
Then, the unique solution of the aforementioned problem is given by
[Equation]
where θ k is the maximizer of the negative k -step logistic Bellman error function
[Equation]
Proposition 5 leads to an actor critic scheme that has three separate and alternating updates: (i) policyupdate stage, (ii) policy evaluation update, and (iii) cost weights update.
Similar actor critic-schemesfor different MDP models, and different policy evaluation objectives (e.g., minimizing the squared
Bellman error) have been also proposed in [11–13].
Contrary to these schemes, in our proximalimitation learning algorithm, the policy evaluation step involves optimization of a single objectiveover both cost and Q -functions.
In this way, we avoid instability or poor convergence in optimizationdue to nested policy evaluation and cost update steps.
In section L.5, we verify numerically that PPM see Figure 4)
[Figure]
We report the results of omitted
L Experiment Details
[Figure]
L.1 Refereences for environments description
[Figure] In the tabular case we used the environments ( DoubleChain [115], SingleChain [115], RiverSwim[116], WideTree [117], Two States Deterministic [118], Two States Stochastic [10] and
WindyGrid [119]).
While for the offline setting, we used the environments CartPole [120],
[Figure]
Acrobot [121] and LunarLander [122].
The curves are averaged over 50 seeds.
For the environments Cartpole and Acrobot , we used a three layer neural network to approximate the value [Figure] function.
In these cases we averaged 5 seeds.
[Figure]
L.2 Hyperparameters
[Figure]
We report the hyperparameters for the tabular online experiments in Table 1 and for the offlineexperiments in Table
Environment n-trajs lr w lr θ η α optimizer
TwoStateStochastic-v  0.
OGD
TwoStateStochastic
[Table]
L.3 On the data sampling
In all the experiments, we perform a relaxation of our theoretical scheme.
In particular, to increase thesample efficiency we sample state action pairs from the Markovian stream of experience.
Analyzingthis setting is an open problem.
[Table] [Table]
As optimizer, we used OGD in all cases.
L.4 Offline experiments setting
We consider a training environment and a test environment with different random seeds.
We trainboth IQLearn and Proximal Point for 2 e 5 environment steps and we evaluate the policy running 10episodes on the evaluation environment every 1 e 3 steps.
We report the maximum evaluation resultachieved at the end of training.
We average the seeds from 0 to 10 for the results shown in Figure 2.
We use two separate instances of the same architecture as function approximation for the Q -valuesand cost respectively.
Finally, since the algorithm operates offline it has no access to the distributionν 0 .
In order to approximate the term ⟨ ν 0 , V ⟩ , we use the Bellman flow constraints and the fact thatthe expert occupancy measure is feasible, i.e. (1 − γ ) ⟨ ν 0 , V ⟩ = (cid:10) µ π E , − γ PV + BV (cid:11) where thelast term can be estimated from the expert samples.
L.5 Comparison with mirror descent
We designed also a mirror descent scheme with alternating updates for imitation learning, brieflydescribed in Appendix K.
The best hyperparameters are given in Table 3.
Furthermore, we show acomparison with our proximal point scheme in Figure 4.
It is interesting to notice that mirror descentand proximal point have been used interchangeably in the RL literature.
Indeed, in that case theobjective is linear therefore the two algorithms coincide.
However, when considering the max-formobjective in imitation learning the equivalence between mirror descent and proximal point does notlonger hold true.
We verify numerically that PPM outperforms mirror descent in simple tabularenvironments (see Figure 4).
L.6 Hyperparameters for Pong (Atari)
We use a convolutional neural network to learn the Q values instead of the linear function approximation class we considered in the theoretical analysis.
We set the parameter α to 1 e − 3 and η to8 e − 2 , we used expert samples to approximate expectation with respect to the initial distribution.
Foroptimizing the network we used Adam [124] with learning rate 1 e − 4 and defaults value for β 1 , β
Instead of hard constraints on the euclidean norm of the elements of W we consider a ℓ 2 penaltyto the loss function.
As expert trajectories we used the dataset released by [34].
This is the onlyhyperparameters configuration we tried using a single seed (using the seed 0 ) on our method becauseof the high computation requirements of this environment.
[Figure]
L.7 Hyperparameters for MuJoCo (continuous control)
[Figure]
The policy network outputs a distribution over continuous action and is parametrized by independentgaussian distributions for every component of the continuous action vector.
We use a three layerneural network to estimate their means and variances.
We used as center point in the divergence
D the expert feature expectation vector.
With further modifications our method can extend alsoto continuous control tasks in MuJoCo [125].
The main challenge is that the policy improvementstep can not be computed in closed form.
We therefore approximate it with a SAC architecture asproposed in [34].
We set α to 1 e − 3 , η to 8 e − 2 , the SAC actor learning rate to 3 e − 5 using
Adam as optimizer using default values of β 1 , β 2 ,for the critic we used again Adam with learningrate 3 e − 4 and default values for β 1 , β 2 .
The actor training of SAC is performed using a transitionbuffer containing expert and learner data in equal proportion.
We used samples from the expert policyto estimate expectations wrt the initial distribution.
We avoid using target networks.
We tested ouralgorithm on both the environment Ant and HalfCheetah using either the data provided in [34] orfresh expert data that we generated training experts with PPO [126].
The results are averaged across5 seeds.
L.8 Acknowledging existing assets and license.
We built on the code and expert data provided in [34].
They are open sourced for academic scopeaccording to their GitHub page https://github.com/Div99/IQ-Learn/blob/main/LICENSE.md .
L.9 On the importance of the dataset
We observed that the performance of our imitation learning algorithm and IQ-Learn can be affectedby the choice of the expert data.
In particular, in Appendix L.9, we show that IQ-Learn works betterwith the expert data provided in [34].
L.10 Hardware
We ran the experiments on our internal cluster.
[Figure]
M Recovered Costs
[Figure]
In this section, we have a closer look at the recovered costs by Algorithm 1.
In Figures 6 and 7,we notice that the recovered cost functions induce a nearly optimal policy with respect to c true .
Inaddition, we try to use the recovered reward in the Gridworld environment to reproduce optimalbehavior under different transition dynamics.
Indeed, we observe that we still obtain an optimalpolicy for c true under different dynamics.
At the same time the imitating policy performs poorlyunder the modified dynamics.
Figure 8 shows the results of this experiment.
Unlike IQ-Learn [34],the advantage of jointly optimizing cost and Q-functions is that we are able to explicitly recover acost function without knowledge of the transition dynamics.
Therefore, the recovered cost functionsshow promising transfer capability to new dynamics.
A unique algorithmic feature of the proposed methodology is that we can explicitly recover a costalong with the Q -function without requiring adversarial training.
In Figures 6 and 7, we visualizeour recovered costs in several simple tabular environments ( River Swim , Single Chain , Double
Chain , and Gridworld , respectively).
Most importantly, we verify that the recovered costs inducenearly optimal policies w.r.t. the unknown true cost function. Compared to IQ-Learn, the we donot require knowledge or further interaction with the environment. Therefore, the recovered costfunctions show promising transfer capability to new dynamics.
Indeed, we experimented with a transfer reward setting on a Gridworld (Figure 8).
We consider twodifferent Gridworld MDP environments, say M and (cid:102) M , with opposite action effects.
This meansthat action Down in (cid:102)
M corresponds to action Left in M and vice versa.
Similarly, the effects of
Up and Right are swapped between (cid:102) M and M .
We denote by V π (cid:102)
M, c true (resp.
V ⋆ (cid:102) M, c true ) the valuefunction of policy π (resp. optimal value function) in the MDP environment (cid:102) M with cost functionc true . Moreover, we denote by π ⋆M, c
the optimal policy in the MDP environment M under costfunction c .
We notice that the recovered cost induces an optimal policy for the new dynamics whilethe imitating policy fails.
Albeit, cost transfer is successful in this experiment we do not expect thisfact to be true in general because we do not tackle the issue of cost shaping [24].
M.1 Preliminary theoretical arguments
We have some preliminary theoretical arguments justifying the near optimality of the recoveredcosts/rewards.
We present briefly the reasoning.
For brevity, we consider the case W
=
B m 1 .
Then π E is optimal for the IL problem.
Moreover, forsimplicity, we consider the case Φ
= I .
Otherwise, in the following derivations, we replace Q -valuesby parameterized Q θ .
Let ( (cid:98) w K , (cid:98) Q K ) be the output (average iterate) of P 2 IL after K outer loop iterations.
We give a sketchof proof that (cid:98) w K converges to an optimal solution to the inverse problem as K → ∞ , i.e., (cid:98) w K [Table]
S t a t e I n d e x 5311(j)
− c K 0.40.20.00.20.41357    (k) - V ⋆ c true 0.40.20.00.20.41357217.5 220.
(l) - V ⋆ c K converges to some w A ∈ W such that π E is optimal for c w A .
To this end, we first introduce thefollowing definition.
Definition 1.
We say that w ∈ W is ε 1 -optimal and ε 2 -feasible for the (Dual) program if-f thereexists V ∈ R |S| , such that
[Equation]
In this case, V ∈ R |S| is called a certificate.
Action index
S t a t e I n
{+d e x      (a) − c true 0 1 
+}
Action index
S t a t e I n d e x      (b) − c K                  (c) - V ⋆ c true                  (d) - V ⋆ c K Note that the definition of ε 1 -optimality for the (Dual) program follows from the fact that thedual optimal value is ζ ⋆ = 0 .
Moreover, in the definition of ε 2 -feasibility we have relaxed thenonnegativity constraint in the dual program (Dual).
We make the following conjecture.
Conjecture: For a sufficiently large number of samples N = O (cid:16) poly (cid:0) 1 ε , log( 1 δ (cid:1) , m ) (cid:17) , with probability at least 1 − δ , the output cost weight (cid:98) w K is ε -optimal and ε -feasible for the (Dual) program,with certificate the corresponding logistic value function V (cid:98) Q K .
This is easy to show for the exact PPM updates, since ( d (cid:98) π K , d (cid:98) π K , (cid:98) w K , V (cid:98) Q K , (cid:98) Q K ) is a saddle-pointof the (SPP).
The proof needs much more effort for the inexact updates used in the sampling-basedalgorithm.
Lemma 18.
Assume that (cid:101) w is ε 1 -optimal and ε 2 -feasible for the (Dual) program.
Then, π E is( ε 1 + ε 2 ) -optimal for c (cid:101) w .
Proof.
There exists (cid:101) V ∈ R |S| , such that
[Equation]
(a
[Figure]
(cid:102)
[Figure]
[Figure]
(b) − V π ⋆M, c true (cid:102) M, c true
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Let (cid:101) π be an optimal policy for c (cid:101) w .
Then, we have that
[Equation]
By using that ( B − γ P ) ⊺ µ (cid:101)
π
= (1 − γ ) ν 0 , we equivalently that
[Equation]
Therefore,
[Equation]
Thus, π E is ( ε 1 + ε 2 ) -optimal for c (cid:101) w .
Claim: As K → ∞ one may approach as closely as desired an optimal solution to the inverseproblem.
Proof for the ideal PPM updates.
We recall that by Proposition 4, the set of such solutions is characterized as the set of w -optimizers to (Dual).
Let (cid:98) V K = V (cid:98) Q K .
By the conjecture, for all K , we have
[Equation]
for some sequence { ε K } ∞ K =1 such that lim K →∞ ε K = 0 .
The sequence { (cid:98) w K } ∞ K =1 ⊂ W is boundedand so there exists a subsequence { (cid:98) w K l } ∞ l =1 , such that lim l →∞ (cid:98) w K l = w A , for some w A ∈ W .
Similarly, by Proposition 3 the sequence { (cid:98) V K l } ∞ l =1 is bounded and so there exists a subsequence{ (cid:98) V K ln } ∞ n =1 , such that lim n →∞ (cid:98) V K
ln = V A , for some V A .
By Equations(92)–(93), we have thatfor all n ∈ N ,
[Equation]
[Equation]
Taking n → ∞ , we end up that
[Equation]
[Equation]
[Equation]
Equivalently,
[Equation]
[Equation]
Therefore, by Proposition 4, π E is optimal for c w A .