On Efﬁcient Online Imitation Learning via Classiﬁcation
Abstract
Imitation learning (IL) is a general learning paradigm for tackling sequential decision-making problems.
Interactive imitation learning, where learners can interactively query for expert demonstrations, has been shown to achieve provably superior sample efﬁciency guarantees compared with its ofﬂine counterpart or reinforcement learning.
In this work, we study classiﬁcation-based online imitation learning (abbrev. COIL) and the fundamental feasibility to design oracle-efﬁcient regret-minimization algorithms in this setting, with a focus on the general nonrealizable case.
We make the following contributions: (1) we show that in the
COIL problem, any proper online learning algorithm cannot guarantee a sublinear regret in general; (2) we propose L OGGER , an improper online learning algorithmic framework, that reduces COIL to online linear optimization , by utilizing a new definition of mixed policy class; (3) we design two oracle-efﬁcient algorithms within the L OGGER framework that enjoy different sample and interaction round complexity tradeoffs, and conduct ﬁnite-sample analyses to show their improvements over naive behavior cloning; (4) we show that under the standard complexitytheoretic assumptions, efﬁcient dynamic regret minimization is infeasible in the
L OGGER framework.
Our work puts classiﬁcation-based online imitation learning, an important IL setup, into a ﬁrmer foundation.
Introduction
Imitation learning (IL), also known as learning from expert demonstrations [47, 44], is a general paradigm for training intelligent behavior for sequential decision making tasks.
IL has been successfully deployed in many applications, such as autonomous driving [47, 45], robot arm control [69], game playing [60], and sequence prediction [23, 8].
It is now well-known that with the help of a demonstrating expert, an IL agent can bypass the exploration challenges of reinforcement learning, achieving a much lower sample requirement than reinforcement learning agents [63].
Two major IL paradigms have been studied in the literature: ofﬂine and interactive.
In ofﬂine
IL [1, 64, 74, 30], the learner receives a set of expert demonstrations ahead of time; in contrast, in interactive IL [23, 56, 55, 32], the learner has the ability to interactively query the expert for demonstrations on states at its disposal, allowing expert feedback to be provided in a targeted manner.
In both settings, the goal of the learner is to output a policy ˆ π that competes with the expert’s policy π E , by consuming as few resources (e.g. expert annotations) as possible.
Between these two, interactive IL is known to be able to achieve superior policy performance than its ofﬂine counterpart under certain favorable assumptions on the expert policy and the environment, in that learning agents can use interaction to address the compounding error challenge [54, 49].
Despite the recent progress in the fundamental limits of the interactive imitation learning in the realizable setting [63, 50, 49], the statistical and computational limits of the interactive imitation learning in the general nonrealizable setting remain open.
One promising and inﬂuential algorithmic framework for studying and analyzing interactive IL in the nonrealizable setting is DA GGER (Data 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Aggregation)
[56], whose key insight is to reduce interactive IL to regret minimization in online learning [59].
Speciﬁcally, it constructs a N -round online learning game, where at every round n ∈ [ N ] , the learner outputs some policy π n from a policy class B , and incurs a loss F n ( π ) ; the loss is carefully constructed so that the learner’s instantaneous loss F n ( π n ) characterizes current policy π n ’s competitiveness compared to the expert π E .
A representative example of F n ( π ) is the expected disagreement between π and π E on the state occupancy distribution induced by π n , used in the original DA GGER paper [56], which can be expressed as the expected zero-one loss of π on a distribution of classiﬁcation examples – we call such setting Classiﬁcation-based Online
Imitation Learning (abbrev.
COIL).
The DA GGER reduction framework has spurred an active line of research on IL [e.g. 55, 63, 15, 18, 17, 38]: it enables conversions from stochastic online optimization algorithms with static or dynamic regret guarantees to IL algorithms with different output policy suboptimality guarantees, allowing the research community to directly translate new results in online learning to those in IL.
Perhaps surprisingly, from a fundamental perspective, rigorous design of efﬁcient regret minimization algorithms for COIL has been largely overlooked by the prior literature.
Speciﬁcally, many works assume a ﬁxed parameterization of policies in B , and assume that F n ( π ) ’s are convex in π ’s underlying parameters to allow for no-regret online convex optimization [e.g. 56, 63, 15].
Although natural, this viewpoint has some issues: (1) in DA GGER ’s reduction, the learner uses ﬁnite-sample approximations of F n ( π ) , which are often discontinuous in π ’s underlying parameters (e.g. given a policy π θ ( s )
= sign( (cid:104) θ, s (cid:105) ) as a linear classiﬁer, its zero-one loss on a state, I ( π θ ( s ) (cid:54) =
π E ( s )) is discontinuous in θ ), making stochastic gradient-based methods inapplicable.
Convex surrogate loss functions has been proposed as a popular workaround [56], but it is well-known that in the nonrealizable setting, even for the special case of supervised learning, minimizing convex surrogate losses can result in very different models compared to minimizing the original zero-one classiﬁcation losses [7]; (2) it makes the usage of policy classes with complex parameterization (e.g. rule-based policies such as decision trees) difﬁcult, as convexity is hard to establish for such classes.
Overview of our results.
In this paper, we bridge the above-mentioned gaps by studying the fundamental feasibility of designing efﬁcient regret minimization algorithms for COIL, putting the study of statistical and computational limits of interactive imitation learning in the general nonrealizable setting into a ﬁrmer foundation.
Our ﬁrst result is that, analogous to Cover’s impossibility result in online classiﬁcation [20], in the COIL setting, any proper online learning algorithm (that outputs a sequence of policies { π n } Nn =1 from the original class B ) cannot guarantee sublinear regret in general (§ 3.1).
The above negative result motivates the design of improper learning algorithms for regret minimization.
To this end, we propose to choose policies from a mixed policy class Π B , and provide an algorithmic framework, L OGGER , that reduces COIL to online linear optimization .
In a nutshell,
L OGGER uses a natural parameterization on Π B that allows to express F n ( π ) as a linear function of the underyling parameters of π ∈ Π B .
We show that any online linear optimization algorithm with (high-probability) regret guarantees can be plugged into L OGGER to obtain an algorithm for COIL with policy suboptimality guarantees (§ 3.2).
Next, enabled by the L OGGER framework, we design computationally efﬁcient algorithms for static regret minimization.
Assuming access to an ofﬂine cost-sensitive classiﬁcation (CSC) oracle O , and a set of unlabeled separator examples for B [66, 24], we design L OGGER -M, a sample and computationally efﬁcient algorithm.
With O (1 /(cid:15) 2 ) interaction rounds and O (1 /(cid:15) 2 ) expert annotations,
L OGGER -M enjoys a per-round static regret of (cid:15) (§ 4.1).
Underlying L OGGER -M is a delicate utilization of the connection between Follow-the-Perturbed-Leader and Follow-the-RegularizedLeader, two well-known online learning algorithm families, ﬁrst observed by [2].
Moreover, by exploiting the predictability of the COIL problem [51, 18, 17], we design an efﬁcient algorithm
L OGGER -ME, that enjoys a per-round static regret of (cid:15) , with O (1 /(cid:15) ) interaction rounds and O (1 /(cid:15) 2 ) expert annotations (§ 4.2).
Its reduced number of interaction rounds can enable a more practical deployment of IL agents, especially when interactive expert annotations come in batches or with delays.
Finally, we study efﬁcient dynamic regret minimization in the L OGGER framework (§ 5).
We show that this is unlikely to be feasible: under a standard complexity-theoretic assumption, no oracleefﬁcient algorithms can output policies in Π B with sublinear dynamic regret.
Due to space constraints, we discuss key related works throughout the paper, and defer additional related works to Appendix A.
Preliminaries
Basic deﬁnitions.
Deﬁne [ n ] := { 1 , . . . , n } .
Deﬁne indicator function I ( · ) such that I ( E )
= 1 if condition E is true, and = 0 otherwise.
We use ∆( W ) to denote the set of probability distributions over a ﬁnite set W , and use Onehot(
w, W ) ∈ ∆( W ) to denote the delta mass on w ∈ W .
For a ﬁnite W , we will oftentimes treat u ∈ R | W | (e.g. u ∈ ∆( W ) ) as a | W | -dimensional vector; for w ∈ W , denote by u [ w ] the w -th coordinate of u .
We abuse the notation of {·} to denote multisets.
Episodic MDPs.
We study imitation learning in episodic Markov decision processes (MDPs).
An episodic MDP M is a tuple ( S , A , H, c, ρ, P ) , where S is a ﬁnite state space (that can be exponentially large), A is a ﬁnite action set, H ∈ N + is the episode length, c : S ×
A → [0 , 1] is the cost function, and ρ ∈ ∆( S ) is the initial state distribution.
Also, P = { P t } H − 1 t =1 denotes M ’s transition dynamics, with P t : S ×
A → ∆( S ) being the transition probability at step t .
Throughout, we use S and A to denote |S| and |A| , respectively.
Without loss of generality, we assume that M is layered, where
S can be partitioned into H disjoint sets {S t } Ht =1 ; the initial distribution ρ is supported on S 1 , and transition distribution P t ( · | s, a ) is supported on S t +1 for all t, s, a .
For state s ∈ S , deﬁne Step( s ) as the step t such that s ∈ S t .
A learning agent interacts with M for one episode using the following protocol: for every step t ∈ [ H ] : it observes a state s t ∈ S t , takes an action a t ∈ A , incurs cost c ( s t , a t ) , and transitions to next state s t +1 ∼ P t ( · | s t , a t ) except for the last step when it stops.
Given a stationary policy π : S → ∆( A ) , we use π ( ·| s ) to denote the action distribution of π on s .
Denote by E π and P π the expectation and probability over executing (i.e. rolling out) policy π in M .
Given policy π , its state occupancy distribution at step t is deﬁned as d tπ ( · ) :
= P π ( s t = · ) ; its average state occupancy distribution is denoted as d π := 1 H (cid:80)
Ht =1 d tπ .
Let J ( π ) :
= E π (cid:104)(cid:80)
Ht =1 c ( s t , a t ) (cid:105) =
H · E s ∼ d π E a ∼ π ( ·| s ) (cid:2) c
( s, a ) (cid:3) denote the expected cumulative cost of π over an episode.
For policy π , we denote its value function V π ( s ) :
=
E (cid:104)(cid:80)
Ht
=Step( s ) c ( s t , a t ) | s, π (cid:105) and action-value function
Q π ( s, a ) :
= c ( s, a ) + E (cid:104)(cid:80)
Ht
=
Step( s )+1 c ( s t , a t ) | s, a, π (cid:105) ; in words, they are the expected costs of rolling out π starting from s and ( s, a ) , respectively.
For policy π , deﬁne its advantage function as
A π ( s, a ) :
= Q π ( s, a ) − V π ( s ) , which measures the expected performance difference by one step deviation of π by taking action a at state s .
Also, we deﬁne the recoverability constant as the ability of π to recover from deviation when rolled out in M :
Deﬁnition 1 ( µ -recoverability) .
A (MDP, policy) pair ( M , π ) is said to be µ
-recoverable
, if ∀ s ∈
S , a ∈ A , (cid:12)(cid:12)
A π ( s, a ) (cid:12)(cid:12) ≤ µ .
Interactive IL.
We study interactive imitation learning [23, 54], where the learner has access to a stationary deterministic demonstrating expert π E and would like to learn a policy with low expected cost.
Throughout, we assume that ( M , π E ) is µ -recoverable, for some µ ≤ H that can possibly be (cid:28) H .
At each interaction round, the learner interacts with M for a few episodes, obtaining trajectories of the form τ =
( s 1 , a 1 , s 2 , a 2 , . .
, s H , a H ) and queries the expert for feedback on some of the states.
Speciﬁcally, given a state s , the feedback given by the expert is of the form ( ζ E ( s, a )) a ∈A ∈ R A .
Two notable examples are: (1) direct expert annotation [56], i.e. given state s , expert provides demonstration π E ( s ) , and we use it to construct an A -dimensional feedback ( ζ E ( s, a )) a ∈A = ( µ · I ( a (cid:54) =
π E ( s )))
a ∈A ; (2) estimates of value functions based on experts’ rollout [55], i.e. ( ζ E ( s, a )) a ∈A =
( A E ( s, a )) a ∈A :=
( A π E ( s, a )) a ∈A .
Throughout, we assume ζ E ( s, a ) satisﬁes ∀ s ∈ S , a ∈ A , A E ( s, a ) ≤ ζ E ( s, a ) ≤ µ · I ( a (cid:54) =
π E ( s )) ; this is satisﬁed by the two examples above.
For a stationary policy π , deﬁne its imitation loss as:
L ( π ) :
=
E s ∼ d π E a ∼ π ( ·| s ) (cid:2) ζ E
( s, a ) (cid:3) .
By the performance difference lemma [34] (see Lemma 55 in
Appendix H), J ( π ) − J ( π E )
≤ H · L ( π ) , implying that if π has a small imitation loss, it will have expected cost competitive with π E .
In light of this connection, in interactive IL, the learner would like to obtain policy ˆ π with low L (ˆ π ) .
Subject to this, throughout the paper, we consider optimizing two measures of data efﬁciency: for n = 1 , . . .
, N do
Learner outputs policy π n .
Loss function F n ( π ) :
=
E s ∼ d πn E a ∼ π ( ·| s ) (cid:2) ζ E
( s, a ) (cid:3) .
Learner draws samples from D Eπ n to obtain information about loss F n , via interacting with M and querying the expert for annotation ζ E .
end for
Goal of learner: minimize (cid:80) Nn
=1 F n ( π n ) =
(cid:80) Nn =1 L ( π n ) .
• Sample complexity : the total number of expert annotations ζ E requested.
A smaller sample complexity reduces the total cost of expert annotations (which often takes human effort).
• Interaction round complexity : total number of adaptive interaction rounds.
A small number of interaction rounds enables more parallelized annotations within an interaction round, and mitigates issues of annotation delay [72, 68].
The DA GGER reduction framework for interactive IL.
The DA GGER framework reduces minimizing L ( π ) to no-regret online learning [56, 55].
It constructs a N -round online learning game, where at every round n ∈ [ N ] , the learner outputs some policy π n , which induces a loss function
F n ( π ) =
E s ∼ d πn E a ∼ π ( ·| s ) (cid:2) ζ E
( s, a ) (cid:3) .
Its key insight is that, by the deﬁnition of { F n } Nn =1 , minimizing the online learning cumulative loss (cid:80) Nn =1 F n ( π n ) is equivalent to minimizing the cumulative imitation losses of π n ’s, i.e. (cid:80)
Nn =1 L ( π n ) .
Research efforts in online learning [59, 43] have mainly focused on the design of algorithms that can output { π n } Nn =1 with static regret SReg N ( B ) or dynamic regret DReg N ( B ) against some benchmark policy class B , formally:
SReg N ( B ) :=
N
[Equation]
Assuming that the learner chooses policies { π n } Nn =1 from another stationary policy class B 0 (which may or may not be B ), the following proposition shows that static and dynamic regret guarantees in the induced online learning game can be converted to policy suboptimality guarantees:
Proposition 2 (e.g. [15]) .
For any N ∈ N + and online learner that outputs { π n } Nn =1 ∈ B N 0 , deﬁne
Bias( B , B 0 , N ) :=
[Equation]
where ¯ d N
[Equation]
= 1 N (cid:80) Nn =1 d υ n .
Then, choosing ˆ π uniformly at random from { π n } Nn =1 has guarantee:
[Equation]
In the above proposition, Bias( B , B 0 , N ) takes the worst-case mixture of state occupancy distributions { d υ n } Nn =1 induced by N policies from B 0 , and measures the expected disagreement between π E and its best approximating policy in B .
Informally, it measures the “approximation error” of benchmark class B : it is always nonnegative, and in the special case of π E ∈ B (which we call the realizable case) , Bias( B , B 0 , N ) = 0 .
Proposition 2 gives two ways to obtain a competitive imitation policy: (1) choose ( B , B 0 ) with a small Bias( B , B 0 , N ) and achieve a low static regret; (2) choose ( B , B 0 ) with a small Bias( B , B 0 , 1) and achieve a low dynamic regret.
Although achieving low dynamic regret can be signiﬁcantly more challenging than achieving low static regret, minimizing dynamic regret has the advantage that its approximation error term Bias( B , B 0 , 1) is smaller than the static regret formulation’s counterpart
Bias( B , B 0 , N ) .
Classiﬁcation-based Online Imitation Learning (COIL).
As we consider a ﬁnite action space
A , a policy π can be equivalently viewed as a (possibly randomized) multiclass classiﬁer.
A costsensitive classiﬁcation (CSC) example is deﬁned to be a pair ( x,(cid:126)c ) , where x ∈ S is its feature part, and (
cid:126)c ∈ R
A is its cost part.
Under the DA GGER reduction framework, the loss at iteration n , F n ( π ) =
E s ∼ d πn E a ∼ π ( ·| s ) (cid:2) ζ E
( s, a ) (cid:3) , can be viewed as the expected cost of policy π on a distribution of cost-sensitive examples D Eπ n (formally, E ( s,(cid:126)c )
∼D Eπn (cid:2) (cid:126)c ( h ( s )) (cid:3) ), where a sample ( s,(cid:126)c ) is drawn from D Eπ by ﬁrst rolling out π and drawing s ∼ d π , and query the expert on s to obtain
( ζ E ( s, a )) a ∈A as its associated (cid:126)c .
The learner can obtain a ﬁnite-sample approximation to F n ( π ) by interacting with M and the expert to draw samples from D Eπ n .
We will focus on designing efﬁcient regret-minimizing algorithms in the COIL setting; see Protocol 1 for a summary.
In addition to data efﬁciency, we also consider the design of imitation learning algorithms with computational efﬁciency guarantees.
To this end, following a sequence of empirically and theoretically successful works on oracle-efﬁcient learning [66, 25, 5, 21], we assume access to the benchmark policy class B , a collection of B stationary and deterministic policies h : S → A , via the following computational oracle:
Deﬁnition 3 (CSC oracle) .
A CSC oracle O for policy class B is such that: given any input multiset of cost-sensitive examples D
= { ( x 1 ,(cid:126)c 1 ) , . . .
, ( x K ,(cid:126)c K ) } ∈ ( S × R A ) K , it outputs the policy in B that has the smallest empirical cost, formally,
[Equation]
where we slightly abuse the notation and and use D to also denote the uniform distribution over it.
We measure an algorithm’s computational efﬁciency by its number of calls to oracle O .
L OGGER : reducing COIL to online linear optimization
In this section, we introduce our main algorithmic framework, L OGGER (an abbreviation for Linear lOss aGGrEgation) for designing regret-minimizing algorithms in the COIL setting.
Section 3.
motivates our approach by showing that natural proper learning-based approaches fail to achieve sublinear regret in general; Section 3.2 introduces our approach of performing improper learning using a carefully-deﬁned mixture policy class, via a reduction to online linear optimization.
3.1 Can we achieve sublinear regret using proper learning?
A natural idea for minimizing regret is proper learning : at round n , the learner chooses some policy π n (possibly at random) from B 0 = B , our benchmark policy class, using some online learning algorithm, based on the information collected in the ﬁrst n − 1 rounds; the learner then collects information on F n via rollouts of π n and expert annotations, and continue to the next iteration.
While this approach has demonstrated sharp online regret guarantees in classical online cost-sensitive classiﬁcation settings [40, 27], perhaps subtly, we show in the following theorem that, this approach is insufﬁcient to guarantee sublinear regret in the COIL setting.
Theorem 4.
Suppose the expert’s feedback ζ E ( s, a ) is of the form µ · I ( a (cid:54) =
π E ( s )) or A E ( s, a ) .
Then, for any H ≥ 3 , there exists an MDP M of episode length H , a deterministic expert policy π E , a benchmark policy class B , such that for any learner that sequentially (and possibly at random) generates a sequence of policies { π n } Nn =1 ∈ B N , its static regret satisﬁes SReg N ( B )
=
Ω( N ) .
The proof of the theorem can be found at Appendix D.1.
Its key insight is that, distinct from the classical online CSC setting, in COIL, the loss at round n , F n , depends on the policy chosen at that round π n , making standard regret minimization results in online classiﬁcation [40, 27] inapplicable.
In more detail, we construct a ﬁxed MDP that “act adversarially” to policies in B , such that any π n ∈ B has F n ( π n ) ≥ H − 1 H , whereas min π ∈B (cid:80) Nn =1 F n ( π ) ≤ N 2 .
Our theorem is similar in spirit to Cover’s impossibility result in online classiﬁcation [20], which shows that an adversary that adapts to the randomness of the learner at each round can force the learner to suffer linear regret.
3.2 A new hypothesis class and the L OGGER algorithmic framework
To sidestep the impossibility result in Theorem 4, we apply the “convexiﬁcation by randomization” technique in online convex optimization [59] by improper learning on a mixed policy class, deﬁned below.
Deﬁnition 5 (Mixed policy class) .
Given policy class B , deﬁne its induced mixed policy class
[Equation]
We slightly abuse notation and use h ( · | s ) ∈ ∆( A ) to denote the delta mass on h ( s ) ∈ A .
At a cursory glance, choosing a policy from Π B seems equivalent to choosing some policy at random from B , which also falls into the failure mode of proper learning (Theorem 4).
We remark that this is not true: rolling out a policy from π u ∈ Π B is equivalent to drawing new policies in an i.i.d.
fashion from B at every step of the episode instead.
As we will see next, the usage of Π B enables the design of COIL algorithms with sublinear regret.
Our key observation is that with the learner outputting policies from the mixed policy class Π B , online regret minimization in IL becomes an online linear optimization problem.
Recall that in online
IL, the loss at round n is F n ( π ) =
E s ∼ d πn E a ∼ π ( ·| s ) (cid:2) ζ E
( s, a ) (cid:3) .
By choosing π n = π u n ∈ Π B and π = π u ∈ Π B for u n , u ∈ ∆( B ) , F n ( π u ) can be viewed as a linear function of u :
F
[Equation]
=
[Equation] [Equation]
u
[Equation]
, where θ ( v ) :
= (cid:16) E s ∼ d πv E a ∼ h ( ·| s ) (cid:2) ζ E ( s, a ) (cid:3)(cid:17) h ∈B .
We have
N (cid:88) n
[Equation]
) , and therefore, minimzing the static regret SReg N ( B ) is equivalent to minimizing the static regret in the online linear optimization problem with losses (cid:110) u
(cid:55)→ (cid:10) θ ( u n ) , u (cid:11)(cid:111)
[Equation]
This motivates L OGGER (Algorithm 2), our main algorithmic framework.
Given input an online linear optimization algorithm OLOA and sample size K , L OGGER outputs policy sequence { π n } Nn =1 .
Speciﬁcally, at round n , L OGGER calls OLOA to perform online linear optimization with respect to linear losses { u (
cid:55)→ (cid:104) g
i , u (cid:105)} n − 1 i =1 and obtains u n ∈ ∆( B ) , which corresponds to a policy π n ∈ Π B (line 3); here for every i , g i is an unbiased estimator of θ ( u i ) .
It then rolls out π n in M for K times to obtain K samples iid from d π n , queries the expert on each sample s to obtain ( ζ E ( s, a )) a ∈A as its associated (cid:126)c , and constructs dataset D n = (cid:8) ( s,(cid:126)c ) (cid:9) (line 4).
Finally L OGGER computes the empirical loss of policies on D n , i.e. g n = (cid:16) E ( s,(cid:126)c ) ∼ D n E a ∼ h ( ·| s ) (cid:2) (cid:126)c ( a ) (cid:3) (cid:17) h ∈B (line 5).
Comparison to prior works.
[16] considers a general online convex optimization formulation for online IL, dubbed “continuous online learning”; our loss function (cid:10) θ ( u ) , · (cid:11) can be viewed as an instantiation of the loss function f u ( · ) therein.
However, their regret minimization results assume that f u ( · ) is strongly convex, which do not cover our COIL setting where f u ( · ) is linear.
Deﬁne LReg N := (cid:80) Nn
=1 (cid:104) g n , u n (cid:105) − min u ∈ ∆( B ) (cid:80)
Nn =1 (cid:104) g n , u (cid:105) as OLOA ’s static regret with respect to {(cid:104) g n , ·(cid:105)}
Nn =1 .
We have the following proposition that links LReg N to SReg N ( B ) , the static regret of { π n } Nn =1 in the online IL problem.
Proposition 6.
For any δ ∈ (0 , 1] , if L OGGER uses some OLOA that outputs { u n } Nn =1 ⊂ ∆( B ) N such that with probability at least 1 − δ/ 3 , LReg N ≤ Reg( N ) .
Then, with probability at least 1 − δ , its output policies { π n } Nn =1 satisfy SReg N ( B )
≤ Reg( N ) + O (cid:18) µ (cid:113) N
[Equation]
K (cid:19) .
Proposition 6 shows that Algorithm 2 is a regret-preserving reduction from online IL to online linear optimization over ∆( B ) , a B -dimensional probability simplex.
The latter is well-known as the “prediction with expert advice” problem [27] (abbrev.
expert problem), where algorithms with different guarantees abound, such as Follow the Regularized Leader (FTRL),
Hedge [27] and its adaptive and optimistic variants [e.g. 61, Section 1], many of which have optimal worst-case regret bounds Reg( N ) =
O (cid:16)(cid:112) N ln( B ) (cid:17) .
Instantiating Algorithm 2 with OLOA set as these algorithms, we obtain a family of online IL algorithms with expected regret of order O ( √ N ) .
Although satisfying from a statistical efﬁciency perspective, such online IL algorithms suffers from computational inefﬁciency: they require explicit calculation of g n and maintenance of u n , which are B -dimensional (dense) vectors whose entries need to be updated separately.
For instance, when
Hedge is chosen as OLOA , u n [ h ] ∝ exp( − η (cid:80) n − 1 i =1 g n [ h ])) for all h ∈ B , which naively requires
O ( B ) time per round to maintain.
To address this computational efﬁciency issue, in the next section, we exploit the cost-sensitive classiﬁcation nature of the COIL problem to design sublinear-regret algorithms that use implicit representations of g n ’s, i.e. D n ’s, that enjoy oracle-efﬁciency guarantees.
Efﬁcient algorithms with static regret guarantees
Using the L OGGER framework, in this section, we propose two oracle-efﬁcient COIL algorithms that have sublinear static regret guarantees against policy class B , in Subsections 4.1 and 4.2 respectively.
4. L OGGER -M: an efﬁcient algorithm with O ( √ N ) static regret
The L OGGER reduction framework calls for a computationally and statistically efﬁcient OLOA , which, if devised, yields an computationally and statistically efﬁcient online imitation learner.
However, when viewed as a general adversarial online learning problem, computational hardness results [29] suggest that, even with access to classiﬁcation oracle O , a prohibitive Ω( √ B ) time complexity is necessary for sublinear regret.
Therefore, in subsequent sections, we adopt an assumption on B in [66], which, to the best of our knowledge, is the state-of-the-art weakest assumption that allows the design of oracle-efﬁcient online CSC algorithms in the adversarial setting:
Assumption 1 (Small separator set) .
There exists a set X ⊂ S (called the separator set) such that, for every pair of distinct policies h, h (cid:48)
∈ B , ∃ x ∈ X , such that h ( x ) (cid:54) =
h (cid:48)
( x ) .
Denote by X := |X| .
Technical challenges.
Even under the small separator set assumption, the design of low-regret oracle-efﬁcient algorithms for imitation learning still remains nontrivial.
A naive application of existing oracle-efﬁcient online CSC algorithms, such as Contextual Follow the Perturbed Leader (CFTPL) [66, 24], still falls into the failure mode of proper learning (Theorem 4 in Section 3.1), where an Ω( N ) regret lower bound is unavoidable in the worst case.
This is in sharp contrast to the classical online CSC setting, where CFTPL enjoys a O ( √ N ) regret [66, 24].
To recap, at round n , CFTPL ﬁrst constructs a random set of “hallucinated” cost-sensitive examples Z based on the separator set X ; it subsequently calls the CSC oracle O on the union of Z and the accumulated dataset ∪ n − 1 i =1 D i to obtain policy π n ∈ B .
CFTPL achieves computational efﬁciency by operating on
D n ’s, an implicit representation of g n ’s, the linear losses of the underlying OLO problem.
Our approach.
The above difﬁculty motivates the need of a new algorithmic approach for efﬁcient classiﬁcation-based online IL.
In view of Section 3.2’s observation that FTRL approaches enjoy a sublinear regret, we ask the question: is it possible to perform FTRL in an oracle-efﬁcient manner?
A positive answer will simultaneously address the computational and statistical challenges of COIL.
We answer this question in the afﬁrmative, by utilizing a connection between FTRL and FTPL ﬁrst observed in [2]: an in-expectation version of FTPL can be viewed as an FTRL algorithm.
Using this observation, we design Algorithm 3, namely Mixed CFTPL (abbrev.
MFTPL), which mimics
FTRL by approximating the in-expectation version of CFTPL in an oracle-efﬁcient manner.
Similar to CFTPL, MFTPL keeps g n implicitly in D n , and calls the CSC oracle.
Different from CFTPL,
MFTPL runs the oracle-call step in CFTPL for T times and outputs the uniform mixture of the T policies.
We refer to T as Algorithm 3’s sparsiﬁcation parameter , due to the algorithm’s resemblance to Maurey’s sparsiﬁcation [46].
Draw
Speciﬁcally, at each iteration j ∈ [ T ] , MFTPL ﬁrst draws ( (cid:96) x,j ( a )) a ∈A ∼ N (0 , I A ) iid for each x in the separator set X , where I A is the identity matrix of dimension A (line 3).
It then constructs a perturbation set of cost-sensitive examples Z j = (cid:110)
( x, Kη (cid:96) x,j ) : x ∈ X (cid:111) that contains each x within the separator set and Kη (cid:96) x,j as associated (cid:126)c , where K accounts for the adjustment on dataset size and η accounts for FTRL’s learning rate (line 4).
It then calls the oracle O with datasets ∪ n − 1 i =
1 D i accumulated so far, together with perturbation examples Z j to obtain an empirical cost minimizer h ∈ B .
This h is represented by a one-hot vector u n,j ∈ ∆( B ) that has weight 1 on the h -th coordinate and 0 elsewhere (line 5).
Finally, after T iterations, MFTPL returns the mean value u n = 1 T (cid:80)
Tj =1 u n,j (line 7).
MFTPL guarantees that:
Lemma 7.
There exists some strongly convex function R : ∆( B ) → R , such that the following holds.
Suppose MFTPL receives datasets { D i } n − 1 i =1 , separator set X , learning rate η , sparsiﬁcation parameter T .
Then, ∀ δ ∈ (0 , 1] , with probability at least 1 − δ , MFTPL makes T calls to the cost-sensitive oracle O , and outputs u n ∈ ∆( B ) such that
[Equation]
with u ∗ n := argmin u ∈ ∆( B ) (cid:16) (cid:104) η (cid:80) n − 1 i =1 g i , u (cid:105) + R ( u ) (cid:17) .
Therefore, by setting T = Ω
( A ln( S )) , the policy π n induced by the MFTPL’s output u n closely mimics π u ∗ n , a policy induced by the FTRL output u ∗ n .
The mild ln( · ) dependence on S makes the lemma useful in large-state-space settings.
Speciﬁcally, a naive attempt to show Lemma 7 is to establish the (cid:107) · (cid:107) 1 closeness of u n and u ∗ n , given that T · u n ∼ Multinomial
( T, u ∗ n ) .
This unavoidably carries an impractical concentration factor of O ( (cid:112) B/T ) , as the bound requires T to be Ω( B ) to be non-vacuous.
We circumvent this challenge by directly showing the closeness of the action distributions π u n and π u ∗ n for all states.
We defer the full version of the lemma, including an explicit form of R , to Appendix E.1.
Lemma 8.
For any δ ∈ (0 , 1] , MFTPL , if called for N rounds, with input learning rate η = 1 µ √ NA (cid:16) ln( B ) X (cid:17) 14 and sparsiﬁcation parameter T =
N ln(2 NS/δ ) √ X 3 ln( B ) , outputs a sequence { u n } Nn =1 , such that with probability at least 1 − δ :
[Equation]
Composing L OGGER with MFTPL, we obtain an efﬁcient online IL algorithm, L OGGER -M. Its regret guarantees immediately follow from combining Lemma 8 with Proposition 6:
Theorem 9.
For any δ ∈ (0 , 1] , L OGGER -M , with K = 1 and MFTPL setting its parameters as in Lemma 8, is such that: (1) with probability at least 1 − δ , its output { π n } Nn =1 satisﬁes:
SReg N ( B ) ≤ O (cid:16) µ (cid:112) NA ln(1 /δ )
( X 3 ln( B )) 14 (cid:17) ; (2) it queries N annotations from the expert; (3) it calls the CSC oracle O for N 2 ln(6 NS/δ ) √ X 3 ln( B ) times.
4. L OGGER -ME: an efﬁcient algorithm with O (1) static regret
Although L OGGER -M is oracle-efﬁcient, it is unclear whether its O ( √ N ) regret guarantee is optimal.
As a lower regret can translate to lower sample and interaction round complexity guarantees, it is desirable to design algorithms with regret as low as possible.
A key observation from prior works [e.g. 15, 16, 38] is that, online IL is a predictable online learning problem [19, 51], and is thus not completely adversarial.
This opens up possibilities to bypass the
O ( √ N ) worst-case regret barrier.
Speciﬁcally, in the L OGGER framework, the coefﬁcient of the linear loss at round n , θ ( u n ) , depends continuously on u n ; more concretely, we can show:
Lemma 10.
For u,
[Equation]
∆( B ) ,
[Equation]
This property and its variants, termed distributional continuity , has been utilized in many online
IL algorithms to achieve sharper regret guarantees.
These works additionally exploit the strong convexity on the loss functions F n ( π ) [15, 16, 38], or use some external predictive model that can predict ∇ F n ( π ) well [18, 17].
Unfortunately, in our COIL setting, neither is the loss function
F n ( π u ) =
(cid:104) θ ( u n ) , u (cid:105) strongly convex in the policy parameter u , nor do we have access to an external predictive model, rendering these approaches inapplicable.
We get around these challenges and design an oracle-efﬁcient algorithm, namely MFTPL-EG (where
EG stands for extra-gradient), with O (1) regret for the online linear optimization problem, which, when composed with the L OGGER framework, yields the L OGGER -ME algorithm with O (1) regret in the COIL setting.
MFTPL-EG is largely inspired by the predictor-corrector framework for policy optimization [17] and extragradient methods in smooth optimization [42, 33]; its details can be found in Appendix E.2.
Its key insight is that, although we do not have a predictive model for θ ( u n ) , we can use an extra round of interaction with M and expert annotations to obtain a good estimate of it.
Based on this, we derive an online linear optimization regret guarantee of MFTPL-EG, deferred to
Appendix E.2.
This immediately implies the following guarantee of L OGGER -ME:
Theorem 11.
For any δ ∈ (0 , 1] , L OGGER -ME , with K and MFTPL-EG ’s parameters set appropriately, is such that: (1) with probability at least 1 − δ , its output { π n } Nn =1 satisﬁes:
SReg N ( B ) ≤ O ( µHA (cid:112) X 3 ln( B )) ; (2) it queries O (cid:18) N
[Equation]
( NB
[Equation] [Equation]
B ) (cid:19) annotations from the expert; (3) it calls the CSC oracle O for O (cid:16) N 3 ln( NS/δ ) µHAX 3 ln( B ) (cid:17) times.
Discussion and comparison.
We now compare the guarantees of L OGGER -M, L OGGER -ME, and the baseline of behavior cloning, where the learner simply draws iid examples from D Eπ E and perform empirical risk minimization over B to learn a policy ˆ π .
All algorithms’ output policy suboptimality guarantees have the following decomposition:
[Equation]
where ApproxErr measures the approximation error of the policy class B to the expert policy π E , and
EstimErr is an estimation error term that vanishes with the number of expert annotation examples and iterations increasing.
By Proposition 2, for L OGGER -M and L OGGER -ME, their ApproxErr terms are both µH · Bias( B , Π B , N ) .
For their EstimErr = H · SReg N ( B ) N , we use Theorems 9 andto calculate the minimum total numbers of interaction rounds I ( (cid:15) ) , expert annotations A ( (cid:15) ) , and oracle calls C ( (cid:15) ) , so that EstimErr is at most (cid:15) with probability at least 1 − δ .
As presented in
Table 1, L OGGER -ME has the same sample complexity order as L OGGER -M, but has a much lower interaction
round complexity ( µH 2 /(cid:15) vs.
µ 2 H 2 /(cid:15) 2 ).
On the other hand, by standard ERM analysis [58] and conversion from supervised learning to imitation learning guarantees ([65, 34]), behavior cloning on B using K samples outputs a policy ˆ π , such that Equation (2) holds with ApproxErr =
H 2 · Bias( B , { π E } , 1) , and EstimErr =
H 2 (cid:112) 2 ln(2 B/δ ) /K
(see Appendix E.3 for a detailed derivation), where Bias( B , { π E } , 1) =
min h ∈B
E s ∼ d πE (cid:2)
I ( h ( s ) (cid:54) =
π E ( s )) (cid:3) .
We also summarize behavior cloning’s performance guarantees in Table 1.
Compared with the two interactive IL algorithms above, behavior cloning requires only one interaction round and one call to the oracle
O , however its ApproxErr has a larger coefﬁcient on the optimal classiﬁcation loss ( H 2 vs. µH ), and needs more expert annotations ( H 4 /(cid:15) 2 vs. H 2 µ 2 /(cid:15) 2 ) to achieve approximation error smaller than (cid:15) with probability at least 1 − δ .
Algorithm
ApproxErr
I ( (cid:15) )
A ( (cid:15) )
C ( (cid:15) )
L OGGER -M µH · Bias( B , Π B , N ) ˜ O
( µ 2 H 2 (cid:15) 2 ) ˜ O ( µ 2 H 2 (cid:15) 2 ) ˜ O
( µ 4 H 4 (cid:15) 4 )
L OGGER -ME µH · Bias( B , Π B , N ) ˜
O ( µH 2 (cid:15) )
˜ O ( µ 2 H 2 (cid:15) 2 ) ˜ O
( µ 2 H 5 (cid:15) 3 )
[Table]
H 2 · Bias( B , { π E } , 1) ˜
O ( H 4 (cid:15) 2 )
Computational hardness of sublinear dynamic regret guarantees
Finally, we study dynamic regret minimization for COIL in the L OGGER framework.
Although in the abstract continuous online learning setup, dynamic regret minimization has recently been shown to be computationally hard [16], given the peculiar linear loss structure of the L OGGER framework, the computational tractability of dynamic regret minimization within this framework still remains open.
We ﬁll this gap by showing that, under a standard complexity-theoretic assumption (that PPADcomplete problems do not admit randomized polynomial-time algorithms), there do not exist polynomial-time algorithms that achieve sublinear dynamic regret in COIL.
Speciﬁcally, we have:
Theorem 12.
Fix γ > 0 , if there exist a COIL algorithm such that for any M and expert π E , it interacts with M , CSC oracle O , expert feedback ζ E ( s, a ) =
A E ( s, a ) , and outputs a sequence of { π n } Nn =1 ∈ Π N B s.t.
with probability at least 1 / 2 ,
[Equation]
in poly( N, S, A, B ) time, then all problems in PPAD are solvable in randomized polynomial time.
The key insight behind our proof of Theorem 12 is that, achieving a sublinear dynamic regret in the
COIL setup is at least as hard as ﬁnding an approximate Nash equilibrium in a two-player general-sum game, a well-known PPAD -complete problem [13].
To establish a reduction from a two-player general-sum game to a COIL problem, we carefully construct a tree-structured MDP whose “leaf states” consist of two major groups: one group has costs encoding the two players’ payoffs, and the other group has a large constant cost, ensuring that any policy in Π B with small dynamic regret encodes near-optimal strategies of both players.
We refer the readers to Appendix F for details.
Conclusion
In this work, we investigate the fundamental statistical and computational limits of classiﬁcationbased online imitation learning (COIL).
On the positive side, we propose the L OGGER framework that enables the design of oracle and regret efﬁcient COIL algorithms with different sample and interaction round complexity tradeoffs, outperforming the behavior cloning baseline.
On the negative side, we establish impossibility results for sublinear static regret using proper learning in the COIL setting.
We also show the computational hardness of sublinear dynamic regret guarantees in the
L OGGER framework.
Looking forward, it would be interesting to investigate the optimality of our sample complexity and interaction round complexity guarantees; we also speculate that it is possible to relax the small separator set assumption on B by utilizing very recent results on smoothed online learning [11, 28].
Finally, we are also interested in empirically evaluating our algorithms.
Acknowledgments and Disclosure of Funding.
We thank the anonymous reviewers for their constructive comments.
We thank Kwang-Sung Jun, Ryn Gray, Jason Pacheco, and members of the
University of Arizona machine learning reading group for helpful discussions.
We thank Wen Sun for helpful communications regarding [63, Theorem 5.3] and pointing us to an updated version [6,
Section 15.5].
We thank Weijing Wang for helping with illustrative ﬁgures.
This work is supported by a startup funding by the University of Arizona.