O PTIMISM IS A LL Y OU N EED : M ODEL -B ASED I MITATION L EARNING F ROM O BSERVATION A LONE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies Imitation Learning from Observations alone (ILFO) where the learner is presented with expert demonstrations that only consist of states encountered by an expert (without access to actions taken by the expert).
This paper presents a provably efﬁcient model-based framework MobILE to solve the ILFO problem.
MobILE uses self-supervision towards (a) training a dynamics model and (b) designing an intrinsic reward signal for exploration.
Using these ideas, MobILE carefully trades off exploration against imitation by integrating the idea of optimism in the face of uncertainty into the distribution matching imitation learning (IL) framework.
We provide a uniﬁed analysis for MobILE , and demonstrate that MobILE enjoys strong performance guarantees for classes of MDP dynamics that satisfy certain well studied notions of complexity.
We also show that the ILFO problem is strictly harder than the standard IL problem by reducing ILFO to a multi-armed bandit problem indicating that strategic exploration is necessary for solving ILFO efﬁciently.
We complement these theoretical results with experimental simulations on benchmark OpenAI Gym tasks that indicate the efﬁcacy of MobILE .
I NTRODUCTION
Imitation Learning (IL) is a paradigm that allows for sample-efﬁcient learning of sequential decision making policies, utilizing expert demonstrations consisting of states and actions.
IL has been successfully applied to applications such as Natural Language Processing (Daum ´ e et al., 2009),
Compilers (Mendis et al., 2019), Robotics (Levine et al., 2015), navigation (Ziebart et al., 2008), etc.
This paper, instead considers the Imitation Learning from Observation Alone (ILFO) setting.
In
ILFO, the learner is presented with sequences of states encountered by the expert, without access to the actions taken by the expert, meaning approaches based on a reduction to supervised learning (e.g., Behavior cloning (BC) (Ross & Bagnell, 2010), DAgger Ross et al.
(2011b)) are not applicable.
ILFO is more general and has potential for applications where learners and experts have different action spaces, applications like sim-to-real (Song et al., 2020; Desai et al., 2020).
Recently, Sun et al.
(2019b) reduces the ILFO problem to a sequence of one-step distribution matching problems that results in obtaining a non-stationary policy.
This approach is sample inefﬁcient for longer horizon tasks since the algorithm does not reuse previously collected samples when solving the current sub-problem.
Another line of work considers model-based methods to infer the expert’s actions with either an inverse (Torabi et al., 2018) or a forward dynamics (Edwards et al., 2019) model; these recovered actions are then fed into an IL approach like BC to output the ﬁnal policy.
These works rely on strong assumptions that tend to be satisﬁed when the underlying Markov Decision
Process (MDP) has deterministic transition dynamics.
See related works for a detailed treatment.
We introduce MobILE , a model-based framework, to solve the ILFO problem.
In contrast to existing model-based efforts, MobILE learns the forward transition dynamics model based on online interactions, and this is a well-deﬁned learnable object .
Importantly, MobILE combines exploration with imitation by utilizing ideas from self-supervised learning to interleave a model learning step with an intrinsic bonus-based, optimistic distribution matching step – a perspective, to the best of our knowledge, that has not been considered in the ILFO setting.
At a high level, our theoretical results and experimental studies demonstrate that
Strategic exploration is necessary for solving ILFO reliably and efﬁciently.
Note that this paper’s result extends the realm of partial information problems where optimism has been shown to be crucial in obtaining strong performance both in theory (e.g., E 3 (Kearns & Singh, 2002b), UCB (Auer et al., 2002)) and practice (e.g., RND (Burda et al., 2018)).
This paper proves that incorporating optimism into the min-max IL framework (Ho & Ermon, 2016; Sun et al., 2019b) is necessary for both the theoretical foundations and empirical performance of ILFO.
Main contributions: MobILE (Algorithm 1) is a provably efﬁcient, model-based framework for
ILFO.
MobILE can be instantiated with ﬂexible implementation choices owing to its modular design; MobILE also presents strong empirical performance on benchmark OpenAI gym tasks.
MobILE uses self supervision by combining ideas from model-based learning and optimism for exploration with adversarial imitation learning.
MobILE achieves global optimality with a regret bound growing sublinearly for classes of MDP dynamics that satisfy certain well studied notions of complexity.
The key idea of MobILE is to use optimism to trade-off imitation and exploration .
We show that optimism-based exploration is necessary for solving ILFO by presenting a reduction from the multi-armed bandit problem to ILFO.
This indicates that ILFO is fundamentally harder than IL where the learner has access to the expert’s actions.
Thus unlike classic methods like BC and DAgger, reductions to supervised learning are not sufﬁcient to provably solve ILFO.
We instantiate MobILE with neural networks and present experimental results on benchmark OpenAI Gym tasks, indicating MobILE compares favorably to or outperforms existing approaches.
Ablation studies indicate that optimism indeed boosts performance in practice
1. R ELATED W ORKS
Imitation Learning (IL) has seen advances through two types of approaches: (a) behavior cloning (BC) (Pomerleau, 1989) which casts IL as supervised or full-information online learning (Ross &
Bagnell, 2010; Ross et al., 2011b), or, (b) inverse RL (Ng & Russell, 2000; Abbeel & Ng, 2004;
Finn et al., 2016; Ke et al., 2019; Ghasemipour et al., 2020), which involves minimizing various distribution divergences to solve the IL problem, either with the transition dynamics known (e.g.,
Ziebart et al.
(2008)), or unknown (e.g., Ho & Ermon (2016)).
MobILE does not assume knowledge of the transition dynamics, is model-based, and operates without access to the expert’s actions.
Imitation Learning from Observation Alone (ILFO) Sun et al.
(2019b) presents a model-free approach that outputs a non-stationary policy by reducing the ILFO problem into a sequence of min-max problems, one per time-step.
This approach is sample inefﬁcient, particularly with long horizons; in contrast, our paper learns a stationary policy using model-based approaches.
Another line of work (Torabi et al., 2018; Edwards et al., 2019;
Yang et al., 2019) relies on learning an estimate of expert action, often through the use of an inverse dynamics models, P e ( a | s, s (cid:48) ) .
Unfortunately, there are many settings where an inverse model is not always well deﬁned.
For instance, inverse dynamics can be well deﬁned for deterministic MDP transition dynamics, whereas, this is not the case for stochastic transition dynamics.
From Bayes rule, to deﬁne an inverse model, we need a state-wise prior distribution and the expert policy, i.e., P e ( a | s, s (cid:48) )
∝ P ( s (cid:48) | s, a ) ρ ( s ) π e ( a | s ) .
Thus to learn P e ( a | s, s (cid:48) ) , we need training data with actions from π e 1 , which is missing in ILFO.
Finally, Edwards et al.
(2019)’s result applies only to MDPs with deterministic transitions and discrete actions.
Refer to section 6 in Sun et al.
(2019b)
for a more detailed treatment.
MobILE instead learns a forward dynamics model which is unique and well-deﬁned for deterministic and stochastic transitions and works with discrete/continuous actions.
Finally, Peng et al.
; Aytar et al.
Schmeckpeper et al.
(2020) solve ILFO using cost function engineering based on task-speciﬁc priors;
MobILE doesn’t require cost function engineering.
Model-Based RL (Sutton, 1990; Li & Todorov, 2004; Deisenroth & Rasmussen, 2011) has seen advances based on deep learning (Lampe & Riedmiller, 2014;
Gu et al., 2016; Janner et al., 2019) which can be translated to ILFO owing to MobILE ’s modularity.
MobILE bears parallels to provably efﬁcient model-based RL approaches including E 3 (Kearns & Singh, 2002a; Kakade et al., 2003),
R-MAX (Brafman & Tennenholtz, 2001), UCBVI (Azar et al., 2017), and LC 3 (Kakade et al., 2020a) which utilize a bonus based approach to trade-off exploration against exploitation.
S ETTING
We consider episodic ﬁnite-horizon MDP M = {S , A , P (cid:63) , H, c, s 0 } , where S , A are the state, action space, P (cid:63) :
S × A (cid:55)→ S is the transition kernel, H the horizon, s 0 is a ﬁxed initial state (note that we can handle a distribution over initial states), and c is the state-dependent cost function c : S (cid:55)→ [0 , 1] .
Notations We denote d πP ∈ ∆( S × A ) as the average state-action distribution of policy π under the transition kernel P , i.e., d πP ( s, a ) :
= 1 H (cid:80) Ht
=1 Pr ( s t = s, a t = a | s 0 , π, P ) , where Pr ( s t = s, a t = a | s 0 , π, P ) is the probability of reaching ( s, a ) at time step t starting from s 0 by following π under transition kernel P .
We abuse notation and write s ∼ d πP to denote a state s is sampled from the state-wise distribution which marginalizes action over d πP ( s, a ) , i.e., d πP ( s ) :
= 1 H (cid:80) Ht
=1 Pr ( s t = s
| s 0 , π, P ) .
For a given cost function f : S (cid:55)→ [0 , 1] , V πP ; f denotes the expected total cost of π under transition P and cost function f .
Similar to IL, in ILFO, the ground truth cost c is unknown .
Instead, we can query the expert, denoted as π e : S (cid:55)→ ∆( A ) , to provide state-only demonstrations τ = { s 0 , s 1 . . .
s H } , where s t +1 ∼ P (cid:63) ( ·| s t , a t ) and a t ∼ π (cid:63) ( ·| s t ) .
Note that the expert demonstration always starts from s 0 , which is the starting state distribution of the
MDP M .
Unlike interactive IL methods (e.g., DAgger (Ross et al., 2011a) and AggreVaTe(D) (Ross & Bagnell, 2014; Sun et al., 2017)), this paper considers the non-interactive expert setting where the algorithm cannot query the expert to provide trajectories starting from any state that the learner visits.
The goal is to leverage expert’s state-wise demonstrations to learn a policy π that performs as well as π e in terms of optimizing the ground truth cost c , with polynomial sample complexity on problem parameters such as horizon, number of expert samples and online samples and underlying MDP’s complexity measures (see section 4 for precise examples).
2. F UNCTION A PPROXIMATION S ETUP
Since the ground truth cost c is unknown, we utilize the notion of a function class F ⊂ S (cid:55)→ [0 , 1] to deﬁne the costs.
Furthermore, we use a model class P ⊂ S
× A (cid:55)→ ∆( S ) to capture the ground truth transition P (cid:63) .
For the theoretical results in the paper, we assume realizability:
Assumption 1.
Assume that F , P captures ground truth cost and transition, i.e., c ∈ F , and P (cid:63) ∈ P .
To permit generalization, we require P to have bounded complexity.
For analysis simplicity, assume
F is discrete (but exponentially large), and we require the sample complexity of any PAC algorithm to scale polynomially with respect to its complexity ln( |F| ) .
The ln |F| complexity can be replaced to bounded conventional complexity measures such as Rademacher complexity and covering number.
For the true model P (cid:63) and model class P we make the following structural assumption that the transition P (cid:63) is determined by a nonlinear deterministic function with additive Gaussian noise, i.e., s (cid:48) ∼ P ( ·| s, a ) , where s (cid:48)
=
g (cid:63) ( s, a ) + (cid:15), (cid:15) ∼ N (0 , σ 2 I ) , (1) where g (cid:63) is unknown and the level of Gaussian noise σ is known.
To learn g (cid:63) , we utilize a function class G ⊂ S
× A (cid:55)→ S .
Together with the known Gaussian noise, we have P =
{N ( g ( s, a ) , σ 2 I ) : g ∈ G} .
Extending our results to Gaussian noise with general positive deﬁnite covariance matrix is straightfoward; for analysis simplicity, we focus on covariance matrix σ 2 I .
Examples: One example is the Kernelized Nonlinear Regulator (KNR) model, where g (cid:63) ( s, a ) =
W (cid:63) φ ( s, a ) where φ : S × A (cid:55)→ H with H being some Hilbert space (e.g., a Reproducing Kernel
Hilbert Space) Kakade et al.
(2020b)
; Mania et al.
Note that this kernelized model captures both linear and nonlinear dynamical system such as hybrid linear systems and has been used in practice extensively (Ko et al., 2007; Deisenroth & Rasmussen, 2011; Fisac et al., 2018; Umlauft et al., 2018).
Another example is that g (cid:63) is captured by a Gaussian Process with some pre-deﬁned kernel k : ( S × A ) 2 (cid:55)→ R .
Moreover, G could also be a general function class.
For purposes of the theory results, we require these problem settings to satisfy certain regularity conditions, for e.g. , bounds on information gain (Srinivas et al., 2009), or, eluder dimension (Russo & Roy, 2013) similar to RL literature.
In Section 3 and 4, we discuss these examples in detail.
1: Require : IPM class F , dynamics model class P , policy class Π , bonus function class B .
2: Initialize
Π replay buffer D −
3: for t = 0 , · · · , T − 1 do 4:
Execute π t in true environment P (cid:63) to get samples τ t = { s k , a k } H − 1 k =0 ∪ s H .
Append to replay buffer D t
=
D t − 1 ∪ τ t .
Update model and bonus: (cid:98) P t +1 : S ×
A → S and b t +1 :
S ×
A → R + using buffer D t .
Get i.i.d expert states D e ≡ { s ei } Ni analysis) 7:
Optimistic model-based min-max IL: obtain π t +1 by solving equation t
D
8: end for 9: Return π T .
A LGORITHM
We introduce MobILE — Mo del- b ased I mitation L earning and E xploring for ILFO.
MobILE utilizes (a) a discriminator class F for performing Integral Probability Metric (IPM) based distribution matching, (b) a dynamics model class P for model learning, (c) a bonus function class B for exploration, (d) a policy class Π for policy learning.
MobILE (in Algorithm 1) iteratively learns a dynamics model (cid:98) P and uses this to learn a policy that aims to match the expert’s state visitation using discriminators F .
At every iteration, MobILE involves:
Dynamics Model Learning: execute current policy in the environment to obtain state-action-next state ( s, a, s (cid:48) ) triples which are appended to the buffer D .
Fit a dynamics model (cid:98) P on D . 2.
Intrinsic Bonus Design: design bonus to incentivize exploration where the learnt dynamics model is uncertain, i.e. the bonus b ( s, a ) is large
at state s where (cid:98) P ( ·| s, a ) is uncertain in terms of estimating P (cid:63) ( ·| s, a ) , while b (
s
, a ) is small where (cid:98) P ( ·| s, a ) is certain.
Trading off Imitation Against Exploration: Given discriminators F , a learned model (cid:98) P , bonus b and expert dataset D e , perform optimistic model-based optimization of the IPM objective:
[Equation]
Intuitively, the bonus cancels out the power of discriminators in parts of state space where the learned model (cid:98) P is inaccurate, thus allowing MobILE to explore.
Below we elaborate on the components of MobILE while highlighting its key property—which is to trade-off exploration and imitation .
3. C OMPONENTS OF M OB ILE
This section presents an overview of MobILE by instantiating these with the general function approximation class G ; refer to the appendix for instantiating MobILE in KNR case.
Learning the dynamics model: For model ﬁtting in line 5, given G , and the assumption on the ground truth model (Eq. 1), one can learn (cid:98) g t via least squares, i.e., (cid:98) g t = arg min g ∈G (cid:80) s,a,s (cid:48)
∈D t (cid:107) g
( s, a ) − s (cid:48)
(cid:107) 22 , and set (cid:98) P t ( ·| s, a ) =
N (cid:0)(cid:98)
g t ( s, a ) , σ 2 I (cid:1) .
In practice, there are a variety of developments, for e.g., learning a standard multi-layer perceptron (MLP) based Gaussian
Dynamics model (Rajeswaran et al., 2020) that can be used to effectively handle this step.
Intrinsic Bonus Parameterization: We utilize bonuses to incentivize the policy to explore unknown parts of the state space for improved model learning (and better distribution matching as a result).
For the general class G , given the least square solution (cid:98) g t , we can deﬁne a version space G t : G t = (cid:110) g ∈ G : (cid:80) t − 1 i =
(cid:80)
H − 1 h =0 (cid:107) g ( s th , a th ) − (cid:98) g t ( s th , a th ) (cid:107) 22 ≤ z t (cid:111) , with z t being a hyper parameter.
The version space G t is an ensemble of functions g ∈ G with training error on D t almost as small as the training error of the least square solution (cid:98) g t .
In other words, version space G t contains functions that nearly agree on D t .
The uncertainty measure at ( s, a ) is then the maximum disagreement among models in the ensemble G t , with b t ( s, a )
∝ sup f 1 ,f 2 ∈G t (cid:107) f 1 ( s, a ) − f 2 ( s, a ) (cid:107) 2 .
Since functions in
G t agree on D t , a large b t ( s, a ) indicates that ( s, a ) is novel.
See example 1 for more details.
Empirically, an ensemble’s model disagreement (Osband et al., 2018;
Azizzadenesheli et al., 2018;
Burda et al., 2019; Pathak et al., 2019; Lowrey et al., 2019) has been used for designing bonuses that incentivize exploration.
This paper utilizes an ensemble of neural networks to approximate the version space G t , where each model is trained on D t using SGD with different initialization.
The bonus is set as a function of maximum disagreement among the ensemble’s predictions.
Optimistic model-based min-max IL:
For model-based imitation (line 7), MobILE takes the current model (cid:98) P t and discriminators F as inputs and searches for a policy that approximately minimizes the divergence deﬁned by (cid:98) P n and F : d t ( π, π
[Equation]
= max f
[Equation]
Note that, for a ﬁxed π , the arg max f ∈F is identical with or without the bonus term, since E s,a ∼ d π (cid:98)
Pt b t ( s, a ) is independent of f .
Thus the bonus does not affect the search for the most powerful discriminators.
In our implementation, we use the Maximum Mean Discrepancy (MMD) with a
Radial Basis Function (RBF) kernel to model discriminators F 2 .
We compute arg min π d n ( π, π e ) by iteratively (1) computing the arg max discriminator f given current π , and (2) using policy gradient (PG) methods (e.g., TRPO) to perform gradient descent on π under (cid:98) P t with f ( s, a ) − b ( s, a ) as the cost.
Speciﬁcally, for line 7, we iterate between the following two steps; (1) update discriminator: ˆ
[Equation]
) − E s ∼ d πe f ( s ) and (2) update policy: ˆ π = ˆ π − η · ∇ π V ˆ π (cid:98) P t , ˆ f − b , where the PG step uses the learnt dynamics model (cid:98) P t and the optimistic IPM cost ˆ f ( s ) − b t ( s, a ) .
3. M OB ILE : E XPLORE A ND I MITATE D ILEMMA
We note that MobILE automatically trades-off exploration and imitation .
More speciﬁcally, the bonus is designed such that it has high values for states in the state space that have not been visited, and low values for states in the state space that have been frequently visited by the sequence of learned policies so far.
By incorporating the bonus into the discriminator f ∈ F (e.g., (cid:101) f ( s, a ) =
f ( s ) − b t ( s, a ) ), we diminish the power of discriminator f at novel state-action space regions.
Thus, when matching the learner’s states to expert’s states, we relax the state-matching constraint at those novel regions so that exploration is encouraged.
On the other hand, for well explored state regions, we force the learner’s states to match the expert’s states using the full power of the discriminators.
Is exploration necessary to solve the ILFO problem?
Conventional wisdom from the IL literature, where, the learner has access to expert actions suggests that exploration is not required for successful imitation – indeed, standard IL approaches such as BC (Ross & Bagnell, 2010) and DAgger (Ross et al., 2011a) reduce IL to supervised learning and do not rely on exploration to yield successful results.
Section 4.2 shows that exploration is necessary to solve the ILFO problem efﬁciently.
A NALYSIS
Recall that Algorithm 1 generates one state-action trajectory τ t :=
{ s th , a th } Hh =0 at iteration t and estimates model (cid:98) P t based on D t which contains previous t trajectories τ 0 , . .
, τ t − 1 .
We provide a uniﬁed analysis under the assumption that the model ﬁtting step gives us a calibrated model (Curi et al., 2020), i.e. one that offers predictions coupled with conﬁdence interval.
More speciﬁcally:
Assumption 2 (Calibrated Model) .
For all iteration t with t ∈ N , with probability 1 − δ , we have a model (cid:98) P t and its associated uncertainty measure σ t : S × A (cid:55)→ R + , such that for all s, a ∈ S
×
A : 3 (cid:13)(cid:13)(cid:13) (cid:98) P t ( ·| s, a ) − P (cid:63) ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 ≤ min { σ t ( s, a ) , 2 } .
Example 1 (General class G ) .
Assume G is discrete (but could be exponentially large with complexity measure ln( |G| ) ), and sup g ∈G ,s,a (cid:107) g
( s, a ) (cid:107) 2 ≤ G ∈ R
+ .
For model learning, (cid:98) g t is the the least square solution, i.e., (cid:98) g t = arg min g ∈G (cid:80) t − 1 k =
(cid:80)
H − 1 h =0 (cid:13) (cid:13) g
( s k h , a k h ) − s k
h
+1 (cid:13) (cid:13) 2 2 .
For the uncertainty measure, compute a version space G t =
(cid:110) g : (cid:80) t − 1 k =
(cid:80)
H − 1 h =
(cid:13)(cid:13)
g ( s kh , a kh ) − (cid:98) g t ( s kh , a kh )
(cid:13)(cid:13) 2 2 ≤ z t (cid:111) , where
For where
(cid:48) )
>
= k
(( s, a ) , ( s (cid:48) , a (cid:48) )) .the
uncertainty measure σ t ( s, a ) will depend on the input failure probability δ , which we drop here for notational simplicity.
When we introduce speciﬁc examples, this dependence on δ will be made explicit.
z t = 2 σ 2 G 2 ln(2 t 2 |G| /δ ) .
Set σ t ( s, a ) =
σ max g 1 ,g 2 ∈G (cid:107) g 1 ( s, a ) − g 2 ( s, a ) (cid:107) 2 , i.e., the maximum disagreement between any two functions in the version space G t .
Refer to Proposition 12 for more details.
The use of maximum disagreement above motivates our practical implementation where we use an ensemble of neural networks to approximate the version space and use the maximum disagreement among the models’ predictions as the bonus.
We refer readers to Appendix for more details.
4. R EGRET B OUND
We bound the regret with the quantity named Information Gain I (Srinivas et al., 2009):
[Equation]
where Alg is any adaptive algorithm (thus including Algorithm 1) that maps from history before iteration t to some policy π t ∈ Π .
After the main theorem, we give concrete examples for I T where we show that I T has extremely mild growth rate with respect to T (i.e., logarithimic).
Denote V π as the expected total cost of π under the true cost function c and the real dynamics P (cid:63) .
Theorem 3 (Main result) .
Assume model learning is calibrated (i.e., Assumption 2 holds for all t ).
In Algorithm 1, set bonus b t ( s, a ) :
=
H min { σ t ( s, a ) , 2 } .
There exists a set of parameters, such that after running Algorithm 1 for T iterations, we have:
[Equation]
N
[Equation]
Appendix A contains proof of Theorem 3.
This theorem indicates that as long as I T grows o ( √ T ) , we ﬁnd a policy that is at least as good as the expert policy when T and N approach inﬁnity.
We present several remarks below regarding specializing the above uniﬁed theorem to special instances.
For KNR and general G with bounded Eluder dimension (Osband & Van Roy (2014)), we can show that I T grows as ln( TH ) .
Thus, the algorithm converges to the expert policy at the rate (cid:101) O
(1 / √ T + 1 / √ N ) .
Corollary 4 (Bounded Eluder dimension (Example 1)) .
For general G , we assume that G has
Eluder-dimension d E ( (cid:15) )
(Deﬁnition 3 in Osband & Van Roy (2014)).
Denote d E = d E (1 /TH ) .
We can upper bound the information gain: I T
=
O (cid:0) Hd E + d E
ln
( T 3 H |G| )
ln( TH ) (cid:1) (see 14 for details).
Thus, E (cid:2) min t ∈ [0 ,...,T − 1] V π
[Equation]
Thus as long as G has bounded complexity in terms of the Eluder dimension, the maximum disagreement among models in the version space G t leads to near-optimal guarantees.
4. I S E XPLORATION N ECESSARY IN ILFO?
To answer this question, we present a novel reduction of the ILFO problem to a Multi-Armed Bandit (MAB) problem, for which we know exploration is necessary (Bubeck & Cesa-Bianchi, 2012); this indicates that exploration is necessary to solve the ILFO problem efﬁciently.
Consider a MAB problem with A actions { a i } Ai =1 .
Each action’s ground truth reward r i ∼ N ( µ i , 1) is from a Gaussian with mean µ i and variance 1 .
Without loss of generality, assume a 1 is the optimal arm, i.e., µ 1 > µ
i ∀ i (cid:54)
= 1 .
We convert this MAB instance into an MDP.
Speciﬁcally, set H = 2 .
Suppose we have a ﬁxed initial state s 0 which has A actions.
For the one step transition, we have
P ( ·| s 0 , a i ) = N ( µ i , 1) , i.e., g ∗ ( s 0 , a i ) = µ i .
Here we denote the optimal expert policy π e as π e ( s 0 ) =
a 1 , i.e., expert policy picks the optimal arm in the MAB instance.
Hence, when executing π e , we note that the state s 1 generated from π e is simply the stochastic reward of a 1 in the original
MAB instance.
Assume that we have observed inﬁnitely many such s 1 from the expert policy π e , i.e., we have inﬁnitely many samples of expert state data, i.e., N → ∞ .
Note, however, we do not have access to expert actions (since this is the ILFO setting).
This expert data is equivalent to revealing the optimal arm’s mean reward µ 1 to the MAB learner a priori.
Hence solving the ILFO problem on this
MDP is no easier than solving the original MAB instance with one additional piece of information which is that optimal arm’s mean reward is µ 1 (but the identity of the best arm is unknown).
[Figure]
exceeds BC’s performance despite BC having access to expert actions.
Theorem 5.
Consider solving a Gaussian MAB with the additional information that the optimal arm’s mean reward is µ (i.e., µ is known but the identity of the best arm is unknown).
For any algorithm, there exists a MAB instance with number of arms A ≥ 2 , such that the expected regret is still Ω( √ AT ) , i.e., the additional information does not help improving the worst-case regret bound.
Appendix A contains proof of Theorem 5.
Theorem 5 shows that solving ILFO, with even inﬁnite expert data, is at least as hard as solving the MAB problem with the known optimal arm’s mean reward which incurs the same worst case √ AT regret bound as the one in the classic MAB setting.
In contrast, in traditional IL and full-information supervised learning settings, methods like BC have sample complexities that scale as poly ln( A ) .
The necessity for exploration in ILFO is manifested in the exponential gap in the sample complexity dependence on A between IL and the IFLO setting.
E XPERIMENTS
This section seeks to answer the following questions:
• MobILE ’s behavior: How does MobILE perform relative to other model-based ILFO methods?
• Importance of optimism: Is optimism important for solving ILFO with MobILE ?
How does MobILE behave with varying levels of optimism?
• Stochastic environments: How does MobILE perform in MDPs with stochastic dynamics?
We consider tasks from Open AI
Gym (Brockman et al., 2016) simulated with Mujoco (Todorov et al., 2012) using Cartpole-v1 , Reacher-v2 , and Hopper-v2 tasks.
For Reacher-v2 , following Sun et al.
(2019b), we discretize every action dimension into ﬁve equally spaced bins.
For Hopper-v2 , we work with continuous actions.
We use TRPO (Schulman et al., 2015) for training an expert with value
≈ 460 , − 10 , 3000 for Cartpole-v1 , Reacher-v2 , Hopper-v2 .
We setup Hopper-v2 similar to prior model-based RL works (Kurutach et al., 2018; Luo et al., 2018).
We report results with 3 , 5 , 10 , 20 expert trajectories for Cartpole-v1 and with 5 , 10 , 20 ,trajectories for Reacher-v2 and Hopper-v2 ; all of our results are averaged over 3 seeds.
We benchmark MobILE against BC instead of other model-based ILFO algorithms.
Note, (a) BC upperbounds performance of other model based approaches such as BC-O (Torabi et al., 2018), (b) other approaches like Edwards et al.
(2019) require deterministic environments and discrete actions, whereas, MobILE works with stochastic environments and continuous actions.
Unlike MobILE and other ILFO methods, BC has access to expert actions and thus is a non-trivial benchmark.
We report the average across 3 seeds of the best policy obtained with BC.
See Appendix C for more details.
[Figure]
5. B ENCHMARKING M OB ILE ON M U J O C O SUITE
Figure 1 compares MobILE with BC.
Note that MobILE matches or exceeds BC’s performance despite BC having access to expert actions while MobILE functions without expert actions.
This implies MobILE improves upon BC-O
(Torabi et al., 2018), since it is outperformed by BC.
Note that
BC offers strong results in these benchmarks owing to deterministic transition dynamics; in section 5.3, we compare MobILE with BC on environments that exhibit stochastic transition dynamics.
5. I MPORTANCE OF THE OPTIMISTIC MDP CONSTRUCTION
We consider Reacher-v2 and Hopper-v2 with 10 expert trajectories.
We vary the level of exploration performed by MobILE by varying λ (the hyper-parameter that trades off discriminator cost against the bonus – for details, see Appendix section C.1).
Figure 2 indicates that the value of λ tends to inﬂuence MobILE ’s performance.
In particular, λ = 0 implies the algorithm is not explicitly incentivized to explore; it explores because of sampling actions from a stochastic policy.
We observe that lower λ ’s are associated with sample inefﬁciency in terms of number of online interactions needed to solve the problem.
A large λ however implies the algorithm over-explores and is not adequately rewarded for distribution matching.
The key to the success of MobILE is balancing exploration with imitation.
Empirically, we observe an initial increase in the bonus signifying that the algorithm explores followed by a decay as the algorithm trades-off exploration for imitation.
5. P ERFORMANCE WITH STOCHASTIC ENVIRONMENTS
[Figure]
BC fails with dynamics
MobILE performs reliable imitation
We consider a stochastic variant of Cartpole-v0 , wherein, zero mean additive Gaussian noise is added to the transition dynamics.
The variance of the noise is not known to the learner.
Figure 3 presents the results of BC in comparison to MobILE for this problem.
We observe that this minor modiﬁcation to the environment leads to rapid degradation of BC’s performance.
This result indicates that stochastic transition dynamics are not straightforward for algorithms relying on BC, for e.g. BC-O, to reliably solve while MobILE successfully solves the ILFO problem even with stochastic transition dynamics.
C ONCLUSIONS
We introduce MobILE , a model-based ILFO approach that works for MDPs with stochastic transition dynamics and continuous action spaces.
MobILE uses ideas from self-supervision by learning dynamics models and using this for bonus-based optimistic imitation.
MobILE automatically balances exploration and imitation and provably matches the expert’s behavior.
We show that exploration is necessary in ILFO as it is fundamentally harder than IL where the experts’ actions are known.
Our experiments on benchmark tasks (with discrete and continuous actions, stochastic and deterministic transitions) demonstrate that MobILE matches the expert efﬁciently and reliably.
A A NALYSIS OF A LGORITHM
We start by presenting the proof for the uniﬁed main result in Theorem 3.
We then discuss the bounds for special instances individually.
The following lemma shows that under Assumption 2, with b t ( s, a ) =
H min { σ t ( s, a ) , 2 } , we achieve optimism at all iterations.
Lemma 6 (Optimism) .
Assume Assumption 2 holds, and set b t ( s, a ) =
H min { σ t ( s, a ) , 2 } .
For all state-wise cost function f : S (cid:55)→ [0 , 1] , denote the bonus enhance cost as (cid:101) f t ( s, a ) :
= f ( s ) − b t ( s, a ) .
For all policy π , we have the following optimism:
[Equation]
Proof.
In the proof, we drop subscript t for notation simplicity.
We consider a ﬁxed function f and policy π .
Also let us denote (cid:98) V π as the value function of π under ( (cid:98) P, (cid:101) f ) , and V π as the value function under ( P, f ) .
Let us start from h = H , where we have (cid:98) V πH ( s ) =
V πH ( s ) = 0 .
Assume inductive hypothesis holds at h + 1 , i.e., for any s, a , we have (cid:98) Q πh +
( s, a ) ≤ Q πh +1 ( s, a ) .
Now let us move to h .
We have:
[Equation]
where the ﬁrst inequality uses the inductive hypothesis at time step h
+ 1 .
Finally, note that
V πh ( s ) =
E a ∼ π ( s ) Q πh ( s, a ) , which leads to (cid:98) V πh ( s )
≤ V πh ( s ) .
This concludes the induction step.
The next lemma concerns the statistical error from ﬁnite sample estimation of E s ∼ d πe f ( s ) .
Lemma 7.
Fix δ ∈ (0 , 1) .
For all t , we have that with probability at least 1 − δ ,
[Equation]
Proof.
For any t , we set the failure probability to be 6 δ/ ( t 2 π 2 ) at iteration t where we abuse notation and point out that π = 3 . 14159 ... .
Thus the total failure probability for all t ∈ N is at most δ .
We then apply classic Hoeffding inequality to bound E s ∼ d πe f ( s ) − (cid:80) Ni
=1 f ( s ei ) /N
with the fact that f ( s ) ∈ [0 , 1] for all s .
We conclude the proof by taking a union bound over all f ∈ F .
Now we conclude the proof for Theorem 3.
Proof of Theorem 3.
Assume that Assumption 2 and the event in Lemma 7 hold.
Denote the joint of these two events as E .
Note that the probability of E is at most 2 δ .
For notation simplicity, denote (cid:15) stats = 2 (cid:113)
ln(2 T 2 |F| /δ ) N .
In each model-based planning phase, recall that we perform model-based optimization on the following objective:
[Equation]
Note that for any π , using the inequality in Lemma 7, we have:
[Equation]
Hence, for π t , since it is the minimizer and π e ∈ Π , we must have:
[Equation]
Note that F contains c , we must have:
[Equation]
which means that V π
[Equation]
Now we compute the regret in episode t .
First recall that b t ( s, a ) =
H min { σ t ( s, a ) , 2 } , which means that (cid:107) b t (cid:107) ∞ ≤ 2 H as (cid:107) c (cid:107) ∞ ≤ 1 , which means that (cid:107) c
− b t (cid:107) ∞ ≤ 2 H .
Thus, (cid:13)(cid:13)(cid:13) V π [Equation] P
[Equation] c − b t (cid:13)(cid:13)(cid:13)
[Equation]
Recall simulation lemma (Lemma 15), we have:
[Equation]
Now sum over t , and denote E π t as the conditional expectation conditioned on the history from iteration 0 to t − 1 , we get:
[Equation]
where in the last inequality we use E [ a (cid:62) b ] ≤ (cid:112) E [ (cid:107) a (cid:107) 22 ] E [ (cid:107) b (cid:107) 22 ] .
Recall that π t are random quantities, add expectation on both sides of the above inequality, and consider the case where E holds and E holds, we have:
[Equation]
where in the last inequality, we use E [ a (cid:62) b ] ≤
[Equation] [Equation]
(cid:107) 22 ] .
This implies that that:
[Equation]
Set δ = 1 / ( HT ) , we get:
[Equation]
N +
[Equation]
T where Alg is any adaptive mapping that maps from history from t = 0 to the end of the t − 1 iteration to to some policy π t .
This concludes the proof.
Below we discuss special cases.
A. KNR S
Example 2 (KNRs) .
We have g (cid:63) ( s, a ) =
W (cid:63) φ ( s, a ) ; φ ( s, a ) ∈ H , and sup s,a (cid:107) φ ( s, a ) (cid:107) ≤ 1 .
We learn (cid:98) P t via Kernel regression, i.e., (cid:98) g t ( s, a ) = (cid:99) W t φ ( s, a ) , where, (cid:99) W [Equation] = arg min [Equation]
s
[Equation]
s [Equation] [Equation] , where (cid:107) · (cid:107) F is the Frobenius norm.
The uncertainty measure σ t ( s, a ) is set as σ t ( s, a ) =
β t σ (cid:107) φ ( s, a ) (cid:107) Σ − 1 t , where,
[Equation]
) + 2 ln [Equation] t 2 /δ ) [Equation]
ln [Equation] 4) + ln (det(Σ t ) / det( λI ))] } 1 / 2 , and,
[Equation]
where λ ∈ R + .Refer to Proposition 10 for more details.
Remark 8.
Similar to RKHS, Gaussian processes (GPs) offer a calibrated model.
Since GPs offer similar regret bounds as RKHS, we refer readers to Curi et al.
(2020) for details.
Corollary 9 (KNRs (Example 2)) .
For simplicity, consider the ﬁnite dimension setting where φ :
S × A (cid:55)→ R d .
We can show that I T = (cid:101) O (cid:0)
Hd +
Hdd s + Hd 2 (cid:1) (see Proposition 11 for details), where d is the dimension of the feature φ ( s, a ) .
Thus, we have
[Equation]
We extend the above result to inﬁnite dimensional RKHS below, where the dimension d in the above corollary is replaced by the intrinsic dimension, which can be bounded for RBF, Matern, and other common kernels (see Srinivas et al.
(2009) for more details).
The following proposition shows that the bonus designed in Example 2 is valid.
Proposition 10 (KNR Bonus) .
Fix δ ∈ (0 , 1) .
With probability at least 1 − δ , for all t ∈ N , we have:
[Equation]
| s, a ) (cid:13)(cid:13)(cid:13) 1 ≤ min (cid:110) β t (cid:107) φ ( s, a ) (cid:107) Σ − 1 t , 2 (cid:111) , ∀ s, a, where β t = (cid:112) 2 λ (cid:107) W (cid:63) (cid:107) 22 + 8 σ 2 ( d s ln(5) + 2 ln( t 2 /δ )
+ ln(4) + ln (det(Σ t ) / det( λI ))) .
Proof.
The proof directly follows the conﬁdence ball construction and proof from Kakade et al.
(2020a).
Speciﬁcally, from Lemma B.5 in Kakade et al.
(2020a), we have that with probability at least 1 − δ , for all t :
[Equation]
Thus, with Lemma 16, we have:
[Equation]
This concludes the proof.
The following proposition bounds the information gain quantity.
Proposition 11 (Information Gain on KNRs) .
For simplicity, let us assume φ : S × A (cid:55)→ R d , i [Equation] e., φ ( s, a ) is a d-dim feature vector.
In this case, we will have:
[Equation]
Proof.
From the previous proposition, we know that σ 2 t ( s, a ) =
( β 2 t /σ 2 ) (cid:107) φ ( s, a ) (cid:107) 2Σ − 1 t .
Setting λ
= σ 2 / (cid:107) W (cid:63) (cid:107) 22 , we will have β 2 t /σ 2 ≥ 1 , which means that min { σ 2 t ( s, a ) , 1 } ≤
[Equation]
Note that β t is non-decreasing with respect to t , so β t ≤ β T for T ≥ t , where
[Equation]
Also we have
[Equation]
since min { a 1 , b 1 } + min { a 2 , b 2 } ≤ min { a 1 + a 2 , b 1 + b 2 } .
Now call Lemma B.6 in Kakade et al.
(2020a), we have:
[Equation]
Finally recall the deﬁnition of I T , we have:
[Equation]
h = min (cid:8) σ
[Equation]
[Equation]
th ) ,
[Equation]
[Table]
≤ β
[Equation]
σ
[Equation]
(cid:88)
[Table]
=
[Table]
[Equation]
[Table] (cid:88) h = min (cid:110) (cid:107) [Equation] ( s th
[Equation]
th ) (cid:107)
[Equation]
1 t
[Equation]
(cid:111) ≤
[Equation]
) ≤
[Equation]
[Equation]
(cid:0)
[Equation]
{+(cid:107) W (cid:63) (cid:107) 22 TH/σ 2 (cid:1)(cid:1)(cid:1) ln (cid:0) 
 + (cid:107) W (cid:63) (cid:107) 22 TH/σ 2 (cid:1) =+} H (cid:0) [Equation]
{+32 dd s + 
+}
d
ln
( T 2 /δ )
[Equation]
32 d +
[Equation]
ln (cid:0) 1 + (cid:107) W (cid:63) (cid:107) 22 TH
[Equation]
σ [Equation] (cid:1)(cid:1) ln (cid:0) 1 + (cid:107) W (cid:63) (cid:107) 22 TH/σ 2 (cid:1) , which concludes the proof. Extension to Inﬁnite Dimensional RKHS When φ : S × A (cid:55)→ H where H is some inﬁnite dimensional RKHS, we can bound our regret using the following intrinsic dimension:
[Equation]
) (cid:62) (cid:33) .
In this case, recall Proposition
[Equation]
we have:
[Equation]
Also recall Eq.
(4), we have:
[Equation]
Combine the above two, following similar derivation we had for ﬁnite dimensional setting [Equation] we have:
[Equation]
A. G ENERAL F UNCTION C LASS G WITH B OUNDED E LUDER DIMENSION
Proposition 12.
Fix δ ∈ (0 , 1) .
Consider a general function class G where G is discrete, and sup g ∈G
[Equation]
s,a [Equation] g (cid:107) 2 ≤ G .
At iteration t , denote (cid:98) g t ∈ arg
[Equation]
ih
[Equation]
ih )
[Equation]
s ih
[Equation]
[Equation]
[Equation]
and denote a version space G t as:
[Equation]
i = H − 1 (cid:88) h
= (cid:13)(cid:13) g ( s ih , a ih ) − (cid:98) g t ( s ih , a ih ) (cid:13)(cid:13) 2 2 ≤ c t (cid:41) , with c t = σG (cid:112) ln(2 t 2 |G| /δ )
tH.
The with probability at least 1 − δ , we have that for all t , and all s, a :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof.
Consider a ﬁxed function g ∈ G .
Let us denote z th = (cid:13)(cid:13)
g
( s th , a th ) − s th +1 (cid:13)(cid:13) 2 2 − [Equation]
g [Equation] s th , a th ) − s th +1 (cid:13)(cid:13) 2 2 .
We have:
[Equation]
Since (cid:15) th ∼ N (0 , σ 2 I ) , we must have:
[Equation]
Since sup g,s,a (cid:107) g ( s, a ) (cid:107) 2 ≤ G , then 2( g ( s th , a th ) − g (cid:63) ( s th , a th )) (cid:62) (cid:15) th is a 2 σG sub-Gaussian random variable.
Call Lemma 3 in Russo & Van Roy (2014), we have that with probability at least 1 − δ :
[Equation]
Note that the above can also be derived directly from Azuma-Bernstein’s inequality.
With a union bound over all g ∈ G , we have:
[Equation]
[Table]
t
[Table]
+
[Table]
t (cid:88) h (cid:13)(cid:13) g (
[Table]
th , a th ) − g (cid:63) ( s th
[Table]
a th ) (cid:13)(cid:13) 2 2 − 4 σ 2 G 2 ln( |G| /δ ) .
Set g
= (cid:98) g t , and use the fact that g t is the minimizer of (cid:80) t (cid:80) h [Equation]
g
[Equation]
s th , a th ) − s [Equation] [Equation] 22 , we must have:
[Equation]
Namely we prove that our version space G t contains g (cid:63) for all t .
Thus, we have:
[Equation]
where the last inequality holds since both g (cid:63) and (cid:98) g t belongs to the version G t .
Now we bound the information gain I T below.
The proof mainly follows from the proof in Osband & Van Roy (2014).
Lemma 13 (Lemma 1 in Osband & Van Roy (2014)) .
Denote β t = 2 σ 2 G 2 ln( t 2 |G| /δ ) .
Let us denote the uncertainty measure w t ; h = sup f 1 ,f 2 ∈G t (cid:107) f 1 ( s th , a th ) − f 2 ( s th , a th ) (cid:107) 2 (note that w t ; h is non-negative).
We have:
[Equation]
Proposition 14 (Bounding I T ) .
Denote d = d E (1 /TH ) .
We have
[Equation]
Proof.
Note that the uncertainty measures w t ; h are non-negative.
Let us reorder the sequence and denote the ordered one as w 1 ≥ w 2 ≥ w 3 · · · ≥ w TH − H .
For notational simplicity, denote
M = TH − H
We have:
[Equation]
Consider any w t where w 2 t ≥ 1 /M .
In this case, we know that w 21 ≥ w 22 ≥ · · · ≥ w 2 t ≥ 1 /M .
This means that:
[Equation]
where the second inequality uses the lemma above, and the last inequality uses the fact that d E ( (cid:15) ) is non-decreasing when (cid:15) gets smaller.
Denote d = d E (1 /M ) .
We have that w 2 t ≤ 4 β T d t − Hd .
This means that for any w 2 t ≥ 1 /M , we must have w 2 t ≤ 4 β T d/ ( t − Hd ) .
Thus, we have:
[Equation]
Finally, recall the deﬁnition of I T [Equation] we have:
[Equation]
This concludes the proof
[Equation]
A. P ROOF OF T HEOREM
This section provides the proof of Theorem 5.
Proof of Theorem 5.
Below, we will construct A many MAB instances where each instance has A many arms and each arm has a Gaussian reward distribution with the ﬁxed variance σ 2 .
Each of the A instance has the maximum mean reward equal to ∆ , i.e., all these A instances have the same maximum arm mean reward.
Consider any algorithm Alg that maps ∆ together with the history of the interactions H t
= { a 0 , r 0 , a 1 , r 1 , . . .
, a t − 1 , r t − 1 } to a distribution over A actions.
We will show for any such algorithm alg that knows ∆ , with constant probability, there must exist a MAB instance from the K many MAB instances, such that Alg suffers at least Ω( √ AT ) regret where T is the number of iterations.
Now we construct the A instances as follows.
Consider the i -th instance ( i = 1 , . . .
, A ).
For arm j in the i-th instance, we deﬁne its mean as µ ij = 1 { i = j } ∆ .
Namely, for MAB instance i , its arms have mean reward zero everywhere except that the i -th arm has reward mean ∆ .
Note that all these MAB instances have the same maximum mean reward, i.e., ∆ .
Hence, we cannot distinguish them by just revealing ∆ to the learner.
We will construct an additional MAB instance (we name it as 0 -th MAB instance) whose arms have reward mean zero.
Note that this MAB instance has maximum mean reward 0 which is different from the previous A MAB instances that we constructed.
However, we will only look at the regret of Alg on the previously constructed A MAB instances.
I.e., we do not care about the regret of Alg(∆ , H t ) on the 0 -th MAB instance.
Let us denote P i (for i = 0 , . . .
, A ) as the distribution of the outcomes of algorithm Alg(∆ , H t ) interacting with MAB instance i for T iterations, and E j [ N i ( T )] as the expected number of times arm i is pulled by Alg(∆ , H t ) in MAB instance j .
Consider MAB instance i with i ≥ 1 :
[Equation]
where the last step uses the fact that we are running the same algorithm Alg (∆ , H t ) on both instance 0 and instance i (i.e., same policy for generating actions), and thus, KL( P 0 , P i )
= (cid:80)
Aj = [Equation] E [-
-][Equation]
[ N j ( T )]KL ( q 0 ( j ) , q i ( j ))
(Lemma 15.1 in Lattimore & Szepesv ´ ari (2020)), where q i ( j ) is the reward distribution of arm j at instance i .
Also recall that for instance
and instance i , their rewards only differ at arm i .
This implies that:
[Equation]
Sum over i = 1 , . . .
, A on both sides, we have:
[Equation]
Now let us calculate the regret of Alg(∆ , H t ) on i -th instance, we have:
[Equation]
Sum over i = 1 , . . .
, A , we have:
[Equation]
[Equation]
Set ∆ =
c (cid:112) A/T for some c that we will specify later, we get:
[Equation]
Set c = 1 / 4 , we get:
[Equation]
assuming A ≥ 2 .
Thus there must exist i ∈ { 1 , . . .
, A } , such that:
[Equation]
Note that the above construction considered any algorithm Alg(∆ , H t ) that maps ∆ and history to action distributions.
Thus it concludes the proof.
B A UXILIARY L EMMAS
Lemma 15 (Simulation Lemma) .
Consider any two functions f : S × A (cid:55)→ [0 , 1] and (cid:98) f : S × A (cid:55)→ [0 , 1] , any two transitions P and (cid:98) P , and any policy π : S (cid:55)→ ∆( A ) .
We have:
[Equation]
where V πP,f ; h denotes the value function at time step h , under π, P, f .
Such simulation lemma is standard in model-based RL literature and the derviation can be found, for instance, in the proof of Lemma 10 from Sun et al.
(2019a).
Lemma 16.
Consider two Gaussian distribution P 1 := N ( µ 1 , σ 2 I ) and P 2 :
=
N ( µ 2 , σ 2 I ) .
We have:
[Equation]
C I MPLEMENTATION D ETAILS C. P RACTICAL I NSTANTIATION OF M OB ILE
We present a brief practical instantiation MobILE ’s components with details in Appendix Section C.
Dynamics model learning: We employ Gaussian Dynamics Models parameterized by an MLP (Rajeswaran et al., 2020), i.e., (cid:98) P ( s, a ) :
= N ( h θ ( s, a ) , σ 2 I ) , where, h θ ( s, a ) = s + σ ∆ s · MLP θ ( s c , a c ) , where, θ are MLP’s trainable parameters, s c
=
( s − µ s ) /σ s , a c
=
( a − µ a ) /σ a .
Next, for ( s, a, s (cid:48) )
∈ D , ∆ s = s (cid:48) − s and σ ∆ s is the standard deviation of the state differences ∆ s ∈ D .
Finally, we use SGD with momentum (Sutskever et al., 2013) for purposes of training the parameters θ of the MLP.
Discriminator parameterization: We utilize MMD as our choice of IPM and deﬁne the discriminator as f ( s ) =
w (cid:62) ψ ( s ) , where, ψ ( s ) are the Random Fourier Features (Rahimi & Recht, 2008).
Bonus parameterization: We utilize the discrepancy between predictions of an ensemble of a pair of dynamics models h
θ 1 ( s, a ) and h θ 2 ( s, a ) for designing the bonus.
Denote the disagreement at any ( s, a ) as δ ( s, a ) = (cid:107) h θ 1 ( s, a ) − h θ 2 ( s, a ) (cid:107) 2 .
For a replay buffer D , δ D = max ( s,a ) ∼D δ ( s, a ) is the maximum discrepancy.
We set bonus as b ( s, a ) =
min( δ ( s, a ) /δ D , 1) · λ where λ > 0 is a tunable parameter.
The normalization 1 /δ D helps the maintaining the bonus magnitude between (0 , 1] .
PG oracle: We use TRPO (Schulman et al., 2015) to perform policy optimization inside the learned model.
E NVIRONMENT S ETUP
This section sketches the details of how we setup the environments.
We utilize the standard environment horizon of 500 , 50 , 200 for Cartpole-v1 , Reacher-v2 , Cartpole-v0 .
For
Swimmer-v2 and Hopper-v2 , we work with the environment horizon set to 400 (Kurutach et al., 2018;
Nagabandi et al., 2018;
Luo et al., 2018; Rajeswaran et al., 2020).
Furthermore, for Hopper-v2 , we add the velocity of the center of mass to the state parameterization (Rajeswaran et al., 2020; Luo et al., 2018).
As noted in the main text, the expert policy is trained using NPG/TRPO (Kakade, 2001; Schulman et al., 2015) until it hits a value of (approximately) 460 , − 10 , 38 , 3000 , 181 for Cartpole-v1 , Reacher-v2 , Swimmer-v2 , Hopper-v2 ,
Cartpole-v0 respectively.
All the results presented in the experiments section are averaged over three seeds.
Furthermore, in terms of baselines, we compare MobILE to BC.
Note that BC has access to expert actions whereas our algorithm does not have access to the expert actions.
For fair comparison, we use the same policy architecture for both MobILE and BC.
We report the average of the best performance offered by BC for 500 epochs of training when run with 3 seeds, even if this occurs at different epochs for each of the runs.
Furthermore, we used 2 CPUs with 16 - 32 GB of RAM usage to perform all our benchmarking runs implemented in Pytorch.
Finally, we build our codebase on top of Open-AI’s implementation of TRPO (Dhariwal et al., 2017) for environments with discrete actions, and the MJRL repository (Rajeswaran et al., 2017) for continuous action environments.
C. P RACTICAL I MPLEMENTATION OF M OB ILE
We will begin with presenting the implementation details of MobILE (refer to Algorithm 2):
C.3. D YNAMICS M ODEL T RAINING
As detailed in the main paper, we utilize a class of Gaussian Dynamics Models parameterized by an MLP (Rajeswaran et al., 2020), i.e. s (cid:48) ∼ s + N ( h θ ( s c , a c ) , σ 2 · I ) , where, h θ ( · ) is an MLP with trainable parameters θ ; s c , a c are centered and normalized state and actions respectively; note that we predict normalized state differences instead of the next state directly.
In practice, we ﬁne tune our estimate of dynamics models based on the new contents of the replay buffer as opposed to re-training the models from scratch, which is computationally more expensive.
In particular, we start from the estimate (cid:98) P t − 1 in the t − 1 epoch and perform multiple updates gradient updates using the contents of the replay buffer D .
We utilize constant stepsize SGD with momentum (Sutskever et al., 2013) for updating our dynamics models.
Furthermore, since the distribution of ( s, a, s (cid:48) ) pairs continually drift as the algorithm progresses (for instance, because we observe a new state), we utilize gradient clipping to ensure our model does not diverge due to the aggressive nature of our updates.
C.3. R EPLAY B UFFER
Since we perform incremental training of our dynamics model, we utilize a replay buffer of a ﬁxed size rather than training our dynamics model on all previously collected online ( s, a, s (cid:48) ) samples.
Note that the replay buffer could contain data from all prior online interactions should we re-train our dynamics model from scratch at every epoch.
C.3. D ESIGN OF B ONUS F UNCTION
We utilize an ensemble of two transition dynamics models incrementally learned using the contents of the replay buffer.
Speciﬁcally, given the models h θ 1 ( · ) and h θ 2 ( · ) , we compute the discrepancy as: δ ( s, a )
=
||
h θ 1 ( s, a ) − h θ
( s, a )
|| 2 .
Moreover, given a replay buffer D , we compute the maximum discrepancy as δ D = max ( s,a,s (cid:48) )
∼D δ ( s, a ) .
We then set the bonus as b ( s, a ) =
min (1 , δ ( s, a ) /δ D ) · λ , thus ensuring the magnitude of our bonus remains bounded between [0 , λ ] .
C.3. D ISCRIMINATOR U PDATE
Recall that f w ( s )
=
w (cid:62) ψ ( s ) , where w are the parameters of the discriminator.
Given a policy π , the update for the parameters w take the following form:
[Equation]
where, ∂ w L ζ ( w ; π, (cid:98) P, b, D e ) denotes the sub-differential of L ζ ( · ) wrt w .
This implies:
Exact Update: w ∗ =
P B ( ζ ) (cid:16) E s ∼ d π (cid:98) P [ ψ ( s )] − E s ∼D e
[ ψ ( s )] (cid:17) , P · is the projection operator, and B ( ζ ) is the ζ − norm ball.
Gradient Ascent Update:
[Equation]
stepsize
[Equation]
Empirically, we ﬁnd either of the updates to work reasonably well.
Furthermore, in certain cases, we empirically observe the gradient ascent update to yield more stability compared to the exact updates.
C.3. M ODEL -B ASED P OLICY U PDATE
Once the maximization of the discriminator parameters w is performed, consider the policy optimization problem, i [Equation] e.,
[Equation]
Hence we perform model-based policy optimization under (cid:98) P and cost function f w ( s ) − b ( s, a ) .
In practice, we perform approximate minimization of L ( · ) by incrementally updating the policy using
K PG -steps of policy gradient, where, K PG is a tunable hyper-parameter.
In our experiments, we ﬁnd that setting K PG to be around 10 to generally be a reasonable choice (for precise values, refer to Table 1).
This paper utilizes TRPO (Schulman et al., 2015) as our choice of policy gradient method; note that this can be replaced by other alternatives including PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018) etc.
Similar to practical implementations of existing policy gradient methods, we implement a reward ﬁlter by clipping the IPM reward f ( s ) by truncating it between c min and c max as this leads to stability of the policy gradient updates.
Note that the minimization is done with access to (cid:98) P , [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
which implies we perform model-based planning.
Empirically, for purposes of tuning the explorationimitation parameter λ , we minimize a surrogate namely: E ( s,a ) ∼ d
[Table] [(1 − λ ) · f w ( s ) − b ( s, a )] (recall that b ( s, a ) has a factor of λ associated with it).
This ensures that we can precisely control the magnitude of the bonuses against the IPM costs, which, in our experience is empirically easier to work with.
C. H YPER - PARAMETER D ETAILS
This section presents an overview of the list of hyper-parameters necessary to implement Algorithmin practice, as described in Algorithm 2.
The list of hyper-parameters is precisely listed out in table 1.
The hyper-parameters are broadly categorized into ones corresponding to various components of
MobILE , namely, (a) environment speciﬁcations, (b) dynamics model, (c) ensemble based bonus, (d) IPM parameterization, (e) Policy parameterization,
(f) Planning algorithm parameters, (g) Critic parameterization.
Note that if there a hyper-parameter that has not been listed, for instance, say, the value of momentum for the ADAM optimizer in the critic, this has been left as is the default value deﬁned in Pytorch.