U NDERSTANDING D OMAIN R ANDOMIZATION FOR S IM - TO - REAL T RANSFER
A BSTRACT
Reinforcement learning encounters many challenges when applied directly in the real world.
Sim-to-real transfer is widely used to transfer the knowledge learned from simulation to the real world.
Domain randomization—one of the most popular algorithms for sim-to-real transfer—has been demonstrated to be effective in various tasks in robotics and autonomous driving.
Despite its empirical successes, theoretical understanding on why this simple algorithm works is limited.
In this paper, we propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters (corresponding to unknown physical parameters such as friction).
We provide sharp bounds on the sim-to-real gap—the difference between the value of policy returned by domain randomization and the value of an optimal policy for the real world.
We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples.
Our theory also highlights the importance of using memory (i.e., history-dependent policies) in domain randomization.
Our proof is based on novel techniques that reduce the problem of bounding the sim-to-real gap to the problem of designing efﬁcient learning algorithms for inﬁnite-horizon MDPs, which we believe are of independent interest.
I NTRODUCTION
Reinforcement Learning (RL) is concerned with sequential decision making, in which the agent interacts with the environment to maximize its cumulative rewards.
This framework has achieved tremendous empirical successes in various ﬁelds such as Atari games, Go and StarCraft (Mnih et al., 2013; Silver et al., 2017; Vinyals et al., 2019).
However, state-of-the-art algorithms often require a large amount of training samples to achieve such a good performance.
While feasible in applications that have a good simulator such as the examples above, these methods are limited in applications where interactions with the real environment are costly and risky, such as healthcare and robotics.
One solution to this challenge is sim-to-real transfer (Floreano et al., 2008; Kober et al., 2013).
The basic idea is to train an RL agent in a simulator that approximates the real world and then transfer the trained agent to the real environment.
This paradigm has been widely applied, especially in robotics (Rusu et al., 2017; Peng et al., 2018; Chebotar et al., 2019) and autonomous driving (Pouyanfar et al., 2019;
Niu et al., 2021).
Sim-to-real transfer is appealing as it provides an essentially unlimited amount of data to the agent, and reduces the costs and risks in training.
However, sim-to-real transfer faces the fundamental challenge that the policy trained in the simulated environment may have degenerated performance in the real world due to the sim-to-real gap —the mismatch between simulated and real environments.
In addition to building higher-ﬁdelity simulators to alleviate this gap, domain randomization is another popular method (Sadeghi & Levine, 2016;
Tobin et al., 2017; Peng et al., 2018; OpenAI et al., 2018).
Instead of training the agent in a single simulated environment, domain randomization randomizes the dynamics of the environment, thus exposes the agent to a diverse set of environments in the training phase.
Policies learned entirely in the simulated environment with domain randomization can be directly transferred to the physical world with good performance (Sadeghi & Levine, 2016; Matas et al., 2018; OpenAI et al., 2018).
In this paper, we focus on understanding sim-to-real transfer and domain randomization from a theoretical perspective.
The empirical successes raise the question: can we provide guarantees for the sub-optimality gap of the policy that is trained in a simulator with domain randomization and directly transferred to the physical world?
To do so, we formulate the simulator as a set of
MDPs with tunable latent variables, which corresponds to unknown parameters such as friction coefﬁcient or wind velocity in the real physical world.
We model the training process with domain randomization as ﬁnding an optimal history-dependent policy for a latent MDP , in which an MDP is randomly drawn from a set of MDPs in the simulator at the beginning of each episode.
Our contributions can be summarized as follows:
• We propose a novel formulation of sim-to-real transfer and establish the connection between domain randomization and the latent MDP model ( Kwon et al., 2021).
The latent MDP model illustrates the uniform sampling nature of domain randomization, and helps to analyze the sim-to-real gap for the policy obtained from domain randomization.
• We study the optimality of domain randomization in three different settings.
Our results indicate that the sim-to-real gap of the policy trained in the simulation can be o ( H ) when the randomized simulator class is ﬁnite or satisﬁes certain smoothness condition, where H is the horizon of the real-world interaction.
We also provide a lower bound showing that such benign conditions are necessary for efﬁcient learning.
Our theory highlights the importance of using memory (i e., history-dependent policies) in domain randomization.
• To analyze the optimality of domain randomization, we propose a novel proof framework which reduces the problem of bounding the sim-to-real gap of domain randomization to the problem of designing efﬁcient learning algorithms for inﬁnite-horizon MDPs, which we believe are of independent interest.
• As a byproduct of our proof, we provide the ﬁrst provably efﬁcient model-based algorithm for learning inﬁnite-horizon average-reward MDPs with general function approximation (Algorithm 4 in Appendix C.3).
Our algorithm achieves a regret bound of ˜ O
( D √ d e T ) , where T is the total timesteps and d e is a complexity measure of a certain function class F that depends on the eluder dimension (Russo & Van Roy, 2013; Osband & Van Roy, 2014).
R ELATED W ORK
Sim-to-Real and Domain Randomization
The basic idea of sim-to-real is to ﬁrst train an RL agent in simulation, and then transfer it to the real environment.
This idea has been widely applied to problems such as robotics (e.g., Ng et al., 2006; Bousmalis et al., 2018; Tan et al., 2018; OpenAI et al., 2018) and autonomous driving (e.g., Pouyanfar et al., 2019; Niu et al., 2021).
To alleviate the inﬂuence of reality gap, previous works have proposed different methods to help with sim-to-real transfer, including progressive networks (Rusu et al., 2017), inverse dynamics models (Christiano et al., 2016) and Bayesian methods (Cutler & How, 2015; Pautrat et al., 2018).
Domain randomization is an alternative approach to making the learned policy to be more adaptive to different environments (Sadeghi & Levine, 2016; Tobin et al., 2017; Peng et al., 2018; OpenAI et al., 2018), thus greatly reducing the number of real-world interactions.
There are also theoretical works related to sim-to-real transfer.
Jiang (2018) uses the number of different state-action pairs as a measure of the gap between the simulator and the real environment.
Under the assumption that the number of different pairs is constant, they prove the hardness of sim-to-real transfer and propose efﬁcient adaptation algorithms with further conditions.
Feng et al.
(2019) prove that an approximate simulator model can effectively reduce the sample complexity in the real environment by eliminating sub-optimal actions from the policy search space.
Zhong et al.
(2019) formulate a theoretical sim-to-real framework using the rich observation Markov decision processes (ROMDPs), and show that the transfer can result in a smaller real-world sample complexity.
None of these results study beneﬁts of domain randomization in sim-to-real transfer.
Furthermore, all above works require real-world samples to ﬁne-tune their policy during training, while our work and the domain randomization algorithm do not.
POMDPs and Latent MDPs Partially observable Markov decision processes (POMDPs) are a general framework for sequential decision-making problems when the state is not fully observable (Smallwood & Sondik, 1973; Kaelbling et al., 1998; Vlassis et al., 2012; Jin et al., 2020a;
Xiong et al., 2021).
Latent MDPs (Kwon et al., 2021), or LMDPs, are a special type of POMDPs, in which the real environment is randomly sampled from a set of MDPs at the beginning of each episode.
This model has been widely investigated with different names such as hidden-model MDPs and multi-model MDPs.
There are also results studying the planning problem in LMDPs, when the true parameters of the model is given (Chades et al., 2012; Buchholz & Scheftelowitsch, 2019;
Steimle et al., 2021) .
Kwon et al.
(2021) consider the regret minimization problem for LMDPs, and provide efﬁcient learning algorithms under different conditions.
We remark that all works mentioned above focus on the problems of ﬁnding the optimal policies for POMDPs or latent MDPs, which is perpendicular to the central problem of this paper—bounding the performance gap of transferring the optimal policies of latent MDPs from simulation to the real environment.
Inﬁnite-horizon Average-Reward MDPs
Recent theoretical progress has produced many provably sample-efﬁcient algorithms for RL in inﬁnite-horizon average-reward setting.
Nearly matching upper bounds and lower bounds are known for the tabular setting (Jaksch et al., 2010; Fruit et al., 2018; Zhang & Ji, 2019; Wei et al., 2020).
Beyond the tabular case, Wei et al.
(2021) propose efﬁcient algorithms for inﬁnite-horizon MDPs with linear function approximation.
To the best of our knowledge, our result (Algorithm 4) is the ﬁrst efﬁcient algorithm with near-optimal regret for inﬁnite-horizon average-reward MDPs with general function approximation.
P RELIMINARIES 3. E PISODIC MDP S
We consider episodic RL problems where each MDP is speciﬁed by M =
( S , A , P, R, H, s 1 ) .
S and A are the state and the action space with cardinality S and A respectively.
We assume that S and A are ﬁnite but can be extremely large.
P : S ×
A → ∆( S ) is the transition probability matrix so that P ( ·| s, a ) gives the distribution over states if action a is taken on state s , R : S
×
A → [0 , 1] is the reward function.
H is the number of steps in one episode.
For simplicity, we assume the agent always starts from the same state in each episode, and use sto denote the initial state at step h = 1 .
It is straight-forward to extend our results to the case with random initialization.
At step h ∈ [ H ] , the agent observes the current state s h ∈ S , takes action a h ∈ A , receives reward R ( s h , a h ) , and transits to state s h +1 with probability P
( s h
+
| s h , a h ) .
The episode ends when s H +1 is reached.
We consider the history-dependent policy class Π , where π ∈ Π is a collection of mappings from the history observations to the distributions over actions.
Speciﬁcally, we use traj h
= { ( s 1 , a 1 , s 2 , a 2 , · · · , s h ) | s i ∈ S , a i ∈ A , i ∈
[ h ] } to denote the set of all possible trajectories of history till step h .
We deﬁne a policy π ∈ Π to be a collection of H policy functions { π h : traj h → ∆( A )
} h ∈
[ H ] .
We deﬁne V π M ,h : S → R to be the value function at step h under policy π on MDP M , i.e., V π M ,h ( s ) =
E M ,π [ (cid:80)
Ht
= h R ( s t , a t ) | s h
= s ] .
Accordingly, we deﬁne Q π M ,h : S ×
A → R to be the Q-value function at step h : Q π M ,h ( s, a ) =
E M ,π [ R ( s h , a h ) + (cid:80)
Ht = h +1 R ( s t , a t ) | s h
= s, a h
= a ] .
We use π ∗M to denote the optimal policy for a single MDP M .
It can be shown that there exists π ∗M such that the policy at step h depends on only the state at step h but not any other prior history.
That is, π ∗M can be expressed as a collection of H policy functions mapping from S to ∆( A ) .
We use
V ∗M ,h and Q ∗M ,h to denote the optimal value and Q-functions under the optimal policy π ∗M at step h .
3. P RACTICAL I MPLEMENTATION OF D OMAIN R ANDOMIZATION
In this subsection, we brieﬂy introduce how domain randomization works in practical applications.
Domain randomization is a popular technique for improving domain transfer (Tobin et al., 2017;
Peng et al., 2018; Matas et al., 2018), which is often used for zero-shot transfer when the target domain is unknown or cannot be easily used for training.
For example, by highly randomizing the rendering settings for their simulated training set, Sadeghi & Levine (2016) trained vision-based controllers for a quadrotor using only synthetically rendered scenes.
OpenAI et al.
(2018) studied the problem of dexterous in-hand manipulation.
The training is performed entirely in a simulated environment in which they randomize the physical parameters of the system like friction coefﬁcients and vision properties such as object’s appearance.
To apply domain randomization in the simulation training, the ﬁrst step before domain randomization is usually to build a simulator that is close to the real environment.
The simulated model is further improved to match the physical system more closely through calibration.
Though the simulation is still a rough approximation of the physical setup after these engineering efforts, these steps ensure that the randomized simulators generated by domain randomization can cover the real-world variability.
During the training phase, many aspects of the simulated environment are randomized in each episode in order to help the agent learn a policy that generalizes to reality.
The policy trained with domain randomization can be represented using recurrent neural network with memory such as LSTM (Yu et al., 2018; OpenAI et al., 2018; Doersch & Zisserman, 2019).
Such a memory-augmented structure allows the policy to potentially identify the properties of the current environment and adapt its behavior accordingly.
With sufﬁcient data sampled using the simulator, the agent can ﬁnd a near-optimal policy w.r.t.
the average value function over a variety of simulation environments.
This policy has shown its great adaptivity in many previous results, and can be directly applied to the physical world without any real-world ﬁne-tuning (Sadeghi & Levine, 2016;
Matas et al., 2018; OpenAI et al., 2018).
F ORMULATION
In this section, we propose our theoretical formulation of sim-to-real and domain randomization.
The corresponding models will be used to analyze the optimality of domain randomization in the next section, which can also serve as a starting point for future research on sim-to-real.
4. S IM - TO - REAL T RANSFER
In this paper, we model the simulator as a set of MDPs with tunable latent parameters.
We consider an MDP set U representing the simulator model with joint state space S and joint action space A .
Each MDP M = ( S , A , P M , R, H, s 1 ) in U has its own transition dynamics P M , which corresponds to an MDP with certain choice of latent parameters.
Our result can be easily extended to the case where the rewards are also inﬂuenced by the latent parameters.
We assume that there exists an
MDP M ∗ ∈ U that represents the dynamics of the real environment.
We can now explain our general framework of sim-to-real.
For simplicity, we assume that during the simulation phase (or training phase), we are given the entire set U that represents MDPs under different tunable latent parameter.
Or equivalently, the learning agent is allowed to interact with any
MDP M ∈ U in arbitrary fashion, and sample arbitrary amount of trajectories.
However, we do not know which MDP M ∈ U represents the real environment.
The objective of sim-to-real transfer is to ﬁnd a policy π purely based on U , which performs well in the real environment.
In particular, we measure the performance in terms of the sim-to-real gap , which is deﬁned as the difference between the value of learned policy π and the value of an optimal policy for the real world:
[Equation]
We remark that in our framework, the policy π is learned exclusively in simulation without the use of any real world samples.
We study this framework because (1) our primary interests—domain randomization algorithm does not use any real-world samples for training; (2) we would like to focus on the problem of knowledge transfer from simulation to the real world.
The more general learning paradigm that allows the ﬁne-tuning of policy learned in simulation using real-world samples can be viewed as a combination of sim-to-real transfer and standard on-policy reinforcement learning, which we left as an interesting topic for future research.
4. D OMAIN R ANDOMIZATION AND LMDP S
We ﬁrst introduce Latent Markov decision processes (LMDPs) and then explain domain randomization in the viewpoint of LMDPs.
A LMDP can be represented as ( U , ν ) , where U is a set of
MDPs with joint state space S and joint action space A , and ν is a distribution over U .
Each MDP
M = ( S , A , P M , R, H, s 1 ) in U has its own transition dynamics P M that may differs from other
MDPs.
At the start of an episode, an MDP M ∈ U is randomly chosen according to the distribution ν .
The agent does not know explicitly which MDP is sampled, but she is allowed to interact with this MDP M for one entire episode.
Domain randomization algorithm ﬁrst speciﬁes a distribution over tunable parameters, which equivalently gives a distribution ν over MDPs in simulator U .
This induces a LMDP with distribution ν .
The algorithm then samples trajectories from this LMDP, runs RL algorithms in order to ﬁnd the near-optimal policy of this LMDP.
We consider the ideal scenario that the domain randomization algorithm eventually ﬁnd the globally optimal policy of this LMDP, which we formulate as domain randomization oracle as follows:
Deﬁnition 1.
(Domain Randomization Oracle)
Let U be the set of MDPs generated by domain randomization and ν be the uniform distribution over U .
The domain randomization oracle returns an optimal history-dependent policy π ∗ DR of the LMDP ( U , ν ) :
[Equation]
Since LMDP is a special case of POMDPs, its optimal policy π ∗ DR in general will depend on history.
This is in sharp contrast with the optimal policy of a MDP, which is history-independent.
We emphasize that both the memory-augmented policy and the randomization of the simulated environment are critical to the optimality guarantee of domain randomization.
We also note that we don’t restrict the learning algorithm used to ﬁnd the policy π ∗ DR , which can be either in a model-based or model-free style.
Also, we don’t explicitly deﬁne the behavior of π ∗ DR .
The only thing we know about π ∗ DR is that it satisﬁes the optimality condition deﬁned in Equation 2.
In this paper, we aim to bound the sim-to-real gap of π ∗ DR , i.e., Gap ( π ∗ DR , U ) under different regimes.
M AIN R ESULTS
We are ready to present the sim-to-real gap of π ∗ DR in this section.
We study the gap in three different settings under our sim-to-real framework: ﬁnite simulator class (the cardinality |U| is ﬁnite) with the separation condition (MDPs in U are distinct), ﬁnite simulator class without the separation condition, and inﬁnite simulator class.
During our analysis, we mainly study the long-horizon setting where
H is relatively large compared with other parameters.
This is a challenging setting that has been widely-studied in recent years (Gupta et al., 2019; Mandlekar et al., 2020; Pirk et al., 2020).
We show that the sim-to-real gap of π ∗ DR is only O (log 3 ( H )) for the ﬁnite simulator class with the separation condition, and only ˜ O ( √ H ) in the last two settings, matching the best possible lower bound in terms of H .
In our analysis, we assume that the MDPs in U are communicating MDPs with a bounded diameter.
Assumption 1 (Communicating MDPs (Jaksch et al., 2010)) .
The diameter of any MDP M ∈ U is bounded by D .
That is, consider the stochastic process deﬁned by a stationary policy π : S → A on an MDP with initial state s .
Let T ( s (cid:48)
|M , π, s ) denote the random variable for the ﬁrst time step in which state s (cid:48) is reached in this process, then max s (cid:54) = s (cid:48) ∈S min
[Equation]
This is a natural assumption widely used in the literature (Jaksch et al., 2010; Agrawal & Jia, 2017;
Fruit et al., 2020).
The communicating MDP model also covers many real-world tasks in robotics.
For example, transferring the position or angle of a mechanical arm only costs constant time.
Moreover, the diameter assumption is necessary under our framework.
Proposition 1.
Without Assumption 1, there exists a hard instance U so that Gap( π ∗ DR ) =
Ω( H ) .
We prove Proposition 1 in Appendix G.1.
Note that the worst possible gap of any policy is H , so π ∗ DR becomes ineffective without Assumption 1.
5. F INITE S IMULATOR C LASS W ITH S EPARATION C ONDITION
As a starting point, we will show the sim-to-real gap when the MDP set U is a ﬁnite set with cardinality M .
Intuitively, a desired property of π ∗ DR is the ability to identify the environment the agent is exploring within a few steps.
This is because π ∗ DR is trained under uniform random environments, so we hope it can learn to tell the differences between environments.
As long as π ∗ DR has this property, the agent is able to identify the environment dynamics quickly, and behave optimally afterwards (note that the MDP set U is known to the agent).
Before presenting the general results, we ﬁrst examine a simpler case where all MDPs in U are distinct.
Concretely, we assume that any two MDPs in U are well-separated on at least one stateaction pair.
Note that this assumption is much weaker than the separation condition in Kwon et al.
(2021), which assumes strongly separated condition for each state-action pair.
Assumption 2 ( δ -separated MDP set) .
For any M 1 , M 2 ∈ U , there exists a state-action pair ( s, a ) ∈ S ×A , such that the L 1 distance between the probability of next state of the different MDPs is at least δ , i.e. (cid:107) ( P M 1 − P M 2 ) ( · | s, a ) (cid:107) 1 ≥ δ.
The following theorem shows the sim-to-real gap of π ∗ DR in δ -separated MDP sets.
Theorem 1.
Under Assumption 1 and Assumption 2, for any M ∈ U , the sim-to-real gap of π ∗ DR is at most
[Equation]
The proof of Theorem 1 is deferred to Appendix D. Though the dependence on M and δ may not be tight, our bound has only poly-logarithmic dependence on the horizon H .
The main difﬁculty to prove Theorem 1 is that we do not know what π ∗ DR does exactly despite knowing a simple and clean strategy in the real-world interaction with minimum sim-to-real gap.
That is, to ﬁrstly visit the state-action pairs that help the agent identify the environment quickly and then follow the optimal policy in the real MDP M ∗ after identifying M ∗ .
Therefore, we use a novel constructive argument in the proof.
We construct a base policy that implements the idea mentioned above, and show that π ∗ DR cannot be much worse than the base policy.
The proof overview can be found in Section 6.
5. F INITE S IMULATOR C LASS W ITHOUT S EPARATION C ONDITION
Now we generalize the setting and study the sim-to-real gap of π ∗ DR when U is ﬁnite but not necessary a δ -separated MDP set.
Surprisingly, we show that π ∗ DR can achieve ˜ O ( √ H ) sim-to-real gap when |U| =
M .
Theorem 2.
Under Assumption 1, when the MDP set induced by domain randomization U is a ﬁnite set with cardinality M , the sim-to-real gap of π ∗ DR is upper bounded by
[Equation]
Theorem 2 is proved in Appendix E. This theorem implies the importance of randomization and memory in the domain randomization algorithms (Sadeghi & Levine, 2016; Tobin et al., 2017; Peng et al., 2018; OpenAI et al., 2018).
With both of them, we successfully reduce the worst possible gap of π ∗ DR from the order of H to the order of √ H , so per step loss will be only ˜ O
( H − 1 / 2 ) .
Without randomization, it is not possible to reduce the worst possible gap (i.e., the sim-to-real gap) because the policy is even not trained on all environments.
Without memory, the policy is not able to implicitly “identify” the environments, so it cannot achieve sublinear loss in the worst case.
We also use a constructive argument to prove Theorem 2.
However, it is more difﬁcult to construct the base policy because we do not have any idea to minimize the gap without the well-separated condition (Assumption 2).
Fortunately, we observe that the base policy is also a memory-based policy, which basically can be viewed as an algorithm that seeks to minimize the sim-to-real gap in an unknown underlying MDP in U .
Therefore, we connect the sim-to-real gap of the base policy with the regret bound of the algorithms in inﬁnite-horizon average-reward MDPs (Bartlett & Tewari, 2012; Fruit et al., 2018; Zhang & Ji, 2019).
The proof overview is deferred to Section 6.
To illustrate the hardness of minimizing the worst case gap, we prove the following lower bound for
Gap ( π, U ) to show that any policy must suffer a gap at least Ω( √ H ) .
Theorem 3.
Under Assumption 1, suppose A ≥ 10 , SA ≥ M ≥ 100 , D ≥ 20 log A M, H ≥ DM , for any history dependent policy π
= { π h : traj
h
→ A} Hh =1 , there exists a set of M MDPs
U =
{M m } Mm =1 and a choice of M ∗ ∈
U such that Gap( π ) is at least Ω( √ DMH ) .
The proof of Theorem 3 follows the idea of the lower bound proof for tabular MDPs (Jaksch et al., 2010), which we defer to Appendix G.2.
This lower bound implies that Ω( √ H ) sim-to-real gap is unavoidable for the policy π ∗ DR when directly transferred to the real environment.
5. I NFINITE S IMULATOR C LASS
In real-world scenarios, the MDP class is very likely to be extensively large.
For instance, many physical parameters such as surface friction coefﬁcients and robot joint damping coefﬁcients are sampled uniformly from a continuous interval in the Dexterous Hand Manipulation algorithms (OpenAI et al., 2018).
In these cases, the induced MDP set U is large and even inﬁnite.
A natural question is whether we can extend our analysis to the inﬁnite simulator class case, and provide a corresponding sim-to-real gap.
Intuitively, since the domain randomization approach returns the optimal policy in the average manner, the policy π ∗ DR can perform bad in the real world M ∗ if most MDPs in the randomized set differ much with M ∗ .
In other words, U must be ”smooth” near M ∗ for domain randomization to return a nontrivial policy.
By ”smoothness”, we mean that there is a positive probability that the uniform distribution
ν returns a MDP that is close to M ∗ .
This is because the probability that ν samples exactly M ∗ in a inﬁnite simulator class is 0, so domain randomization cannot work at all if such smoothness does not hold.
Formally, we assume there is a distance measure d ( M 1 , M 2 ) on U between two MDPs M 1 and
M 2 .
Deﬁne the (cid:15) -neighborhood
C M ∗ ,(cid:15) of M ∗ as C M ∗ ,(cid:15) def = {
M ∈ U : d ( M , M ∗ ) ≤ (cid:15) } .
The smoothness condition is formally stated as follows:
Assumption 3 (Smoothness near M ∗ ) .
There exists a positive real number (cid:15) 0 , and a Lipchitz constant L , such that for the policy π ∗ DR , the value function of any two MDPs in C M ∗ ,(cid:15) 0 is L -Lipchitz w.r.t the distance function d , i.e.
[Equation]
For example, we can set d ( M 1 , M 2 ) =
I [ M 1 (cid:54) =
M 2 ] in the ﬁnite simulator class.
For complicated simulator class, we need to ensure there exists some d ( · , · ) that L is not large.
With Assumption 3, it is possible to compute the sim-to-real gap of π ∗ DR .
In the ﬁnite simulator class, we have shown that the gap depends on M polynomially, which can be viewed as the complexity of
U .
The question is, how do we measure the complexity of U when it is inﬁnitely large?
Motivated by Ayoub et al.
(2020), we consider the function class
F = { f M ( s, a, λ ) : S ×
A × Λ → R such that f M ( s, a, λ ) =
P M λ ( s, a ) for M ∈ U , λ ∈ Λ } , (6) where Λ = { λ ∗M , M ∈ U} is the optimal bias functions of M ∈ U in the inﬁnite-horizon averagereward setting (Bartlett & Tewari (2012);
Fruit et al.
; Zhang & Ji (2019)).
We note this function class is only used for analysis purposes to express our complexity measure; it does not affect the domain randomization algorithm.
We use the the (cid:15) -log-covering number and the (cid:15) -eluder dimension of F to characterize the complexity of the simulator class U .
In the setting of linear combined models (Ayoub et al., 2020), the (cid:15) -log-covering number and the (cid:15) -eluder dimension are
O ( d log(1 /(cid:15) )) , where d is the dimension of the linear representation in linear combined models.
For readers not familiar with eluder dimension or inﬁnite-horizon average-reward MDPs, please see
Appendix A for preliminary explanations.
Here comes our bound of sim-to-real gap for the inﬁnite simulator class setting, which is proved in
Appendix F.
Theorem 4.
Under Assumption 1 and 3, the sim-to-real gap of the domain randomization policy π ∗ DR is at most for 0 ≤ (cid:15) < (cid:15)
[Equation]
Here ν ( C M ∗ ,(cid:15) ) is the probability of ν sampling a MDP in C M ∗ ,(cid:15) , d e = dim E ( F , 1 /H ) is the 1 /Heluder dimension F , and N ( F , 1 /H ) is the 1 /H
-covering number of F w.r.t.
L ∞ norm.
Theorem 4 is a generalization of Theorem 2, since we can reduce Theorem 4 to Theorem 2 by setting d ( M 1 , M 2 )
=
I [ M 1 (cid:54) =
M 2 ] and (cid:15) = 0 , in which case ν ( C M ∗ ,(cid:15) )
= 1 /M and d e ≤ M .
The proof overview can be found in Section 6.
The main technique is still a reduction to the regret minimization problem in inﬁnite-horizon average-reward setting.
We construct a base policy and shows that the regret of it is only ˜ O ( √ H ) .
A key point to note is that our construction of the base policy also solves an open problem of designing efﬁcient algorithms that achieve ˜ O ( √ T ) regret in the inﬁnite-horizon average-reward setting with general function approximation.
This base policy is of independent interests.
To complement our positive results, we also provide a negative result that even if the MDPs in
U have nice low-rank properties (e.g., the linear low-rank property (Jin et al., 2020b; Zhou et al., 2020)), the policy π ∗ DR returned by the domain randomization oracle can still have Ω( H ) sim-to-real gap when the simulator class is large and the smoothness condition (Assumption 3) does not hold.
This explains the necessity of our preconditions.
Please refer to Proposition 2 in Appendix G.3 for details.
P ROOF O VERVIEW
In this section, we will give a short overview of our novel proof techniques for the results shown in section 5.
The main proof technique is based on reducing the problem of bounding the sim-to-real gap to the problem of constructing base policies.
In the settings without separation conditions, we further connect the construction of the base policies to the design of efﬁcient learning algorithms for the inﬁnite-horizon average-reward settings.
6. R EDUCING TO C ONSTRUCTING B ASE P OLICIES
Intuitively, if there exists a base policy ˆ π ∈ Π with bounded sim-to-real gap, then the gap of π ∗ DR will not be too large since π ∗ DR deﬁned in Eqn 2 is the policy with the maximum average value.
Lemma 1.
Suppose there exists a policy ˆ π ∈ Π such that the sim-to-real gap of ˆ π for any MDP
M ∈ U satisﬁes V ∗M , 1 ( s 1 ) − V
ˆ π M , 1 ( s 1 ) ≤ C , then we have Gap( π ∗ DR ) ≤ MC when U is a ﬁnite set with |U| =
M .
Furthermore, when U is an inﬁnite set satisfying the smoothness condition (assumption 3), we have for any 0 < (cid:15) < (cid:15) 0 , Gap( π ∗ DR )
≤ C/ν ( C M ∗ ,(cid:15) ) + L(cid:15).
We defer the proof to Appendix B.1.
Now with this reduction lemma, the remaining problem is deﬁned as follows:
Suppose the real MDP M ∗ belongs to the MDP set U .
We know the full information (transition matrix) of any MDP in the MDP set U .
How to design a history-dependent policy ˆ π ∈ Π with minimum sim-to-real gap max M∈U (cid:0) V [Equation] [Equation] [Equation]
ˆ
[Equation] M [Equation] [Equation]
[Equation]
6. T HE C ONSTRUCTION OF THE B ASE P OLICIES
With separation conditions With the help of Lemma 1, we can bound the sim-to-real gap in the setting of ﬁnite simulator class with separation condition by constructing a history-dependent policy ˆ π .
The formal deﬁnition of the policy ˆ π can be found in Appendix C.1.
The idea of the construction is based on elimination: the policy ˆ π explicitly collects samples on the “informative” state-action pairs and eliminates the MDP that is less likely to be the real MDP from the candidate set.
Once the agent identiﬁes the real MDP representing the dynamics of the physical environment, it follows the optimal policy of the real MDP until the end of the interactions.
Without separation conditions The main challenge in this setting is that, we can no longer construct a policy ˆ π that “identify” the real MDP using the approaches as in the settings with separation conditions.
In fact, we may not be able to even “identify” the real MDP since there can be MDPs in
U that is very close to real MDP.
Here, we use a different approach, which reduces the minimization of sim-to-real gap of ˆ π to the regret minimization problem in the inﬁnite-horizon average-reward
MDPs.
The inﬁnite-horizon average-reward setting has been well-studied (e.g., Jaksch et al., 2010;
Agrawal & Jia, 2017; Fruit et al., 2018; Wei et al., 2020).
The main difference compared with the episodic setting is that the agent interacts with the environment for inﬁnite steps.
The gain of a policy is deﬁned in the average manner.
The value of a policy π is deﬁned as ρ π ( s ) =
E [lim T →∞ (cid:80) Tt =1 R ( s t , π ( s t )) /T
| s 1 = s ] .
The optimal gain is deﬁned as ρ ∗ ( s ) def = max s ∈S max π ρ π ( s ) , which is shown to be state-independent in Agrawal & Jia (2017), so we use ρ ∗ for short.
The regret in the inﬁnite-horizon setting is deﬁned as Reg ( T ) =
E (cid:104)
Tρ ∗ − (cid:80) Tt =1 R ( s t , a t ) (cid:105) , where the expectation is over the randomness of the trajectories.
A more detailed explanation of inﬁnite-horizon average-reward MDPs can be found in Appendix A.1.
For an MDP M ∈ U , we can view it as a ﬁnite-horizon MDP with horizon H ; or we can view it as an inﬁnite-horizon MDP.
This is because Assumption 1 ensures that the agent can travel to any state from any state
s H encountered at the H -th step (this may not be the case in the standard ﬁnitehorizon MDPs, since people often assume that the states at the H -th level are terminating state).
The following lemma shows the connection between these two views.
Lemma 2.
For a MDP M , let ρ ∗M and V ∗M , 1 ( s 1 ) to be the optimal expected gain in the inﬁnitehorizon view and the optimal value function in the episodic view respectively.
We have the following inequality:
[Equation]
This lemma indicates that, if we can design an algorithm (i.e. the base policy) ˆ π in the inﬁnitehorizon setting with regret Reg ( H ) , then the sim-to-real gap of this algorithm in episodic setting satisﬁes Gap (ˆ π ) =
V ∗M , 1 ( s 1 ) − V
ˆ π M , 1 ( s 1 ) ≤ Reg ( H ) + D .
This lemma connects the sim-to-real gap of ˆ π in ﬁnite-horizon setting to the regret in the inﬁnite-horizon setting.
With the help of Lemma 1 and 2, the remaining problem is to design an efﬁcient exploration algorithm for inﬁnite-horizon average-reward MDPs with the knowledge that the real MDP M ∗ belongs to a known MDP set U .
Therefore, we propose two optimistic-exploration algorithms (Algorithmand Algorithm 4) for the setting of ﬁnite simulator class and inﬁnite simulator class respectively.
The formal deﬁnition of the algorithms are deferred to Appendix C.2 and Appendix C.3.
Note that our Algorithm 4 is the ﬁrst efﬁcient algorithm with ˜ O ( √ T ) regret in the inﬁnite-horizon averagereward MDPs with general function approximation, which is of independent interest for efﬁcient online exploration in reinforcement learning.
C ONCLUSION
In this paper, we study the optimality of policies learned from domain randomization in sim-to-real transfer without real-world samples.
We propose a novel formulation of sim-to-real transfer and view domain randomization as an oracle that returns the optimal policy of an LMDP with uniform initialization distribution.
Following this idea, we show that the policy π ∗ DR can suffer only o ( H ) loss compared with the optimal value function of the real environment when the simulator class is ﬁnite or satisﬁes certain smoothness condition, thus this policy can perform well in the long-horizon cases.
We hope our formulation and analysis can provide insight to design more efﬁcient algorithms for sim-to-real transfer in the future.
A CKNOWLEDGMENTS
Liwei Wang was supported by National Key R&D Program of China (2018YFB1402600), Exploratory Research Project of Zhejiang Lab (No. 2022RC0AN02), BJNSF (L172037), Project 2020BD006 supported by PKUBaidu Fund.
A A DDITIONAL P RELIMINARIES A. I NFINITE - HORIZON A VERAGE - REWARD MDP S
The inﬁnite-horizon average-reward setting has been well-explored in the recent few years (e.g. Jaksch et al.
; Agrawal & Jia (2017); Fruit et al.
; Wei et al.
The main difference compared with the episodic setting is that the agent interacts with the environment for inﬁnite steps instead of restarting every H steps.
The gain of a policy is deﬁned in the average manner.
Deﬁnition 2.
(Deﬁnition 4 in Agrawal & Jia (2017))
The gain ρ π ( s ) of a stationary policy π from starting state s 1 = s is deﬁned as:
[Equation]
In this setting, a common assumption is that the MDP is communicating (Assumption 1).
Under this assumption, we have the following lemma.
Lemma 3.
(Agrawal & Jia, 2017, Lemma 2.1)
For a communicating MDP M with diameter D : (a)
The optimal gain ρ ∗ is state-independent and is achieved by a deterministic stationary policy π ∗ DR ; that is, there exists a deterministic policy π ∗ such that
[Equation]
(b) The optimal gain ρ ∗ satisﬁes the following equation:
[Equation]
− λ
[Equation]
s
[Equation]
[Equation]
s
[Equation]
[Equation]
where Pλ ( s, a ) =
(cid:80) s (cid:48)
P ( s (cid:48) | s, a ) λ ( s (cid:48) ) , and λ ∗ is the bias vector of the optimal policy π ∗ DR satisfying
[Equation]
The regret minimization problem has been widely studied in this setting, with regret to be deﬁned as Reg ( T ) =
E (cid:104)
Tρ ∗ − (cid:80) Tt =1 R ( s t , a t ) (cid:105) , where the expectation is over the randomness of the trajectories.
For example, Jaksch et al.
(2010) proposed an efﬁcient algorithm called UCRL2, which achieves regret upper bound ˜ O
( DS √ AT ) .
For notation convenience, we use PV ( s, a ) or Pλ ( s, a ) as a shorthand of (cid:80) s (cid:48)
A. E LUDER D IMENSION
Proposed by Russo & Van Roy (2013), eluder dimension has become a widely-used concept to characterize the complexity of different function classes in bandits and RL (Wang et al., 2020;
Ayoub et al., 2020; Jin et al., 2021; Kong et al., 2021).
In this work, we deﬁne eluder dimension to characterize the complexity of the function F :
F = { f M ( s, a, λ ) : S × A × Λ → R such that f M ( s, a, λ ) =
P M λ ( s, a ) for M ∈ U , λ ∈ Λ } , (12) where Λ = { λ ∗M , M ∈ U} is the optimal bias functions of M ∈ U in the inﬁnite-horizon averagereward setting (Bartlett & Tewari (2012); Fruit et al.
; Zhang & Ji (2019)).
Deﬁnition 3.
(Eluder dimension).
Let (cid:15) ≥ 0 and Z = { ( s i , a i , λ i ) } ni =1 ⊂ S ×A× Λ be a sequence of history samples.
• A history sample ( s, a, λ ) ∈ S
×
A × Λ is (cid:15) -dependent on Z with respect to F
if any f, f (cid:48) ∈ F satisfying (cid:107) f − f (cid:48) (cid:107) Z ≤ (cid:15) also satisﬁes (cid:107) f
( s, a ) − f (cid:48)
( s, a ) (cid:107) ≤ (cid:15) .
Here (cid:107) f − f (cid:48) (cid:107) Z is a shorthand of (cid:113)(cid:80) ( s,a,λ ) ∈Z ( f − f (cid:48) )
2 ( s, a, λ ) .
An ( s, a, λ ) is (cid:15) -independent of Z with respect to F if ( s, a, λ ) is not (cid:15) -dependent on Z .
• The (cid:15) -eluder dimension of a function class F is the length of the longest sequence of elements in S ×
A × Λ such that, for some (cid:15) (cid:48) ≥ (cid:15) , every element is (cid:15) (cid:48) -independent of its predecessors.
B O MITTED P ROOF IN S ECTIONB. P ROOF OF L EMMA
Proof.
We ﬁrstly study the case where U is a ﬁnite set with |U| =
M .
For ˆ π , we have
[Equation]
By the optimality of π ∗ DR , we know that
[Equation]
Therefore,
[Equation]
≤ C
[Equation]
Since the gap V ∗M , 1 ( s 1 ) − V π ∗ DR M , 1 ( s 1 ) ≥ 0 for any i ∈ [ M ] , we have 1 M (cid:16) V ∗ M ∗ , [Equation] 1 ) − [Equation]
π ∗
[Equation] [Equation] ≤ C .
That is,
[Equation]
For the case where U is an inﬁnite set satisfying Assumption 3, by the optimality of π ∗ DR , we have
[Equation]
Therefore,
E
[Equation]
[Equation]
By Assumption 3, for any M ∈ C ( M ∗ , (cid:15) ) , we have
[Equation]
[Equation]
Therefore, we have
[Equation]
Combining Inq 18 and Inq 20, we have
[Equation]
The lemma can be proved by reordering the above inequality.
B. P ROOF OF L EMMA
Proof.
For MDP M , denote π ∗ in as the optimal policy in the inﬁnite-horizon setting and { π ∗ ep,h } Hh = as the optimal policy in the episodic setting.
By the optimality of π ∗ ep , we have V ∗M , 1 ( s 1 ) =
V π ∗ ep
[Equation]
By the Bellman equation in the inﬁnite-horizon setting, we know that
[Equation]
[Equation] For notation simplicity, we use d h ( s 1 , π ) to denote the state distribution at step h after starting from state s 1 at step 1 following policy π .
From the above equation, we have
[Equation]
[Equation]
That is,
[Equation]
[Equation]
where (cid:80)
Hh =1 E s h ∼ d h ( s 1 ,π ∗ in )
R ( s h , , π ∗ in ( s h )) =
V π ∗ in M , 1 ( s 1 ) .
Therefore, we have Hρ ∗M − D ≤
V π ∗ in M , 1 ( s 1 ) ≤ V ∗M , 1 ( s 1 ) .
For the second inequality, by the Bellman equation in the inﬁnite-horizon setting, we have
[Equation]
That is,
[Equation]
(26) where (cid:80)
Hh =1 E s h
[Equation]
[Equation]
ep
[Equation]
R ( s h
[Equation]
, π ∗ ep,h ( s h ))
=
V ∗M , 1 ( s 1 ) .
C T HE C ONSTRUCTION OF L EARNING A LGORITHMS C. F INITE S IMULATOR C LASS WITH S EPARATION C ONDITION
In this subsection, we explicitly deﬁne the base policy ˆ π with sim-to-real gap guarantee under the separation condition.
Note that a history-dependent policy for LMDPs can also be regarded as an
1: Initialize: the MDP set D = U , n 0 = c 0 log 2 ( SMH )log( MH ) δfor a constant c2: (cid:46) Stage 1: Explore
and ﬁnd the real MDP M ∗ 3: while |D| ≥ 1 do 4:
Randomly select two MDPs M 1 and M 2 from the MDP set D 5:
Choose ( s 0 , a 0 ) =
arg max ( s,a ) ∈S×A (cid:107) ( P M 1 − P M 2 ) ( · | s, a ) (cid:107)6:
Call Subroutine 2 with parameter ( s 0 , a 0 ) and n 0 to collect history samples H M 1 , M7: if ∃ s (cid:48) ∈ H M 1 , M 2 , P M 2 ( s (cid:48) | s 0 , a 0 )
= 0 or (cid:81)
s (cid:48)
∈H M 1 , M 2 P M 1 ( s (cid:48) | s 0 ,a 0 ) P M 2 ( s (cid:48) | s 0 ,a 0 ) ≥ 1 then 8:
Eliminate M 2 from the MDP set D 9: else 10:
Eliminate M 1 from the MDP set D 11: end if 12: end while 13: (cid:46) Stage 2: Run the optimal policy of M ∗ 14: Denote ˆ M as the remaining MDP in the MDP set D 15: Run the optimal policy of ˆ M for the remaining steps
Algorithm
Input: informative state-action pair ( s 0 , a 0 ) , required visitation count n
Initialize: counter N ( s 0 , a 0 ) = 0 , history data H = ∅
Denote π M ( s, s (cid:48) ) as the policy with the minimum expected travelling time E
[ T ( s (cid:48) | M , π, s )] for MDP M (Deﬁned in Assumption 1) while N ( s 0 , a 0 ) ≤ n 0 do 5: for i = 1 , · · · , M do
Denote the current state as s init
Run the policy π M i ( s init , s 0 ) for 2 D steps, breaking the loop immediately once the agent enters state send for if the agent enters state s 0 then 10:
Execute a 0 , enter the next state s (cid:48) .
counter N ( s 0 , a 0 ) =
N ( s 0 , a 0 ) + 1 ,
H = H (cid:83) { s (cid:48) } end if end while
Output: history data H algorithm for ﬁnite-horizon MDPs. By deriving an upper bound of the sim-to-real gap for ˆ π , we can upper bound Gap( π ∗ DR , U ) with Lemma 1. The policy ˆ π is formally deﬁned in Algorithm 1. There are two stages in Algorithm 1.
In the ﬁrst stage, the agent’s goal is to quickly explore the environment and ﬁnd the real MDP M ∗ from the
MDP set U . This stage contains at most M − 1 parts. In each part, the agent randomly selects two MDPs M 1 and M 2 from the remaining MDP set D .
Since the agent knows the transition dynamics of M 1 and M 2 , it can ﬁnd the most informative state-action pair ( s 0 , a 0 ) with maximum totalvariation difference between P M 1 ( ·| s 0 , a 0 ) and P M 2 ( ·| s 0 , a 0 ) .
The algorithm calls Subroutineto collect enough samples from ( s 0 , a 0 ) pairs, and then eliminates the MDP with less likelihood.
At the end of the ﬁrst stage, the MDP set D is ensured to contain only one MDP M ∗ with high probability.
Therefore, the agent can directly execute the optimal policy for the real MDP till step
H + 1 in the second stage.
Subroutine 2 is designed to collect enough samples from the given state-action pair ( s 0 , a 0 ) .
The basic idea in Subroutine 2 is to quickly enter the state s 0 and take action a 0 , until the visitation counter N ( s 0 , a 0 ) = n 0 .
Denote π M ( s, s (cid:48) )
as the policy with the minimum expected travelling time
E
[ T ( s (cid:48) | M , π, s )] for MDP M .
Suppose the agent is currently at state s init and runs the policy π M ∗ ( s init , s 0 ) in the following steps.
By Assumption 1 and Markov’s inequality, the agent will enter state s 0 in 2 D steps with probability at least 1 / 2 .
Therefore, in Subroutine 2, we runs the policy π M i ( s init , s 0 ) for 2 D steps for i ∈
[ M ] alternatively.
This ensures that the agent can enter state s 0 in 2 MD steps with probability at least 1 / 2 .
Theorem 5 states an upper bound of the sim-to-real gap for Algorithm 1, which is proved in Appendix D.1.
Theorem 5.
Suppose we use ˆ π to denote the history-dependent policy represented by Algorithm 1.
Under Assumption 1 and Assumption 2, for any M ∈ U , the sim-to-real gap of Algorithm 1 is at most
[Equation]
C. F INITE S IMULATOR C LASS WITHOUT S EPARATION C ONDITION
In this subsection, we propose an efﬁcient algorithm in the inﬁnite-horizon average-reward setting for ﬁnite simulator class.
Our algorithm is described in Algorithm 3.
In episode k , the agent executes the optimal policy of the optimistic MDP M k with the maximum expected gain ρ ∗M k .
Once the agent collects enough data and realizes that the current MDP M k is not M ∗ that represents the dynamics of the real environment, the agent eliminates M k from the MDP set.
Algorithm 3 Optimistic Exploration
Take action a h = π ∗M k ( s
) , obtain the reward R ( s h , a h ) , and observe the next s h +  if (cid:12)(cid:12)(cid:12)(cid:80) ht
= h 0 (cid:0) P M k λ ∗M k ( s t , a t ) − λ ∗M k
( s t +1 ) (cid:1)(cid:12)(cid:12)(cid:12) >
D (cid:112) 2( h − h 0 ) log(2 HM )
then 6:
Eliminate M k from the MDP set U k , denote the remaining set as U k + 7:
Calculate {+[Equation] Set h 
 = h + 1 , and+} k = k
+ 1 . 9: end if 10: end for
To indicate the basic idea of the elimination condition deﬁned in Line 5 of Algorithm 3, we brieﬂy explain our regret analysis of Algorithm 3.
Suppose the MDP M k selected in episode k satisﬁes the optimistic condition ρ ∗M k ≥ ρ ∗M ∗ , then the regret in H steps can be bounded as:
[Equation]
[Equation]
[Equation]
Here we use K to denote the total number of episodes, and we use τ ( k ) to denote the ﬁrst step of episode k .
The ﬁrst inequality is due to the optimistic condition ρ ∗M k ≥ ρ ∗M ∗ .
The ﬁrst equality is due to the Bellman equation in the ﬁnite-horizon setting (Eqn 10).
The last inequality is due to 0 ≤ λ ∗M ( s ) ≤ D .
From the above inequality, we know that the regret in episode k depends on the summation (cid:80) τ
( k +1)
− 1 h
= τ ( k )
[Equation]
λ ∗M k ( s h , a h ) − λ ∗M k
( s h +1 ) (cid:1) .
If this term is relatively small, we can continue following the policy π ∗M k with little loss.
Since λ ∗M k ( s h +1 ) is an empirical sample of P M ∗ λ
∗M k ( s h , a h ) , we can guarantee that M k is not M ∗ with high probability if this term is relatively large.
Based on the above discussion, we can upper bound the regret of Algorithm 3.
We defer the proof of Theorem 6 to Appendix E.1.
Theorem 6.
Under Assumption 1, the regret of Algorithm 3 is upper bounded by
[Equation]
C. I NFINITE S IMULATOR C LASS
In this subsection, we propose a provably efﬁcient model-based algorithm solving inﬁnite-horizon average-reward MDPs with general function approximation.
To the best of our knowledge, our result is the ﬁrst efﬁcient algorithm with near-optimal regret for inﬁnite-horizon average-reward
MDPs with general function approximation.
Considering the model class U which covers the true MDP M ∗ , i.e. M ∗ ∈
U , we deﬁne the function space Λ = { λ ∗M , M ∈ U} , and space X = S
×
A × Λ .
We deﬁne the function space
F = { f M ( s, a, λ ) :
X → R such that f M ( s, a, λ ) =
P M λ ( s, a ) for M ∈ U , λ ∈ Λ } .
Our algorithm, which is described in Algorithm 4, also follows the well-known principle of optimism in the face of uncertainty.
In each episode k , we calculate the optimistic MDP M k with maximum expected gain ρ ∗M k .
We execute the optimal policy of M ∗ to interact with the environment and collect more samples.
Once we have collected enough samples in episode k , we update the model class U k and compute the optimistic MDP for episode k + 1 .
Compared with the setting of episodic MDP with general function approximation (Ayoub et al., 2020; Wang et al., 2020; Jin et al., 2021), the additional problem in the inﬁnite-horizon setting is that the regret technically has linear dependence on the number of total episodes, or the number of steps that we update the optimistic model and the policy.
This corresponds to the last term ( KD ) in Inq 32.
Therefore, to design efﬁcient algorithm with near-optimal regret in the inﬁnite-horizon setting, the algorithm should maintain low-switching property (Bai et al., 2019; Kong et al., 2021).
Taking inspiration from the recent work that studies efﬁcient exploration with low switching cost in episodic setting (Kong et al., 2021), we deﬁne the importance score, sup
[Equation]
as a measure of the importance for new samples collected in current episode, and only update the optimistic model and the policy when the importance score is greater than 1 .
Here (cid:107) f 1 − f 2 (cid:107)
Z is a shorthand of (cid:80) ( s,a,s (cid:48) ,λ ) ∈Z ( f 1 ( s, a, λ ) − f 2 ( s, a, λ )) 2 . [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Equation]
[Table]
[Table]
[Table]
[Equation]
h
[Equation]
h
[Equation]
(cid:16)(cid:16)
[Equation]
(cid:27) 12:
Compute M k +
[Equation]
= arg max M∈U
[Equation]
Episode counter k = k
+14: end if 15: end for
We state the regret upper bound of Algorithm 4 in Theorem 5, and defer the proof of the theorem to
Appendix F.1.
Theorem 7.
Under Assumption 1, the regret of Algorithm 4 is uppder bounded by
[Equation]
where d e is the (cid:15) -eluder dimension of function class F with (cid:15)
= 1 H , and N ( F , 1 /H ) is the 1 Hcovering number of F w.r.t L ∞ norm.
For α > 0 , we say the covering number N ( F , α ) of F w.r.t the L ∞ norm equals m if there is m functions in F such that any function in F is at most α away from one of these m functions in norm (cid:107) · (cid:107) ∞ .
The (cid:107) · (cid:107) ∞
norm
of function
f is deﬁned as (cid:107) f (cid:107) ∞ def = max x ∈X | f ( x ) | .
[-+
-]
D O MITTED P ROOF FOR FINITE SIMULATOR CLASS WITH SEPARATION CONDITION D. P ROOF OF T HEOREM
The formal deﬁnition of ˆ π is given in Algorithm 1.
To upper bound the sim-to-real gap of ˆ π , we discuss on the following two benign properties of Algorithm 1 in Lemma 4 and Lemma 7.
Lemmastates that the true MDP M ∗ will never be eliminated from the MDP set D .
Therefore, in stage 2, the agent will execute the optimal policy in the remaining steps with high probability.
Lemmastates that the total number of steps in stage 1 is upper bounded by ˜ O
( M 2 δ 4 ) .
This is where the ﬁnal bound in Theorem 5 comes from.
Lemma 4.
With probability at least 1 − 1 H , the true MDP M ∗ will never be eliminated from the
MDP set D in stage 1.
The while-loop in stage 1 will last for M − 1 times.
To prove Lemma 4, we need to prove that, if the true MDP M ∗ is selected in a certain loop, then (cid:81) ( s,a,s (cid:48) )
∈H M 1 , M 2 P M∗
( s (cid:48) | s,a ) P M ( s (cid:48) | s,a ) ≥ 1 holds with high probability.
This is illustrated in the following lemma.
Lemma 5.
Suppose H = { s (cid:48) i } n
i
=1 is a set of n
= c 0 log 2 ( SMH/δ )log( MH )
δindependent samples from a given state-action pair ( s 0 , a 0 ) and MDP M ∗ for a large constant c 0 .
Let M 1 denote another MDP satisfying (cid:107) ( P M ∗ − P M 1 )( ·| s 0 , a 0 ) (cid:107) 1 ≥ δ , then the following inequality holds with probability at least 1 − 1 MH :
[Equation]
Proof.
The proof of Lemma 5 is inspired by the analysis in Kwon et al.
To prove Inq 36, it is enough to show that
[Equation]
holds with probability at least 1 − 1 MH .
Note that P M∗
( s (cid:48) | s 0 ,a 0 ) P M 1 ( s (cid:48) | s 0 ,a 0 ) can be unbounded since P M 1 ( s (cid:48) | s 0 , a 0 ) can be zero for certain s (cid:48) .
To tackle this issue, we deﬁne ˜ P M 1 for a sufﬁciently small α ≤ δ 4 S :
[Equation]
We have (cid:13)(cid:13)(cid:13)(cid:16)
˜
P M 1 − P M 1 (cid:17) ( ·| s, a ) (cid:13)(cid:13)(cid:13) ≤ 2 Sα ≤ δ 2 , thus (cid:13)(cid:13)(cid:13)(cid:16) ˜
P M 1 − P M ∗ (cid:17) ( ·| s, a ) (cid:13)(cid:13)(cid:13) ≥ δ 2 .
Also, we have ln (cid:16)
[Equation]
M 1 ( s (cid:48) | s 0 ,
[Equation]
[Equation]
(cid:17) ≤ ln(1 /α ) for any s (cid:48) ∈ S .
With the above deﬁnition, we can decompose
Inq 37 into two terms:
[Equation]
[Equation]
Taking expectation over s (cid:48) ∼ P M ∗ ( ·| s, a ) for the ﬁrst term, we have E
[Equation]
where the last inequality is due to Pinsker’s inequality.
Lemma 6.
(Lemma C.2 in Kwon et al.
Suppose X is an arbitrary discrete random variable on a ﬁnite support X .
Then, ln(1 /P ( X )) is a sub-exponential random variable (Vershynin, 2010) with Orcliz norm ln(1 /P
( X ))
φ 1 = 1 /e .
From the above Lemma, we know that both ˜ P M 1 ( s (cid:48) | s 0 , a 0 ) and P M ∗ ( s (cid:48) | s 0 , a 0 ) are sub-exponential random variables.
By Azuma-Hoefﬁng’s inequality, we have with probability at least 1 − δ 0 / 2 , (cid:88)
[Equation]
By Proposition 5.16 in Vershynin (2010), with probability at least 1 − δ 0 / 2 ,
[Equation]
for a certain constant c > 0 .
Therefore, we can lower bound the ﬁrst term in Eqn 39, (cid:88)
[Equation]
with probability at least 1 − δ 0 .
For the second term in Eqn 39, by the deﬁnition of ˜ P M 1 , (cid:88)
[Equation]
Combining Inq 45 and Inq 46, we have
[Equation]
Setting α = δ 2 8 S , δ 0 = 1 MH , and n 0 = c 0 log 2 ( SMH/δ )log( MH ) δ, we have
[Equation]
(48) holds with probability at least 1 − 1 MH .
Lemma 7.
Suppose Stage 1 in Algorithm 1 ends in h 0 steps.
We have E [ h
[Equation]
O
[Equation]
) , where the expectation is over all randomness in the algorithm and the environment.
Proof.
Recall that π M ( s, s (cid:48) ) is the policy with the minimum expected travelling time
E
[ T ( s (cid:48) | M , π, s )] for MDP M .
By Assumption 1, we have
[Equation]
Given state s and s (cid:48) , by Markov’s inequality, we know that with probability at least 12 ,
[Equation]
Consider the following stochastic process:
In each episode k , the agent starts from a state s k that is arbitrarily selected, and run the policy π M ∗ ( s k , s 0 ) for 2 D steps on the MDP M ∗ .
The process terminates once the agent enters a certain state s 0 .
By Inq 50, the probability that the process terminates within k episodes is at least 1 − 12 k .
By the basic algebraic calculation, the expected stopping episode can be bounded by a constant 4 .
Now we return to the proof of Lemma 7.
In Subroutine 2, we run policy π M i for each MDP
M i ∈
D alternately.
By Lemma 4, the true MDP M ∗ is always contained in the MDP set D .
Therefore, the expected travelling time to enter state s 0 for n 0 times is bounded by n 0 · M · 8 D .
In stage 1, we call Subroutine 2 for M − 1 times, which means that the expected steps in stagesatisﬁes E
[ h 0 ] ≤ 8 M 2 n 0 d
= O ( DM
[Equation]
[Equation] [Equation]
[Equation]
Proof.
(Proof of Theorem 5) Recall that we use h 0 to denote the total steps in stage 1.
Firstly, we prove that Gap(ˆ π, U ) is upper bounded by O ( E [ h 0 ] + D ) .
[Equation]
[Equation]
The outer expectation is over all possible h 0 , while the inner expectation is over the possible trajectories given ﬁxed h 0 .
By Lemma 4, we know that ˆ π = π ∗ DR after h 0 steps with probability at least 1 − 1 H .
If this high-probability event happens, the second part in the above inequality equals to
[Equation]
We can prove that this term is upper bounded by 2 D .
Given ﬁxed
[Equation]
[Equation]
is the difference of the value function in step h
+ 1 under two different distribution of s h 0 +1 .
We use d h ( s, π ) to denote the state distribution in step h after starting from state s in step h 0 + 1 following policy π .
[Equation]
[Equation]
where the second equality is due to the Bellman equation in inﬁnite-horizon setting (Eqn 10).
Note that by the communicating property, we have 0 ≤ λ ∗M ∗ ( s ) ≤ D for any s ∈ S .
Therefore, we have
[Equation]
If the high-probability event deﬁned in Lemma 4 does not hold (This happens with probability at most 1 H )
[Equation]
we still have E M
[Equation]
,π
[Equation]
thus this does not inﬂuence the ﬁnal bound.
Taking expectation over all possible h 0 , and plugging the result back to Inq [Equation] we have
[Equation]
D. P ROOF OF T HEOREM
Proof.
The theorem can be proved by combining Theorem 5 and Lemma 1.
By Theorem
[Equation]
we prove that the constructed policy ˆ π satisﬁes
[Equation]
By Lemma 1, the sim-to-real gap of policy π ∗ DR is bounded by
[Equation]
E O MITTED PROOF FOR FINITE SIMULATOR CLASS WITHOUT SEPARATION CONDITION E. P ROOF OF T HEOREM
Lemma 8.
(Optimism)
With probability at least 1 − 1 MH , we have ρ ∗M k ≥ ρ ∗M ∗ for any k ∈ [ K ] .
Proof.
For any ﬁxed M ∈ U , and ﬁxed step h ∈ [ H ] , by Azuma’s inequality, we have with probability at least 1 −
[Equation]
Taking union bounds over all possible M and h , we know that the above event holds for all possible
M and h with probability 1 − 1 MH .
Under the above event, the true MDP M ∗ will never be eliminated from the MDP set U k .
Therefore, we have ρ ∗M k ≥ ρ ∗M ∗ .
Proof.
(Proof of Theorem 6)
By Lemma 8, we know that ρ ∗M k ≥ ρ ∗M ∗ for any k ∈ [ K ] .
We use τ ( h ) to denote the episode that step h belongs to.
We can upper bound the regret in H steps as follows:
[Equation]
By Azuma’s inequality
[Equation]
we know that
[Equation]
holds with probability at least 1 − 1 MH .
Since 0 ≤ λ ∗ M ≤ D [Equation] we have P M ∗ λ ∗ M τ ( h ) ( s h 0 , a h 0 ) − λ ∗ M τ ( h ) ( s 1 ) ≤ D .
Therefore, we have
[Equation]
The ﬁrst term in (71) is bounded by the stopping condition (line 5 of Algorithm 3).
By Lemma 2, we have V ∗M ∗ , 1 ( s 1 )
≤
Hρ ∗M ∗ + D .
Therefore, we have V ∗M ∗ , 1 ( s 1 ) − (cid:80)
Hh =1 R ( s h , a h ) ≤
O (cid:16) D (cid:112) MH log( MH ) (cid:17) with probability at least 1 − 2 MH .
Therefore, we have
[Equation]
E. P ROOF OF T HEOREM
Theorem 2 can be proved by combining Theorem 6 , Lemma 1 and Lemma 2.
By Theorem 6, for any M ∈ U , the policy ˆ π represented by Algorithm 3 has regret bound Hρ ∗M − (cid:80)
Hh =1 R ( s h , a h ) ≤
O (cid:16) D (cid:112) MH log( MH ) (cid:17) .
Taking expectation over { s h , a h } h ∈ [ H ] and combining the inequality with Lemma 2, we have for any M ∈ U .
[Equation]
Then the theorem can be proved by Lemma 1.
F O MITTED P ROOF FOR I NFINITE S IMULATOR C LASS F. P ROOF OF T HEOREM
Lemma 9.
(Low Switching)
The total number of episode K is bounded by
[Equation]
Proof.
By Lemma 5 of Kong et al.
(2021), we know that
[Equation]
for some constant C > 0 .
Our idea is to use this result to upper bound the number of total switching steps.
Let τ ( k ) be the ﬁrst step of episode k .
By the deﬁnition of the function class F , we have ( f 1 − f 2 ) 2 ( x t ) ≤ 4 D 2 for any f 1 , f 2 ∈ F and x t .
By the switching rule, we know that, once the agent starts a new episode after step τ ( k + 1) − 1 , we have
[Equation]
Therefore, we have
[Equation]
Now we upper min [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
Suppose the number of episodes is at most K .
If we set α = 4 D 2 + 1 , we have
K [Table]
[Equation]
By the switching rule, we have sup
[Equation]
≥ 1 .
Therefore, the LHS of the above inequality is exactly K .
Thus we have
[Equation]
Lemma 10.
(Optimism)
With probability at least 1 − 1 H , M ∗ ∈
U k holds for any episode k ∈
[ K ] .
Proof.
This lemma comes directly from Theorem 6 of Ayoub et al.
Deﬁne the Filtration F =
( F h ) h>
so that F h − 1 is generated by ( s 1 , a 1 , λ 1 , · · · , s h , a h , λ h ) .
Then we have E [ λ h ( s h +1 ) | F h − 1 ] =
P M ∗ λ h ( s h , a h ) =
f M ∗ ( s h , a h , λ h ) .
Meanwhile, λ h ( s h +1 )
− f M ∗ ( s h , a h , λ h ) is conditionally D 2 -subgaussian given F h − 1 .
By Theorem 6 of Ayoub et al.
, we can directly know that f M ∗ ∈
U k for any k ∈
[ K ] with probability at least 1 − 1 H .
Proof.
(Proof of Theorem 7) Let τ ( k ) be the ﬁrst step of episode k .
Under the high-probability event deﬁned in Lemma 10, we can decompose the regret using the same trick in previous sections.
[Equation]
where the ﬁrst inequality is due to optimism condition in Lemma 10.
The ﬁrst equality is due to the
Bellman equation 10 and λ h
= λ ∗M k for τ ( k ) ≤ h ≤ τ ( k + 1) − 1 .
The last inequality is due to 0 ≤ λ h ( s ) ≤ D for any s ∈ S .
Now we bound the ﬁrst two terms in Eqn 90.
The second term can be regarded as a martingale difference sequence.
By Azuma’s inequality, with probability at least 1 − 1 H , K
[Equation]
Now we focus on the upper bound of the ﬁrst term in Eqn 90.
Under the high-probability event deﬁned in Lemma 10, the true model P is always in the model class U k .
For episode k , from the construction of U k we know that any f 1 , f 2 ∈
U k satisﬁes (cid:107) f 1 − f 2 (cid:107) 2 Z τ ( k ) ≤ 2 β .
Since M k , M ∗ ∈
U k , we have
[Equation]
Moreover, by the if condition in Line 8 of Alg. 4, we have for any τ ( k ) ≤ h ≤ τ ( k + 1) − 1 ,
[Equation]
Summing up the above two equations, we have
[Equation]
We invoke Lemma 26 of Jin et al.
(2021) by setting G = F − F , Π =
{ δ x ( · ) | x ∈ X} where δ x ( · ) is the dirac measure centered at x , g t = f M k − f M ∗ , ω
= 1 /H , β = 4 β + α + D 2 and µ t = 1 {· = ( s t , a t , λ t ) } , then we have
[Equation]
Plugging the results back to Eqn 90, we have
[Equation]
By Lemma
[Equation]
we have V ∗M ∗ , 1 ( s 1 )
≤
Hρ ∗M ∗ + D .
Therefore, we have
[Equation]
with probability at least 1 − 2 MH .
If the high-probability event doesn’t holds (This happens with probability at most 2 H ), then the gap V ∗M ∗ , 1 ( s 1 ) − V π ∗ DR M ∗ , 1 ( s 1 ) still can be bounded by H .
Taking expectation over the trajectory { s h } h , we have
[Equation]
F. P ROOF OF T HEOREM
Proof.
Theorem 4 can be proved by combining Theorem 7 , Lemma 1 and Lemma 2.
By Theorem 7, for any M ∈ U , the policy ˆ π represented by Algorithm 4 can obtain regret bound Hρ ∗M − (cid:80)
Hh =1 R ( s h , a h ) ≤ O (cid:16) D (cid:112) d e H log( H · N ( F , 1 /H )) (cid:17) .
Taking expectation over { s h , a h } h ∈ [ H ] and combining the inequality with Lemma 2, we have for any M ∈ U .
[Equation]
[Equation]
Then the theorem can be proved by Lemma 1.
G L OWER B OUNDS G. P ROOF OF P ROPOSITION
Proof.
Consider the following construction of U .
There are 3 M + 1 states.
There are M actions in the initial state s 0 , which is denoted as { a i } Mi =1 .
After taking action a i in state s 0 , the agent will transit to state s
i, 1 with probability 1 .
In state s i, 1 for i ∈
[ M ] , the agent can only take action a 0 , and then transits to state s i, 2 with probability p i , and transits to state s i, 3 with probability 1 − p i .
State { s i, 2 } Mi =1 and { s i, 3 } Mi =1 are all absorbing states.
That is, the agent can only take one action ain these states, and transits back to the current state with probability 1 .
The agent can only obtain reward 1 in state s i, 2 for i ∈
[ M ] , and all the other states have zero rewards.
Therefore, if the agent knows the transition dynamics of the MDP, it should take action a i with i = arg max i p i in state s 0 .
Now we deﬁne the transition dynamics of each MDP M i .
For each MDP M i ∈ U , we have p i =and p j = 0 for all j ∈ [ M ] , j (cid:54)
=
i .
Therefore, the agent cannot identify M ∗ in state s 0 .
The best policy in state s 0 for latent
MDP U is to randomly take an action a i .
In this case, the sim-to-real gap can be at least Ω( H ) since the agent takes the wrong action in state s 0 with probability 1 − 1 M .
G. P ROOF OF T HEOREM
Proof.
We ﬁrst show that Ω( √ MH ) holds with the hard instance for multi-armed bandit (Lattimore & Szepesv´ari, 2020).
Consider a class of K-armed bandits instances with K = M .
For the bandit instance M i , the expected reward of arm i is 12 + (cid:15) , while the expected rewards of other arms are 12 .
Note that this is exactly the hard instance for K -armed bandits.
Following the proof idea of the lower bound for multi-armed bandits, we know that the regret (sim-to-real gap) is at least Ω( √ MH ) .
We restate the hard instance construction from Jaksch et al.
This hard instance construction has also been applied to prove the lower bound in episodic setting (Jin et al., 2018).
We ﬁrstly introduce the two-state MDP construction.
In their construction, the reward does not depend on actions but states.
State 1 always has reward 1 and state 0 always has reward 0.
From state 1, any action takes the agent to state 0 with probability δ , and to state 1 with probability 1 − δ .
In state 0, there is one action a ∗ takes the agent to state 1 with probability δ + (cid:15) , and the other action a takes the agent to state 1 with probability δ .
A standard Markov chain analysis shows that the stationary distribution of the optimal policy (that is, the one that takes action a ∗ in state 0) has a probability of being in state 1 of
[Equation]
In contrast, acting sub-optimally (that is taking action a in state 0) leads to a uniform distribution over the two states.
The regret per time step of pulling a sub-optimal action is of order (cid:15)/δ .
Consider O ( S ) copies of this two-state MDP where only one of the copies has such a good action a ∗ .
These copies are connected into a single MDP with an A -ary tree structure.
In this construction, the agent needs to identify the optimal state-action pair over totally SA different choices.
Setting δ = 1 D and (cid:15)
=
c (cid:113) SATD , Jaksch et al.
(2010) prove that the regret in the inﬁnite-horizon setting is Ω( √ DSAT ) .
Our analysis follows the same idea of Jaksch et al.
For the MDP instance M i , the optimal state-action pair is ( s i , a i ) ( ( s i , a i ) (cid:54) =
( s j , a j ) , ∀ i (cid:54) = j ).
With the knowledge of the transition dynamics of each M i , the agent needs to identify the optimal state-action pair over totally M different pairs in our setting.
Therefore, we can similarly prove that the lower bound is Ω( √ DMH ) following their analysis.
G. L OWER B OUND IN THE L ARGE S IMULATOR C LASS
Proposition 2.
Suppose All MDPs in the MDP set U are linear mixture models (Zhou et al., 2020) sharing a common low dimensional representation with dimension d = O (log( M )) , there exists a hard instance such that the sim-to-real gap of the policy π ∗ DR returned by the domain randomization oracle can be still Ω( H ) when M ≥ H .
Proof.
We can consider the following linear bandit instance as a special case.
Suppose there are two actions with feature φ ( a 1 ) =
(1 , 0) and φ ( a 2 ) =
In the MDP set M , there are M − 1 MDPs with parameter θ i =
( 12 , − p i ) with 14 < p i < 12 , i ∈ [ M − 1] , and one MDP with parameter θ M =
Suppose M = 4 H + 5 , the optimal policy of such an LMDP with uniform initialization will never pull the action a 2 , which can suffer Ω( H ) sim-to-real gap in the MDP with parameter θ M .