T ARGETED
E NVIRONMENT
D ESIGN
FROM
O FFLINE
D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
In reinforcement learning (RL) the use of simulators is ubiquitous, allowing cheaper and safer agent training than training directly in the real target environment.
However, this approach relies on the simulator being a sufﬁciently accurate reﬂection of the target environment, which is difﬁcult to achieve in practice, resulting in the need to bridge sim2real gap.
Accordingly, recent methods have proposed an alternative paradigm, utilizing ofﬂine datasets from the target environment to train an agent, avoiding online access to either the target or any simulated environment but leading to poor generalization outside the support of the ofﬂine data.
We propose to combine the two paradigms: ofﬂine datasets and synthetic simulators, to reduce the sim2real gap by using limited ofﬂine data to train realistic simulators.
We formalize our approach as ofﬂine targeted environment design (OTED), which automatically learns a distribution over simulator parameters to match a provided ofﬂine dataset, and then uses the learned simulator to train an RL agent in standard online fashion.
We derive an objective for learning the simulator parameters which corresponds to minimizing a divergence between the target ofﬂine dataset and the state-action distribution induced by the simulator.
We evaluate our method on standard ofﬂine
RL benchmarks and show that it learns using as few as 5 demonstrations, and yields up to 17 times higher score compared to strong existing ofﬂine RL, behavior cloning (BC), and domain randomization baseline, thus successfully leveraging both ofﬂine datasets and simulators for better RL.
I NTRODUCTION
Recent years have witnessed the proliferation of reinforcement learning (RL) as a way to train agents to solve a variety of tasks, encompassing game playing (Brown & Sandholm, 2019; Silver et al., 2016) and some robotics applications (Akkaya et al., 2019; Schulman et al., 2015; Chiang et al., 2019), among others.
These successes often rely on the use simulators, providing several beneﬁts to online RL agent training: large amounts of low cost and more diverse data not readily available in reality (Tolani et al., 2021), and safe training that avoiding physical damage (Krishnan et al., 2021) or human privacy concerns (Tolani et al., 2021).
In many instances, for example in game playing, an accurate simulator is easy to devise.
In others, rough approximations of the real target environment are sufﬁcient (e.g. Chiang et al.
to close sim2real gap.
However, in other applications such as robot manipulation (Kalashnikov et al., 2018; Todorov et al., 2012), health (Ernst et al., 2006), or web navigation (Gur et al., 2021) designing the simulator to match a desired target environment with sufﬁcient ﬁdelity can be a highly manual process (Nachum et al., 2019a; Krishnan et al., 2021), potentially requiring repeated online access to the target environment (Chebotar et al., 2019; Ramos et al., 2019; Mehta et al., 2020).
Due to the difﬁculty in matching simulators to target environments and motivated by the successes of supervised learning on large static datasets, a recent line of works Fujimoto et al.
Agarwal et al.
; Wu et al.
; Levine et al.
Fu et al.
; Gulcehre et al.
(2020) has proposed to circumvent online training altogether (whether on the simulator or in the target environment) and instead perform RL on an ofﬂine dataset of logged experience in the target environment, typically collected by an unknown behavior policy.
This approach avoids any potential mismatch between training and test environment.
However, existing algorithms for ofﬂine RL are highly liable to generalization errors, and so these methods often impose a strong regularization on the learned policy to maintain proximity to the ofﬂine dataset (Fujimoto et al., 2019; Wu et al., 2019; Kostrikov et al.,
; Yu et al., 2020).
Thus, such methods can rarely approach the performance of online RL on an accurate simulator.
Here, we propose to leverage both approximate simulators and ofﬂine datasets to train agents.
Namely, we consider settings in which one has access to an ofﬂine dataset collected from a target environment as well an approximate (but nevertheless inaccurate) simulator.
How can we leverage both of these to learn a near-optimal policy for the target environment?
To tackle this problem, we propose an approach which, at a high level, is composed of two stages: (1) use the ofﬂine dataset to learn a distribution over simulator parameters; (2) train an RL agent in an online fashion on the simulator using any off-the-shelf RL algorithm.
The key question is then how to appropriately learn the simulator parameters from an ofﬂine dataset, which we term ofﬂine targeted environment design (OTED).
For this problem, we devise an objective quantifying the divergence between the state-action distribution appearing in the ofﬂine dataset and the state-action distribution induced by a learned behavior policy acting in the simulator.
We show that the simulator parameters and the target behavior policy can be jointly learned such that even a suboptimal behavior policy, that would otherwise perform poorly on the target environment, can help learn simulator parameters very effectively.
We further show how the performance of the ﬁnal returned policy can be improved by employing OTED not only for policy training but also for ofﬂine policy selection (Yang et al., 2020;
Fu et al., 2021).
Namely, we show how one may use the OTED-learned simulators to both provide a set of candidate policies as well as evaluate those policies in order to rank and choose the best policy to return, reminiscent of cross-validation techniques popular in supervised learning (Arlot & Celisse, 2010) but so far elusive in RL contexts (Paine et al., 2020).
We apply our method to a variety of domains, including D4RL (Fu et al., 2020) and MiniGrid (Chevalier-Boisvert et al., 2018), showing that our method is able to recover the target environment parameter using as few as 5 demonstrations and learn from high-dimensional image observations.
We also show that an online agent trained in designed simulators yields improved performance compared to methods which leverage only simulators or only ofﬂine datasets, as well as existing methods for domain transfer (Tobin et al., 2017).
On the D4RL (Fu et al., 2020) benchmark, we demonstrate that our method can even outperform an online SAC agent trained on the ground-truth target environment.
Our method achieves up to 17 times higher score compared to previous state-of-the-art ofﬂine RL and behavior cloning (BC) methods as well as domain randomization.
On datasets with medium-level behavior that can be collected without domain experts, we show more relative improvements where our model even reaches the performance of the ofﬂine RL methods trained on expert-level behaviors.
R ELATED W ORK
Our work focuses on learning simulator parameters to match a target environment and is thus related to a large and diverse literature on domain transfer in RL.
One of the most common and simple approaches to handling unknown simulator parameters is domain randomization (Sadeghi & Levine, 2016;
Tobin et al., 2017; Matas et al., 2018; Chiang et al., 2019;
Tolani et al., 2021; Krishnan et al., 2021), which randomizes the setting of those unknown parameters during training, thus learning an agent which is robust to a wide distribution of possible simulators.
While domain randomization has demonstrated successes, the process of choosing appropriate ranges for unknown parameters can be a highly manual process (Andrychowicz et al., 2020), in which one must balance between either too wide range that hampers agent training, and too narrow range that may exclude the target environment parameterization (Nachum et al., 2019a).
While some works have proposed heuristic mechanisms to better automate this process (Akkaya et al., 2019; Krishnan et al., 2021), others have explored incorporating online interaction with the environment into this tuning process (Chebotar et al., 2019; Ramos et al., 2019; Mehta et al., 2020).
In this work, we avoid any online interactions with the target environment during training, and only assume access to a static ofﬂine dataset.
The proposed OTED method, which uses an ofﬂine dataset to learn a simulator, is related to recent model-based approaches to ofﬂine RL (Matsushima et al., 2020;
Yu et al., 2020; Argenson &
Dulac-Arnold, 2020).
In these works, the ofﬂine dataset learns approximate dynamics and reward models of the environment, and the agent optimizes behavior in this approximate model of the environment.
Although this makes minimal assumptions on the target environment, using reward and dynamics models as proxies for the environment leads to extrapolation issues which necessitate strong regularizers on the learned policy.
Moreover, these existing approaches mostly ignore the fact that in many scenarios, one already has an approximate simulator, albeit not differentiable; e.g., the physical laws in a continuous control environments are straightforward to implement (Todorov et al., 2012), although parameters more speciﬁc to the environment (friction coefﬁcient, actuator gain, gravity) are unknown.
Leveraging these simulators and focusing learning on the unknown parameters, as performed by our approach, can thus improve over modelling the full dynamics from scratch.
Our proposed objective is based on a distribution matching loss.
Similar losses have appeared in the past, especially in generative models (Goodfellow et al., 2014), off-policy evaluation (Nachum et al., 2019b), off-policy learning (Nachum et al., 2019c), and imitation learning (Ho & Ermon, 2016;
Stadie et al., 2017; Kostrikov et al., 2020).
These predominantly focus on learning a policy via an adversarial objective and ignore the mismatch between simulators and real environments.
Even when the distribution matching objective is optimal, this mismatch might give very suboptimal policies, due to potential poor support of the data distribution.
This is a very realistic scenario since tuning simulator parameters accurately is a very tedious process and requires domain-experts.
Finally, we emphasize that our proposed approach is distinct from previous work which suggests to combine ofﬂine learning on static datasets with online learning on the target environment (Nair et al., 2020;
Ajay et al., 2020; Yang & Nachum, 2021;
Xie et al., 2021; Lee et al., 2021).
Speciﬁcally, we avoid any online interactions with the target environment, and only allow online training on the simulator, which may or may not be a good approximation to the target environment.
B ACKGROUND
We consider an RL setting in which we are given access to both an ofﬂine dataset of experience from the target environment and an approximate simulator, in which the agent can gather more experience.
We elaborate on the notation and background relevant to this setting.
Reinforcement Learning We deﬁne an environment as a Markov Decision Process (MDP) given by a tuple ( S, A, ⇢ 0 , T, r,  ) where S is a state space, A is an action space, ⇢ 0 is an initial state distribution, T ( s 0 | s, a ) is a state transition distribution, r : S ⇥ A !
R is a reward function over state and actions, and  2 [0 , 1) is a discount factor.
A stationary policy ⇡ in this environment is a function from states to distributions over actions.
In reinforcement learning, we are interested in ﬁnding a policy ⇡ that maximizes the cumulative discounted returns: E [[-(cid:80) ∞ t =
-]
{+P 1 t =
 +} t r ( s t , a t ) | s 0 ⇠ ⇢ 0 , a t ⇠ ⇡ ( s t ) , s t +1 ⇠ T ( s t , a t )] .
We will use d ⇡ ( s, a ) to denote the state-action distribution of ⇡ : d ⇡ ( s, a ) :
[-t =
-]= (
[Equation]   ) [Equation]
Simulator vs. Target Environment Typically in online reinforcement learning, a policy is trained based on experience collected by the agent in a simulator .
While this implicitly assumes that the simulator can perfectly mimic the target environment, in practice this is often not the case.
The simulator typically only provides an approximation to the true initial state distribution ⇢ 0 and state transition distribution T .
For example in robotics applications, while physics simulators can be built, it is infeasible to accurately set all appropriate parameters (e.g., friction coefﬁcients, actuator gains, wind speed).
In this work, we assume such a parameterized simulator exists, and it is given by M ( z ) :
= ( S, A, ⇢ 0 ( ·| z ) , T ( ·|· , · , z ) , r ( z ) ,  ) , where z denotes the parameters of the simulated environment.
Naive approaches would either set z heuristically or using randomization methods.
In this work, we aim to use ofﬂine datasets to learn z so that M ( z ) approximates the target MDP, which we denote as M ⇤ .
We use d ⇡ ( s, a | z ) to denote the state-action distribution of ⇡ in the parameterized simulator M ( z ) .
Ofﬂine Reinforcement Learning Ofﬂine RL, as an alternative to online RL on simulators, assumes that an agent cannot collect any online experience and is restricted to a dataset D of ( s, a, s 0 ) collected by an unknown behavior policy µ ; i.e., ( s, a ) ⇠ d µ , s {+
 ⇠+} p ( s, a ) .
We use d D ( s, a ) to denote the distribution over state-action pairs appearing in D .
When trained on a ﬁxed dataset, previous state-ofthe-art online reinforcement learning algorithms such as SAC (Haarnoja et al., 2019) underperform due to extrapolation errors (Fujimoto et al., 2019; Kumar et al., 2019), which stem from sparsity of the samples in the dataset.
Accordingly, many RL algorithms designed speciﬁcally for the ofﬂine setting make use of strong regularizations to maintain proximity to the ofﬂine dataset (Wu et al., 2019; Kostrikov et al., 2021; Yu et al., 2020).
OTED: O FFLINE T ARGETED E NVIRONMENT D ESIGN
We now describe our method for leveraging both ofﬂine datasets and simulators to learn returnmaximizing policies, thus alleviating both of the above issues: (i) data sparsity and extrapolation issues when performing ofﬂine learning and (ii) inaccuracy of an approximate simulator for online learning.
At a high-level, our algorithm can be summarized as follows, also illustrated in Figure 1.
Learn an approximate behavior policy ˆ µ from D using any off-the-shelf imitation learning algorithm; e g., behavioral cloning (Pomerleau, 1991) or ValueDICE (Kostrikov et al., 2020).
Learn a distribution q ( z ) , also called designer , over simulator parameters so that d ˆ µ ( s, a | z ) approximates the state-action distribution appearing in D . 3.
Use any off-the-shelf online RL algorithm – e.g., SAC (Haarnoja et al., 2019) – to learn a return-maximizing policy ⇡ on the simulator M ( z ) , where z is sampled anew from the learned distribution q ( z ) at the beginning of each episode.
Steps (1) and (3) above are straightforward, and so we focus this section on ﬁrst elaborating on step (2) below, and this is also summarized in Algorithm 1.
We then continue to show how we reduce variance inherent in the procedural three-step process above by using replicated experiments, distributions over mixtures of simulator parameters, and model selection without online access.
4. L EARNING S IMULATOR P ARAMETERS VIA D ISTRIBUTION M ATCHING
Given an approximate behavior policy ˆ µ , we propose to learn a distribution q ( z ) over simulator parameters so that d ˆ µ ( s, a | z ) approximates the state-action distribution appearing in D .
Thus, our objective for q ( z ) is based on a probability divergence between d ˆ µ ( s, a | z ) and d D ( s, a ) .
To encourage better generalization, we also introduce a prior p ( z ) , and measure a divergence between d ˆ µ ( s, a | z ) q ( z ) and d D ( s, a ) p ( z ) .
Our proposed objective is given by,
[Equation]
As desired, it is clear that the ﬁrst term above encourages the parameter distribution q ( z ) to match the state-action distribution of the approximate behavior policy ˆ µ in the simulator to that of the dataset; meanwhile, the second term regularizes the simulator distribution from collapsing.
We can immediately see that the objective above is an RL (actually, a bandit) objective for q , where the actions are z , the stochastic reward is given by log d D ( s,a ) d ˆ µ ( s,a | z ) for ( s, a ) ⇠ d ˆ µ ( s, a | z ) , and a relative entropy regularizer is applied to q .
Therefore, one may use any off-the-shelf policy gradient algorithm to learn q , as long as one has access to the density ratios d D ( s,a ) d ˆ µ ( s,a | z ) .
However, in general one does not have access to the densities d ˆ µ or d D explicitly.
Nevertheless, one can easily sample from either density; samples from d ˆ µ are given by running the simulator
M ( z ) and samples from d D are given by the dataset D itself.
A number of algorithms exist for estimating density ratios of two distributions given sampling access.
In our implementation, we use the discriminative approach based on (Nguyen et al., 2010).
Namely, we train a discriminator g ( s, a ) to minimize the objective
[Equation]
similar to the discriminator in GAIL (Ho & Ermon, 2016).
When trained to optimality, the discriminator will satisfy the following, log g ⇤
[Equation]
s
[Equation]
, and this can be used as a reward signal for training q .
In practice, to allow for more ﬂexibility in the trade-off between exploitation and regularization, we extend the objective with a tunable parameter  :
[Equation]
A small  encourages q to exploit the signal from the log-ratios, while a large  encourages q to stay close to the prior p .
In addition to serving as a trade-off parameter,  can also be interpreted as (i) a reward scaling term in front of log ratios, i.e., 1  log d D ( s,a ) d µ ( s,a | z ) or (ii) a distribution smoothing term applied to both d µ and d D , i.e., log ( d D ( s,a ))
1  ( d µ ( s,a | z )) 1  .
Similar to (Kostrikov et al., 2019), we use samples stored in a replay buffer rather than online samples to improve sample efﬁciency.
In practice, we adopt the commonly used approximation of log ratios:  log (1  g ( s, a )) instead of  log (1  g ( s, a ))  log ( g ( s, a )) in Eq.
Input: Dataset D, initial behavior policy ˆ µ , initialized discriminator g , tunable simulator S 1: Gather experience E , a sequence of ( s, a ) pairs, by running the policy ˆ µ in the simulator S .
{+2: Estimate discriminator output,  log(
  g ( s, a )) for each sample ( s, a ) in+} the experience E . 3: Update q ( z ) using the objective in Eq. 3 with the average  log(1  g ( s, a )) as the reward. 4: Update g ( s, a ) using the objective in Eq.
2 with mini-batches sampled from ( E, D ) .
5: Repeat above.
Input: Simulator distributions q ( ·| h i ) from n different experiments, simulator S , empty replay R 1: Sample i ⇠ U [1 , n ] and set the simulator S using the mode z
i of the distribution q ( ·| h i ) .
2: Run the online agent ⇡ ( a | s ) in the simulator and populate the replay with new
Our objective, Eq. 1, resembles mutual information when the joint distribution is deﬁned using the proposal distribution and marginal distributions are deﬁned using expert data distribution.
The main difference is that we use the behavior policy in the joint distribution and use expert policy in marginals while in mutual information they are the same.
Through optimizing the behavior policy to mimic the expert better, the objective will get closer to a mutual information formulation.
4. R EDUCING V ARIANCE WITH R EPLICATED E XPERIMENTS
Our proposed three step process – approximate ˆ µ , then learn q ( z ) , and ﬁnally training – introduces noise at each step due to randomness in optimizers and learning algorithms.
It is important to reduce the variance inherent in this process so that the returned policy is best prepared for evaluation in the target environment.
We tackle this problem via two techniques: simulator selection and simulator-based policy selection .
Simulator selection As is typical, we train the objective in Eq.
3 using different hyper parameter settings where each setting will give a different simulator model.
Since the target parameter distributions are unknown, this raises the question – how do we choose the ideal set of hyperparameters?
We propose to use the ﬁrst term in Eq. 1, namely expected log ratios, to score and rank candidate simulators trained with different hyperparameters.
We detail our procedure to select simulator hyperparameters in below steps:
Given a hyperparameter setting h , replicate it with n different seeds and train a new OTED for each seed.
This will output a design distribution q ( .
| h i ) , a discriminator g ( .
| h i ) , and an approximate behavior policy ˆ µ ( .
| h i ) , where we use h i to denote the use of hyperparameters h with seed i . 2.
Run approximate behavior policies ˆ µ ( .
| h i ) in their corresponding candidate simulators M ( z ) where z ⇠ q ( .
| h i ) and collect experience E . 3.
Estimate discriminator outputs, c ( h i ) =
| E | P ( s,a ) ⇠ E  log(1  g ( s, a | h i )) to score h i . 4.
Score each hyperparameter h by averaging scores of their corresponding seeds, c ( h )
= n P c ( h i ) .
Pick the hyperparameter with the highest score c ( h ) , corresponding to the simulator with the lower approximated KL divergence from the ofﬂine data.
Simulator-based policy selection Even once a speciﬁc simulator is chosen according to the above procedure, training an online RL agent on the simulator can result in drastically different policies, again due to randomness in the learning process.
This presents the problem of ofﬁne policy selection, which is a well-known challenge for real-world RL (Yang et al., 2020; Paine et al., 2020; Fu et al., 2021).
We propose a simple approach to this, described in below steps:
Sample design parameters z i (mode of the q ( .
| h i ) )
for each seed from the selected hyperparameter h in the above procedure.
Train m seeded on-policy agents ⇡ j on the distribution of simulators M ( z i ) (see Algorithm 2).
Evaluate the performance of policies ⇡ j in each of the simulators M ( z i ) .
Select the top performing policies according to their achieved the rewards averaged over all simulators.
One can view this ofﬂine evaluation as a form of rudimentary cross-validation, similar to existing techniques in ofﬂine policy selection which evaluate a learned policy according to Q -values learned separate from the Q -values used to train the policy (Paine et al., 2020).
Although the separatelytrained Q -values (like q ( z ) ) are trained on the same ofﬂine dataset, this simple approach has been demonstrated to achieve strong performance, even compared to more sophisticated methods (Fu et al., 2021). [Figure]
We see that OTED is able to accurately learn the true parameters
E XPERIMENTS 5. E XPERIMENT S ETUP
We conduct several key experiments to understand (i) if OTED approximates target design distributions well; if not, what factors contribute to its failure, (ii) if we can select a good simulator model from a large pool of candidates without knowing target design distribution, and (iii) if an online policy trained in a simulator designed by OTED performs well across different environments.
We evaluate OTED, in a variety of settings; different tasks, data sources and simulator parameterizations.
We ﬁrst evaluate our method on 3 continuous control tasks from the Mujoco (Todorov et al., 2012) simulator, HalfCheetah, Hopper, and Walker2d , against state-of-the-art ofﬂine RL methods on the previous D4RL benchmark (Fu et al., 2020).
D4RL datasets are divided into 3 levels, medium , medium-expert , and expert , reﬂecting the competency of the behavior policy used to collect the data.
We modify the Mujoco simulator so that three 1-dimensional physics parameters – geom-friction , actuator-gainprm , and gravity – that cause different effects on the underlying simulator physics are unknown (See Appendix B for their ground truth values).
We learn them both independently and jointly.
Next, we experiment with a discrete MiniGrid (Chevalier-Boisvert et al., 2018) environment to study if OTED can also learn from high-dimensional image observations.
We parameterize a
MiniGrid environment by the density of obstacles in a 10-by-10 grid.
We use SAC as the online agent speciﬁcally as it is not designed to work in the ofﬂine RL setting and call an online SAC agent trained in designed environments, OTED-SAC.
We compare its performance to (i) previous state-of-the-art ofﬂine RL methods, behavioral cloning (BC), F-BRC (Kostrikov et al., 2021), MBOP (Argenson & Dulac-Arnold, 2020), (ii) online SAC agents trained with domain randomization (DR)
(Tobin et al., 2017) and a baseline called interval search (IS) that is inspired by ADR ? where parameters are sampled from a search interval that is gradually expanded over time based on the performance of an agent in the simulator, and (iii) a hybrid BC algorithm trained with both provided ofﬂine data and simulated data where the latter is collected using DR (BC+DR).
We report (mean) absolute error to measure the error of ﬁnal design predictions and ground truth designs.
We also report online reward in ground truth target environments for OTED-SAC and above baselines.
We compare our model selection method to 2 baseline scoring metrics: (i) design deviation , standard deviation of ﬁnal design predictions across different seeds and (ii) learning progress , average log ratios collected during training.
We pick simulator models with the lowest score for the former and highest for the latter.
We report two metrics to evaluate model selection: (i) design error , difference between a design prediction and the ground truth design, and (ii) design regret , difference between the design errors of a design prediction and the best prediction possible from the pool of available candidates.
We give more details on our model implementation in the Appendix A and will open source our code to public upon publication.
We also present experiments on alternative datasets taken from Kostrikov et al.
(2020) in the Appendix C.
5. R ESULTS ON D4RL B ENCHMARK
We present our results in three stages, ﬁrst focusing on the ability of our distribution matching objective to recover the true simulator parameters (Section 5.2.1), then presenting the results of the ﬁnal learned policy according to OTED-SAC (Section 5.2.2), and ﬁnally performing an ablation highlighting the importance of our model selection procedure (Section 5.2.3).
5.2. A CCURACY OF S IMULATOR P ARAMETER P REDICTIONS
In Figure 2, we evaluate OTED on D4RL expert dataset using absolute error metric.
We observe that
OTED exhibits low error in learning every parameter both independently or jointly.
This shows that
OTED is capable of learning when parameters can interact in unknown and complicated ways.
As an interesting use case, HalfCheetah emits a very high variance on the geom-friction parameter early on but converges to a smaller error.
This is due to one experiment giving a much higher error and taking much longer to ﬁnd the correct parameter rather than sensitivity of the HalfCheetah environment for this parameter.
Upon inspection all parameter learning experiments, we observe that geom-friction is consistently the hardest parameter to learn and is the most prone to trapping a simulator model in a local optima.
One particular reason is that, usually, the other two parameters are learned very quickly, allowing the discriminator to return high log ratios even before geom-friction is learned.
In some experiments, this leads to a simulator model following the inaccurate discriminator and converging into sub-optimal solutions.
But, thanks to our simulator selection, we mitigate this issue and train online agents with accurate simulators.
We present more results with all expert levels in Appendix D.
5.2. P ERFORMANCE OF THE O NLINE A GENT WITH D ESIGNED S IMULATORS
As OTED exhibits low absolute error on average, the natural question to ask is – does this correlate well with the performance of an online agent trained on designed simulators?
In Table 1, we compare OTED-SAC to baseline RL algorithms using different expert levels.
100% performance in D4RL benchmark corresponds to a SAC agent trained in the target environment.
For mediumlevel expert datasets, OTED-SAC clearly outperforms previous RL algorithms.
This is particularly important as collecting medium-level demonstrations can be done effortlessly without an optimal policy.
While previous algorithms fail when the ofﬂine dataset is collected via a sub-optimal policy, OTED is still able to recover the unknown design distribution.
Note that we also use BC to induce a behavior policy which is very sub-optimal across all medium-level datasets.
Even though Walker2D environments exhibit higher variance on geom-friction parameter for medium and medium-expert level experts, OTED-SAC outperforms previous models and also performs better compared to other parameterizations.
In majority of the cases, OTED-SAC achieves more than 100% performance, outperforming a baseline SAC agent trained on the ground-truth target environment.
Using ofﬂine and simulated data in BC algorithm (BC+DR), we get better performance than BC in medium and medium-expert datasets, particularly for Walker2D environments.
Our method still outperforms this strong baseline, showing improvements beyond combining ofﬂine and simulated datasets.
5.2. S IMULATOR S ELECTION
As presented above, success of an online agent is closely related to a simulator being an accurate reﬂection of the target environment.
As a result, it is very critical to pick a simulator model from a large pool of candidates.
In Table 2, we compare our simulator selection method to other baseline methods.
Our proposed method outperforms baselines and picks the best model available in almost of cases (zero regret).
Interestingly, design deviation is the second best metric and usually has low regret.
We hypothesize the success of this metric to its ability to distinguish models with a number of different sub-optimal predictions; hence high deviation from the average prediction.
5. M INI G RID E NVIRONMENT R ESULTS
We now move on MiniGrid, which presents us with image-based tasks.
We focus our evaluation on the simulator learned by OTED, showing that OTED is able to recover the true simulator parameters, even in image-based environments. [Table]
[Table]
We present the design errors. We ﬁnd that OTED’s use of expected log ratios for hyperparameter selection performs best.
We show our MiniGrid results in Figure 3 where we present absolute error between learned parameter and ground truth parameter.
After being trained for only several hundred steps, the discriminator learns to distinguish between generated and expert data, emitting a very strong signal for the designer to learn the simulator parameter.
Note that 0 .
26 is not in the support set of the parameter distribution as we quantize by increments of 0 .
but OTED is still able to ﬁnd the optimal prediction.
This shows (i) the capability of OTED to generalize to observations that are not seen during training and (ii) the robustness to quantization errors.
This also explains why it takes slightly longer for the discriminator to learn.
C ONCLUSION
We introduced a principled method, OTED, to learn simulator parameters of a target environment from ofﬂine datasets, combining two different paradigms, namely ofﬂine and online RL.
We proposed a KLdivergence regularized distribution matching objective where a distribution over simulator parameters is learned to minimize the discrepancy between the ofﬂine dataset and state-action distribution of an online behavior policy in the induced simulator.
We also introduced unsupervised simulator and policy selection methods.
By jointly learning the behavior policy and design distribution, we showed that OTED is able to recover the unknown parameter with high accuracy.
When evaluated on previous benchmarks, including D4RL, OTED exhibited low absolute error on average.
The proposed model selection method consistently outperforms baselines and leads to successful online agent training.
An online SAC agent (OTED-SAC) trained with designed simulators performed on par or better than previous state-of-the-art ofﬂine and online RL methods.
While it is still an open question what is an ideal simulator interface, we believe this depends on the [-designer actor and ν 
 and ν 2 are two critics that are initialized independently.-]problem domain.
For example, friction is an important parameter in robotics which depends on the type of environment such as water or air.
We also would like to emphasize that any available parameter can be made explicit and learned using ofﬂine data.
We showed that using some domain knowledge to set some of the parameters and reduce parameter space is plausible and can give competitive results while reducing complexity.
As future directions, we are looking forward to testing OTED in more complex environments such as UI navigation with real human traces