Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classiﬁers
Abstract
We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning.
Our approach stems from the idea that the agent’s experience in the source domain should look similar to its experience in the target domain.
Building off of a probabilistic view of
RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function .
This modiﬁed reward function is simple to estimate by learning auxiliary classiﬁers that distinguish sourcedomain transitions from target-domain transitions.
Intuitively, the modiﬁed reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain.
Our approach is applicable to domains with continuous states and actions and does not require learning a model of the dynamics.
1. Introduction
Reinforcement learning (RL) is often touted as a promising approach for costly and risk-sensitive applications, yet learning to act in those domains directly is costly and risky.
How can an intelligent agent learn to solve tasks in environments in which it cannot practice?
In this paper we study the problem of domain adaptation in reinforcement learning (RL).
In the context of RL, domains refer to different environments (MDPs) that have different dynamics (transition functions).
Our aim is to learn a policy in the source domain that will achieve high reward in a different target domain, using a limited amount of experience from the target domain.
RL algorithms today require a large amount of experience in the target domain .
Experience in the target domain is expensive to collect: it costs time (e.g., when the target domain is the real world, we cannot progress faster than real-time); it costs money (e.g., a robot might break itself); it could even be dangerous to humans (Matsakis, 2018).
For many tasks, such as assistive robotics and self-driving cars, we may have access to a different but structurally similar source domain .
While the source domain has different dynamics than the target domain, experience in the source domain is much cheaper to collect.
For example, a computer simulation of the real world can run much faster than real time, collecting (say) a year of experience in an hour; it is much cheaper to simulate 1000 robot manipulators in parallel than to maintain 1000 robot manipulators.
The source domain need not be a simulator, but rather could be any “practice” facility, such as a “farm” of robot arms (Levine et al., 2018), a “playpen” for learning to walk (Raibert, 2019), or a testing facility for self-driving vehicles (Madrigal, 2018).
Domain adaptation in RL is challenging because strategies which are effective in the source domain may not be effective in the target domain.
For example, a good approach to driving a car around a dry racetrack (the source domain) may entail aggressive acceleration and cutting corners.
If the target domain is an icy, public road, this approach may cause the car to skid off the road or hit oncoming trafﬁc.
While prior work has thoroughly studied the domain adaptation of observations in RL (Bousmalis et al., 2018; Ganin et al., 2016; Higgins et al., 2017), it ignores the domain adaptation of the dynamics .
For space constraints, we defer the discussion of related work to Appendix A.
This paper presents a simple and practical approach for domain adaptation in RL, illustrated in Fig proach stems from the idea that the agent’s experience in the source domain should look similar to its experience in the target domain.
Building off of a probabilistic view of
RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function .
This modiﬁed reward function is simple to estimate by learning auxiliary classiﬁers that distinguish source-domain transitions from target-domain transitions.
Because our method learns a classiﬁer, rather than a dynamics model, we expect it to handle high-dimensional tasks better than model-based methods, a conjecture supported by experiments on the 111-dimensional Ant task.
Intuitively, the modiﬁed reward function penalizes the agent for visiting states and taking actions where the source domain and target domain differ.
The agent is penalized for taking transitions which would indicate whether the agent is interacting with the source or target domain.
On a range of discrete and continuous control tasks, we both illustrate the mechanics of our approach and demonstrate its scalability to higherdimensional tasks.
broadly-applicable approach for learning from inaccurate models.
2. Preliminaries
Our problem setting will consider two MDPs: M source represents the source domain (e.g., a practice facility, simulator, or learned approximate model of the target domain) while
M target represents a the target domain.
We assume that the two domains have the same state space S , action space A , reward function r , and initially state distribution p 1 ( s 1 ) ; the only difference between the domains is the dynamics, p source ( s t +1 | s t , a t ) and p target ( s t +1 | s t , a t ) .
We will learn a Markovian policy π θ ( a | s ) , parametrized by θ .
Our objective is to learn a policy π that maximizes rewards on M target , E π, M target [ (cid:80) t γ t r ( s t , a t )] .
We now deﬁne our problem setting:
Deﬁnition 1.
Domain Adaptation for RL is the problem of using interactions in the source MDP M source together with a small number of interactions in the target MDP M target to acquire a policy that achieves high reward in the target
MDP, M target .
We will assume every transition with non-zero probability in the target domain will have non-zero probability in the source domain: p target ( s t +1 | s t , a t ) > 0 = ⇒ p source ( s t +1 | s t , a t ) >for all states s t , s t +1 ∈ S and actions a t ∈ A .
This assumption is very weak, and common in work on importance sampling (Koller & Friedman, 2009, §12.2.2).
3. A Variational Perspective on Domain Adaptation in RL
The probabilistic inference interpretation of RL (Kappen, 2005; Todorov, 2007; Toussaint, 2009; Ziebart, 2010; Rawlik et al., 2013; Levine, 2018) treats the reward function as deﬁning a desired distribution over trajectories.
The agent’s task is to sample from this distribution by picking trajectories with probability proportional to their exponentiated reward.
This section will reinterpret this model in the context of domain transfer, showing that domain adaptation of dynamics can be done by modifying the rewards .
To apply this model to domain adaptation, deﬁne p ( τ ) as the desired distribution over trajectories in the target domain,
[Equation]
, and q ( τ ) as our agent’s distribution over trajectories in the source domain,
[Equation]
As noted in Section 2, we assume both trajectory distributions have the same initial state distribution.
Our aim is to learn a policy whose behavior in the source domain both receives high reward and has high likelihood under the target domain dynamics.
We codify this objective by minimizing the reverse KL divergence between these two distributions:
[Equation]
where
[Equation]
The constant c is the partition function of p ( τ ) , which is independent of the policy and source dynamics.
While ∆ r is deﬁned as the difference of transition probabilities, in
Sec.
4.1 we show how to estimate ∆ r without learning transition probabilities directly.
In the special case where the source and target dynamics are equal, the correction term ∆ r is zero and we recover maximum entropy RL (Ziebart, 2010; Todorov, 2007).
We emphasize that our reward correction is different from prior work that adds log β ( a | s ) to the reward to regularize the policy to be close to the behavior policy β (Schulman et al., 2015; Jaques et al., 2017;
Schroecker & Isbell, 2020; Abdolmaleki et al., 2018; Jaques et al., 2019; Thananjeyan et al., 2020).
In the case where the source dynamics are not equal to the true dynamics, this objective is not the same as maximum entropy RL on trajectories sampled from the source domain.
Instead, this objective suggests a corrective term ∆ r that should be added to the reward function to account for the discrepancy between the source and target dynamics.
The correction term, ∆ r , is quite intuitive.
If a transition ( s t , a t , s t +1 ) has equal probability in the source and target domains, then ∆ r ( s t , a t ) =
so no correction is applied.
For transitions that are likely in the source but are unlikely 1: for num do 2:
D 3: if then 4: D 5: θ ←
− θ ) 6: ˜ ( s , a +
, a ) 7: π M 8: return π in the target domain, ∆ r < 0 , so the agent is penalized for “exploiting” inaccuracies or discrepancies in the source domain by taking these transitions.
For the example environment in Figure 1, transitions through the center of the environment are blocked in the target domain but not in the source domain.
For these transitions, ∆ r would serve as a large penalty, discouraging the agent from taking these transitions and instead learning to navigate around the wall.
Appendix C presents additional interpretations of ∆ r in terms of coding theory, mutual information, and a constraint on the discrepancy between the source and target dynamics.
Appendix B shows how prior work that adapts observation is a special case of our approach.
4. Domain Adaptation with Learned Rewards
The variational perspective on model-based RL in the previous section suggests that we should modify the reward in the source domain by adding ∆ r .
While ∆ r is deﬁned above in terms of transition probabilities, we will show below how it can be estimated via binary classiﬁcation, without learning an explicit dynamics model.
We then use this observation to develop a practical algorithm for off-dynamics RL.
4.1. Estimating the Reward Correction with Classiﬁers
The transition probabilities in the modiﬁed reward function are rarely known and are hard to estimate.
Instead, we show that we can estimate this log ratio using a pair of (learned) binary classiﬁers, which will infer whether transitions came from the source or target domain by rewriting ∆ r ( s t , a t , s t +1 ) as log p ( target | s t , a t , s t +1 ) .............................
− log p ( target | s t , a t ) − log p ( source | s t , a t , s t +1 ) ..............................
+ log p ( source | s t , a t )
The ........orange terms are the difference in logits from the classiﬁer conditioned on s t , a t , s t +1 , while the blue terms are the difference in logits from the classiﬁer conditioned on just s t , a t .
Intuitively, ∆ r answers the following question: for the task of predicting whether a transition came from the source or target domain, how much better can you perform after observing s t +1 ?
4.2. Algorithm Summary
Our algorithm, Domain Adaptation with Rewards from Classiﬁers (DARC), is presented in Alg.
The algorithm [Figure] modiﬁes an existing MaxEnt RL algorithm to additionally learn two classiﬁers, q θ SAS ( target | s t , a t , s t +1 ) and q θ SAS ( target | s t , a t ) .
We use the classiﬁers to modify the rewards from the source domain, and apply MaxEnt RL to this experience.
We use SAC (Haarnoja et al., 2018) as our
MaxEnt RL algorithm, but emphasize that DARC is applicable to any MaxEnt RL
algorithm (e.g., on-policy, off-policy, and model-based).
Code has been released.
5. Experiments
We start with a didactic experiment to build intuition for the mechanics of our method, and then evaluate on more complex tasks.
Appendix D.3 includes additional experiments.
Illustrative example.
We start with a simple gridworld example, shown on the right, where we can apply our method without function approximation.
The goal is to navigate from the top left to the bottom left.
The real environment contains an obstacle (shown in red), which is not present in the source domain.
If we simply apply RL on the source domain, we obtain a policy that navigates directly to the goal (blue arrows), and will fail when used in the target domain.
We then apply our method: we collect trajectories from the source domain and real world to ﬁt the two tabular classiﬁers.
These classiﬁers give us a modiﬁed reward, which we use to learn a policy in the source domain.
The modiﬁed reward causes our learned policy to navigate around the obstacle.
Scaling to more complex tasks.
We now apply DARC to the more complex tasks shown in Fig.
We deﬁne three tasks by crippling one of the joints of each robot in the target domain, but using the fully-functional robot in the source domain.
We use three simulated robots taken from OpenAI
Gym (Brockman et al., 2016): 7 DOF reacher, half cheetah, and ant.
The broken reacher is based on the task described by Vemula et al.
We also include a task where the shift in dynamics is external to the robot, by modifying the cheetah task to reward the agent for running both forward and backwards.
It is easier to learn to run backwards, but the target domain contains an obstacle that prevents the agent from running backwards.
We compare our method to seven baselines.
RL on Source and RL on Target directly perform RL on the source and target domains, respectively.
The Finetuning baseline takes the result of running RL on the source domain, and further ﬁnetunes the agent on the target domain.
The Importance
Weighting baseline performs RL on importance-weighted samples from the source domain; the importance weights are exp(∆ r ) .
To account for the fact that our method performs more gradient updates per environment step in the source domain, we trained a version of the RL on source baseline likewise does 10 gradient updates per source domain step.
Finally, we compared against two model-based RL methods:
MBPO (Janner et al., 2019) and PETS (Chua et al., 2018).
We show the results of this experiment in Fig. 4, plotting the reward on the target domain as a function of the number of transitions in the target domain.
On all tasks, the RL on source baseline (shown as a dashed line because it observes no target transitions) performs considerably worse than the optimal policy from RL on the target domain, suggesting that good policies for the source domain are suboptimal for the target domain.
Nonetheless, on three of the four tasks our method matches (or even surpasses) the asymptotic performance of doing RL on the target domain, despite never doing RL on experience from the target domain, and despite observing 5 - 10x less experience from the target domain.
On the broken reacher and broken half cheetah tasks, we observe that ﬁnetuning on the target domain performs on par with our method.
On the simpler broken reacher task, just doing RL on the target domain with a large number of gradient steps works quite well (we did not tune this parameter for our method).
However, as we scale to the more complex broken ant and half cheetah obstacle tasks, we observe that all baselines perform poorly.
To gain more intuition for our method, we recorded the reward correction ∆ r throughout training on the broken reacher environment.
In this experiment, we ran RL on the source domain for 100k steps before switching to our method.
Said another way, we ignored ∆ r for the ﬁrst 100k steps of training.
As shown in Fig.
5, ∆ r steadily decreases during these ﬁrst 100k steps, suggesting that the agent is learning a strategy that takes transitions where the source domain and target domain have different dynamics: the agent is making use of its broken joint.
After 100k steps, when we maximize the combination of task reward and reward correction ∆ r , we observe that ∆ r increases, so the agent’s transitions in the source domain are increasingly consistent with target domain dynamics.
After around 1e training steps ∆ r is zero: the agent has learned a strategy that uses transitions that are indistinguishable between the source and target domains.
6. Discussion
In this paper, we proposed a simple, practical, and intuitive approach for domain adaptation to changing dynamics in RL.
We formally motivate this method from a novel variational perspective on domain adaptation in RL, which suggests that we can compensate for differences in dynamics via the reward function.
Experiments on a range of tasks show that our method can leverage the source domain to learn policies that work well in the target domain, despite observing only a handful of transitions from the target domain.
A. Related Work
While our work will focus on domain adaptation applied to RL, we start by reviewing more general ideas in domain adaptation, and defer to Kouw & Loog (2019) for a recent review of the ﬁeld.
Two common approaches to domain adaptation are importance weighting and domain-agnostic features.
Importance-weighting methods (e.g., (Zadrozny, 2004; Cortes &
Mohri, 2014; Lipton et al., 2018)) estimate the likelihood ratio of examples under the target domain versus the source domain, and use this ratio to re-weight examples sampled from the source domain.
To estimate the likelihood ratio, some methods directly estimate two density models and then take the difference (e.g., (Sugiyama & Müller, 2005a; Yu & Szepesvári, 2012; Baktashmotlagh et al., 2014)), other methods directly estimate the ratio (Sugiyama & Müller, 2005b; Sugiyama et al., 2007; 2008; Mohamed & Lakshminarayanan, 2016; Kanamori et al., 2009; Uehara et al., 2016).
A number of these direct estimation methods operate by learning a classiﬁer to distinguish examples from the source domain versus examples from the target domain (Bickel et al., 2007;
Sønderby et al., 2016; Mohamed & Lakshminarayanan, 2016; Uehara et al., 2016).
We refer to Huszár (2017) for a recent review of direct estimation methods.
Methods based on domain-agnostic features aim to map examples from the source and target domain into a common feature space (Fernando et al., 2013; Hoffman et al., 2016; Wulfmeier et al., 2017).
Our method is similar to classiﬁer-based density-ratio estimation, with two important distinctions.
First, we will need to estimate the density ratio of conditional distributions (transition probabilities), which is different from modeling conditional distributions as a density ratio (Sugiyama et al., 2010).
To do this, we will learn not one but two classiﬁers.
Second, we will use the logarithm of the density ratio to modify the reward function instead of weighting samples by the density ratio, which is often numerically unstable (see, e.g., Schulman et al.
Prior methods for applying domain adaptation to RL include approaches based on system identiﬁcation, domain randomization, and observation adaptation.
Perhaps the most established approach, system identiﬁcation (Ljung, 1999), uses observed data to tune the parameters of a simulator (Feldbaum, 1960; Werbos, 1989; Wittenmark, 1995; Ross & Bagnell, 2012;
Tan et al., 2016; Zhu et al., 2017b;
Farchy et al., 2013)
More recent work has successfully used this strategy to bridge the sim2real gap (Chebotar et al., 2019; Rajeswaran et al., 2016).
Closely related is work on online system identiﬁcation and meta-learning, which directly uses the inferred system parameters to update the policy (Yu et al., 2017; Clavera et al., 2018;
Tanaskovic et al., 2013; Sastry & Isidori, 1989).
However, these approaches typically require either a model of the environment or a manually-speciﬁed distribution over potential test-time dynamics, requirements that our method will lift.
Another approach, domain randomization , randomly samples the parameters of the source domain and then ﬁnds the best policy for this randomized environment (Sadeghi & Levine, 2016;
Tobin et al., 2017; Peng et al., 2018; Cutler et al., 2014).
While often effective, this method is sensitive to the choice of which parameters are randomized, and the distributions from which these simulator parameters are sampled.
A third approach, observation adaptation , modiﬁes the observations of the source domain to appear similar to those in the target domain.
While this approach has been successfully applied to video games (Gamrian & Goldberg, 2018) and robot manipulation (Bousmalis et al., 2018), it ignores the fact that the source and target domains may have differing dynamics.
The theoretical derivation of our method is heavily inspired by prior work which formulates control as a problem of probabilistic inference (Dayan & Hinton, 1997; Attias, 2003; Kappen, 2005; Todorov, 2007; Toussaint, 2009; Ziebart, 2010;
Theodorou et al., 2010; Rawlik et al., 2013; Levine & Koltun, 2013; Levine, 2018; Abdolmaleki et al., 2018).
These methods aim to make an agent’s experience in the target domain look like the expert’s experience in the target domain, whereas our method aims to make an agent’s experience in the source domain look like the expert’s experience in the target domain.
We emphasize that the domain shift we consider is caused by domains having different dynamics, not by actions being sampled from different policies.
Algorithms for model-based RL (e.g., (Polydoros & Nalpantidis, 2017; Sutton, 1991; Janner et al., 2019; Deisenroth & Rasmussen, 2011; Wang et al., 2019; Williams et al., 2015; Hafner et al., 2018; Chua et al., 2018; Finn & Levine, 2017) and off-policy RL (e.g., (Munos et al., 2016; Fujimoto et al., 2018; Dann et al., 2014; Dudík et al., 2011) similarly aim to improve the sample efﬁciency of RL, but do use the source domain to accelerate learning.
Our method is applicable to any maximum entropy RL algorithm, including on-policy (Song et al., 2019), off-policy (Abdolmaleki et al., 2018; Haarnoja et al., 2018), and model-based (Janner et al., 2019; Williams et al., 2015) algorithms.
We will use the soft actor critic algorithm (Haarnoja et al., 2018) in our experiments.
Recently, Vemula et al.
(2020) proposed a method for planning with an inaccurate model by assigning a high, ﬁxed, cost to transitions where the model was inaccurate.
Our method similarly accounts for discrepancies in dynamics via rewards, but does so with a learned classiﬁer, allowing our method to be applied in stochastic environments with continuous states and actions.
B. The Special Case of an Observation Model
To highlight the relationship between domain adaptation of dynamics versus observations, we now consider a special case.
In this subsection, we will assume that the state s t (cid:44) ( z t , o t ) is a combination of the system latent state z t (e.g., the poses of all objects in a scene) and an observation o t (e.g., a camera observation).
We will deﬁne q ( o t | z t ) and p ( o t | z t ) as the observation models for the source and target domains.
In this special case, we can decompose the KL objective into three terms:
[Equation]
Prior methods that perform observation adaptation (Bousmalis et al., 2018; Gamrian & Goldberg, 2018) effectively minimize the observation adaptation term, 2 but ignore the effect of dynamics.
In contrast, the ∆ r reward correction in our method provides one method to address both dynamics and observations.
These approaches could be combined; we leave this as future work.
C. Additional Interpretations of the Reward Correction
This section presents four additional interpretations of the reward correction, ∆ r .
C.1. Coding Theory
The reward correction ∆ r can also be understood from the perspective of coding theory.
Suppose that we use a data-efﬁcient replay buffer that exploits that fact that the next state s t +1 is highly redundant with the current state and action, s t , a t .
If we assume that the replay buffer compression has been optimized to store transitions from the target environment, (negative)
∆ r is the number of additional bits (per transition) needed for our source replay buffer, as compared with our target replay buffer.
Thus, an agent which maximizes ∆ r will seek those transitions that can be encoded most efﬁciently, minimizing the size of the source replay buffer.
C.2. Mutual Information
We can gain more intuition in the modiﬁed reward by writing the expected value of ∆ r in terms of mutual information:
[Equation]
The mutual information I ( s t +1 ; target | s t , a t ) reﬂects how much better you can predict the next state if you know that you are interacting with the target domain, instead of the source domain.
Our approach does exactly this, rewarding the agent for taking transitions that provide information about the target domain while penalizing transitions that hint to the agent that it is interacting with a source domain rather than the target domain.
While we derived DARC by minimizing a reverse KL divergence
[Equation]
we can also show that DARC maximizes a lower bound on a risk-sensitive reward objective (Mihatsch & Neuneier, 2002):
[Equation]
The inequality on the last line is an application of Jensen’s inequality.
One interesting question is when it would be preferable to maximize Eq.
1 rather than Eq.
While Eq. 2 provides a loser bound on the risk sensitive objective, empirically it may avoid the risk-seeking behavior that can be induced by risk-sensitive objectives.
We leave the investigation of this trade-off as future work.
C.4. A Constraint on Dynamics Discrepancy
Our method regularizes the policy to visit states where the transition dynamics are similar between the source domain and target domain:
[Equation]
This objective can equivalently be expressed as applying MaxEnt RL to only those policies which avoid exploiting the dynamics discrepancy.
More precisely, the KKT conditions guarantee that there exists a positive constant (cid:15) > 0 such that our objective is equivalent to the following constrained objective:
[Equation]
where Π DARC denotes the set of policies that do not exploit the dynamics discrepancy:
[Equation]
p target ( s t +1 | s t , a t ))
(cid:35) ≤ (cid:15) (cid:41) .
One potential beneﬁt of considering our method as the unconstrained objective is that it provides a principled method for increasing or decreasing the weight on the ∆ r term, depending on how much the policy is currently exploiting the dynamics discrepancy.
We leave this investigation as future work.
D. Additional Experiments D.1. Visualizing the reward modiﬁcation in stochastic domains.
In this experiment, we use an “archery” task to visualize how the modiﬁed reward accounts for differences in dynamics.
The task, shown in Fig. 6, requires choosing an angle at which to shoot an arrow.
The practice range (i.e., the source domain) is outdoors, with wind that usually blows from left to right.
The competition range (i.e., the target domain) is indoors with no wind.
The reward is the negative distance to the target.
We plot the reward as a function of the angle in both domains in
Fig.
The optimal strategy for the outdoor range is to compensate for the wind by shooting slightly to the left ( θ = − 0 . 8 ), while the optimal strategy for the indoor range is to shoot straight ahead ( θ = 0 ).
We estimate the modiﬁed reward function with DARC, and plot the modiﬁed reward in the windy outdoor range and indoor range.
We aggregate across episodes using
J ( θ ) = log E p ( s (cid:48) | θ ) [exp( r ( s (cid:48) ))] ; see Appendix E.4 for details.
We observe that maximizing the modiﬁed reward in the windy range does not yield high reward in the windy range, but does yield a policy that performs well in the indoor range.
D.2.
Safety emerges from domain adaptation to the termination condition.
The termination condition is part of the dynamics (White, 2017), and our next experiment studies how our method copes with domain shift in the termination condition.
We use the humanoid shown in Fig.
7 for this experiment and set the task reward to 0.
In the source domain episodes have a ﬁxed length of 300 steps; in the target domain the episode terminates when the robot falls.
The scenario mimics the real-world setting where robots have freedom to practice in a safe, cushioned, practice facility, but are preemptively stopped when they try to take unsafe actions in the real world.
Our aim is for the agent to learn to avoid unsafe transitions in the source domain that would result in episode termination in the target domain.
As shown in Fig.
7, our method learns to remain standing for nearly the entire episode.
As expected, baselines that maximize the zero reward on the source and target domains fall immediately.
While DARC was not designed as a method for safe RL (Tamar et al., 2013; Achiam et al., 2017; Eysenbach et al., 2017; Berkenkamp et al., 2017), this experiment suggests that safety may emerge automatically from DARC, without any manual reward function design.
D.3. Ablation Experiments
Our ﬁnal experiment examines the importance of using two classiﬁers to estimate ∆ r .
We compared our method to an ablation that does not learn the SA classiﬁer, effectively ignoring the blue terms in ∆ r .
As shown in Fig.
8, this ablation performs considerably worse than our method.
Intuitively, this makes sense: we might predict that a transition came from the source domain not because the next state had higher likelihood under the source dynamics, but rather because the state or action was visited more frequently in the source domain.
The second classiﬁer used in our method corrects for this distribution shift.
Finally, we examine the importance of input noise regularization in classiﬁers.
As we observe only a handful of transitions from the target domain, we hypothesized that regularization would be important to prevent overﬁtting.
We test this hypothesis in Fig.
9 by training our method on the broken reacher environment with varying amounts of input noise.
With no noise or little noise our method performs poorly (likely due to overﬁtting); too much noise also performs poorly (likely due to underﬁtting).
We used a value of 1 in all our experiments, and did not tune this value.
See Appendix D.3 for more plots of both ablation experiments.
E. Experiment Details and Hyperparameters
Our implementation of DARC is built on top of the implementation of SAC from Guadarrama et al.
Unless otherwise speciﬁed, all hyperparameters are taken from Guadarrama et al.
All neural networks (actor, critics, and classiﬁers) have two hidden layers with 256-units each and ReLU activations.
Since we ultimately will use the difference in the predictions of the two classiﬁers, we use a residual parametrization for the SAS classiﬁer q ( target | s t , a t , s t +1 ) .
Using f SAS ( s t , a t , s t +1 ) , f SA ( s t , a t ) ∈ R 2 to denote the outputs of the two classiﬁer networks, we compute the classiﬁer predictions as follows:
[Equation]
For the SAS classiﬁer we propagate gradients back through both networks parameters, θ SAS and θ SA .
Both classiﬁers use
Gaussian input noise with σ = 1 .
Optimization of all networks is done with Adam (Kingma & Ba, 2014) with a learning rate of 3e-4 and batch size of 128.
Most experiments with DARC collected 1 step in the target domain every 10 steps in the source domain (i.e., r = 10 ).
The one exception is the half cheetah obstacle domain, where we tried increasing r beyondto 30, 100, 300, and 1000.
We found a large beneﬁt from increasing r to 30 and 100, but did not run the other experiments long enough to draw any conclusions.
Fig.
4 uses r = 30 for half cheetah obstacle.
We did not tune this parameter, and expect that tuning it would result in signiﬁcant improvements in sample efﬁciency.
We found that DARC was slightly more stable if we warm-started the method by applying RL on the source task without ∆ r for the ﬁrst t warmup iterations.
We used t warmup = 1 e 5 for all tasks except the broken reacher, where we used t warmup = 2 e 5 .
This discrepancy was caused by a typo in an experiment, and subsequent experiments found that DARC is relatively robust to different values of t warmup ; we did not tune this parameter.
E.1. Baselines
The RL on Source and RL on Target baselines are implemented identically to our method, with the exception that ∆ r is not added to the reward function.
The RL on Target (10x) is identical to RL on Target, with the exception that we takegradient steps per environment interaction (instead of 1).
The Importance Weighting baseline estimates the importance weights as p target ( s t +1 | s t , a t )
/p source ( s t +1 | s t , a t ) ≈ exp(∆ r ) .
The importance weight is used to weight transitions in the
SAC actor and critic losses.
PETS (Chua et al., 2018) :
The PETS baseline is implemented using the default conﬁgurations used by (Chua et al., 2018) for the environments evaluated.
That is, the broken-half-cheetah environment uses the hyperparameters as used by the half-cheetah environment in (Chua et al., 2018).
(Janner et al., 2019)
E.2. Environments
Broken Reacher
This environment uses the 7DOF robot arm from the Pusher environment in OpenAI Gym.
The observation space is the position and velocities of all joints and the goal.
The reward function is
[Equation]
, and episodes are 100 steps long.
In the target domain the 2nd joint (0-indexed) is broken: zero torque is applied to this joint, regardless of the commanded torque.
Broken Half Cheetah This environment is based on the HalfCheetah environment in OpenAI Gym.
Episodes aresteps long.
In the target domain the 0th joint (0-indexed) is broken: zero torque is applied to this joint, regardless of the commanded torque.
Broken Ant
This environment is based on the Ant environment in OpenAI Gym.
We use the standard termination condition and cap the maximum episode length at 1000 steps.
In the target domain the 3rd joint (0-indexed) is broken: zero torque is applied to this joint, regardless of the commanded torque.
In all the broken joint environments, we choose which joint to break to computing which joint caused the “RL on Source” baseline to perform worst on the target domain, as compared with the “RL on Target” baseline.
Half Cheetah Obstacle
This environment is based on the HalfCheetah environment in OpenAI Gym.
Episodes are 1000 steps long.
We modiﬁed the standard reward function to use the absolute value in place of the velocity, resulting the following reward function:
[Equation]
where s x vel is the velocity of the agent along the forward-aft axis and ∆ t = 0 .
01 is the time step of the simulator.
In the target domain, we added a wall at x = − 3 m , roughly 3 meters behind the agent.
Humanoid Used for the experiment in Fig.
7, we used a modiﬁed version of Humanoid from OpenAI Gym.
The source domain modiﬁed this environment to ignore the default termination condition and instead terminate after exactly 300 time steps.
The target domain uses the unmodiﬁed environment, which terminates when the agent falls.
E.3. Figures
Unless otherwise noted, all experiments were run with three random seeds.
Figures showing learning curves (Figures 4, 5, 8, and 9) plot the mean over the three random seeds, and also plot the results for each individual random seed with semitransparent lines.
E.4. Archery Experiment
We used a simple physics model for the archery experiment.
The target was located 70m North of the agent, and wind was applied along the East-West axis.
The system dynamics:
[Equation]
We trained the classiﬁer by sampling θ ∼ U [ − 2 , 2] (measured in degrees) for 10k episodes in the source domain and 10k episodes in the target domain.
The classiﬁer was a neural network with 1 hidden layer with 32 hidden units and ReLU activation.
We optimized the classiﬁer using the Adam optimizer with a learning rate of 3e-3 and a batch size of 1024.
We trained until the validation loss increased for 3 consecutive epochs, which took 16 epochs in our experiment.
We generated
Fig.
6 by sampling 10k episodes for each value of θ and aggregating the rewards using J ( θ ) =
log E p ( s (cid:48) | θ ) [exp( r ( s (cid:48) ))] .
We found that aggregating rewards by taking the mean did not yield meaningful results, perhaps because the mean corresponds to a (possibly loose) lower bound on J (see Appendix C.3).