R OBUST V ISUAL D OMAIN R ANDOMIZATION FOR R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Producing agents that can generalize to a wide range of visually different environments is a signiﬁcant challenge in reinforcement learning.
One method for overcoming this issue is visual domain randomization, whereby at the start of each training episode some visual aspects of the environment are randomized so that the agent is exposed to many possible variations.
However, domain randomization is highly inefﬁcient and may lead to policies with high variance across domains.
Instead, we formalize the visual domain randomization problem, and show that minimizing the policy’s Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies.
We propose a regularization method where the agent is only trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant.
We conduct experiments that demonstrate that our technique leads to more efﬁcient and robust learning than standard domain randomization, while achieving equal generalization scores.
I NTRODUCTION
Deep Reinforcement Learning (RL) has proven very successful on complex high-dimensional problems ranging from games like Go (Silver et al., 2017) and Atari games (Mnih et al., 2015) to robot control tasks (Levine et al., 2016).
However, one prominent issue is that of overﬁtting, illustrated in ﬁgure 1: agents trained on one domain fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2018; Zhang et al., 2018b; Packer et al., 2018;
Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018).
Good generalization is essential for problems such as robotics and autonomous vehicles, where the agent is often trained in a simulator and is then deployed in the real world where novel conditions will certainly be encountered.
Transfer from such simulated training environments to the real world is known as crossing the reality gap in robotics, and is well known to be difﬁcult, thus providing an important motivation for studying generalization.
We focus on the problem of generalizing between environments that visually differ from each other, for example in color or texture, but where the underlying dynamics are the same.
In reinforcement learning, prior work to address this topic has studied both domain adaptation and domain randomization .
Domain adaptation techniques aim to update the data distribution in simulation to match the real distribution through some form of canonical mapping or using regularization methods (James et al., 2018; Bousmalis et al., 2017; Gamrian & Goldberg, 2018).
Alternatively, domain randomization, in which the visual and physical properties of the training domains are randomized at the start of each episode during training, has also been shown to lead to improved generalization and transfer to the real world with little or no real world data (Tobin et al., 2017; Sadeghi & Levine, 2016;
Antonova et al., 2017; Peng et al., 2017; Mordatch et al., 2015; Rajeswaran et al., 2016; OpenAI, 2018).
However, domain randomization has been empirically shown to often lead to suboptimal policies with high variance in performance over different randomizations (Mehta et al., 2019).
This issue can cause the learned policy to underperform in any given target domain.
We propose a regularization method for learning policies that are robust to irrelevant visual changes in the environment.
Our work combines aspects from both domain adaptation and domain randomization, in that we maintain the notion of randomized environments but use a regularization method to achieve good generalization over the randomization space.
Our contributions are the following:
• We formalize the visual domain randomization problem, and show that the Lipschitz constant of the agent’s policy over visual variations provides an upper bound on the agent’s robustness to these variations.
• We propose an algorithm whereby the agent is only trained on one variation of the environment but its learned representations are regularized so that the Lipschitz constant is minimized.
• We experimentally show that our method is more efﬁcient and leads to lower-variance policies than standard domain randomization, while achieving equal or better returns and generalization ability.
This paper is structured as follows.
We ﬁrst review related work, formalize the visual generalization problem, and present our theory contributions.
We then describe our regularization method, and illustrate its application to a toy gridworld problem.
Finally, we compare our method with standard domain randomization and other regularization techniques in complex visual environments.
R ELATED W ORK 2. G ENERALIZATION IN D EEP R EINFORCEMENT L EARNING
Generalization to novel samples is well studied in supervised learning, where evaluating generalization through train/test splits is ubiquitous.
However, evaluating for generalization to novel conditions through such train/test splits is not common practice in Deep RL.
Zhang et al.
(2018b) show that
Deep RL algorithms are shown to suffer from overﬁtting to training conﬁgurations and to memorize training scenarios in discrete maze tasks.
Packer et al.
(2018) study performance under train-test domain shift by modifying environmental parameters such as robot mass and length to generate new domains.
Farebrother et al.
(2018) propose using different game modes of Atari games to measure generalization.
They turn to supervised learning for inspiration, ﬁnding that both L2 regularization and dropout can help agents learn more generalizable features.
These works all show that standard Deep RL algorithms tend to overﬁt to the environment used during training, hence the urgent need for designing agents that can generalize better.
2. D OMAIN R ANDOMIZATION
We distinguish between two types of domain randomization: visual randomization, in which the variability between domains should not affect the agent’s policy, and dynamics randomization, in which the agent should learn to adjust its behavior to achieve its goal.
Visual domain randomization, which we focus on in this work, has been successfully used to directly transfer RL agents from simulation to the real world without requiring any real images (Tobin et al., 2017; Sadeghi & Levine, 2016; Kang et al., 2019).
These approaches used low ﬁdelity rendering and randomized scene properties such as lighting, textures, camera position, and colors, which led to improved generalization.
Other work has also combined domain randomization and domain adaptation techniques (James et al., 2018; Chebotar et al., 2018; Gamrian & Goldberg, 2018).
These approaches both randomize the simulated environment and penalize the gap between the trajectories in the simulations and the real world, either by adding a term to the loss, or learning a mapping between the states of the simulation and the real world.
However, these methods require a large number of samples of real world trajectories, which can be expensive to collect.
Prior work has, however, noted the inefﬁciency of domain randomization.
Mehta et al.
(2019) show that domain randomization may lead to suboptimal policies that vary a lot between domains, and propose to train on the most informative environment variations within the given randomization ranges.
Zakharov et al.
(2019) also guide the domain randomization procedure by training a DeceptionNet , that learns which randomizations are actually useful to bridge the domain gap for image classiﬁcation tasks.
2. L EARNING D OMAIN -I NVARIANT F EATURES AND D OMAIN A DAPTATION
Learning domain-invariant features has emerged as a promising approach for taking advantage of the commonalities between domains.
For instance, in the semi-supervised context, Bachman et al.
; Sajjadi et al.
; Coors et al.
; Miyato et al.
; Xie et al.
(2019) enforce that predictions of their networks be similar for original and augmented data points, with the objective of reducing the required amount of labelled data for training.
Our work extends such methods to reinforcement learning.
In the reinforcement learning context, several other papers have also explored this topic.
Tzeng et al.
(2015) and Gupta et al.
(2017) add constraints to encourage networks to learn similar embeddings for samples from both a simulated and a target domain.
Daftry et al.
(2016) apply a similar approach to transfer policies for controlling aerial vehicles to different environments.
Bousmalis et al.
(2017) compare different domain adaptation methods in a robot grasping task, and show that they improve generalization.
Wulfmeier et al.
(2017) use an adversarial loss to train RL agents in such a way that similar policies are learned in both a simulated domain and the target domain.
While promising, these methods are designed for cases when simulated and target domains are both known, and cannot straightforwardly be applied when the target domain is only known to be within a distribution of domains.
Concurrently and independently of our work, Aractingi et al.
(2019) also propose a regularization scheme to learn policies that are invariant to randomized visual changes in the environment without any real world data. Our work differs from theirs in that we propose a theoretical justiﬁcation for this regularization and an analysis of the effects of this regularization on the learned representations. Crucially, whereas Aractingi et al. (2019) propose regularizing the network outputs, we regularize intermediate layers instead.
In the appendix, we experimentally compare their regularization to ours and show that regularizing the network outputs leads to an undesirable trade-off between agent performance and generalization.
P ROBLEM F ORMULATION
We consider Markov decision processes (MDP) deﬁned by ( S , A , R, T, γ ) , where S is the state space,
A the action space, R : S ×
A → R the reward function, T : S ×
A → Pr ( S ) the transition dynamics, and γ the discount factor.
In reinforcement learning, an agent’s objective is to ﬁnd a policy π that maps states to distributions over actions such that the cumulative discounted reward yielded by its interactions with the environment is maximized.
3. V ISUAL DOMAIN RANDOMIZATION
We consider a framework in which we are given a set of N parameters that can be changed to visually modify the environment, deﬁned within a randomization space Ξ ⊂ R N .
These parameters can for example control textures, colors, or lighting.
Denoting J ( π, ξ ) the cumulative returns of a policy π , the goal is to solve the optimization problem deﬁned by J ( π ∗ )
= max π E ξ [ J ( π, ξ )] .
Standard domain randomization, in which parameters ξ are randomly sampled at the start of each training episode, empirically produces policies with strongly varying performance over different regions of the randomization space, as demonstrated by Mehta et al.
This high variance can cause the learned policy to underperform in any given target domain.
To yield insight into the robustness of policies learned by domain randomization, we start by formalizing the notion of a visually randomized MDP.
Deﬁnition 1 Let M =
( S , A , R, T, γ ) be an MDP.
A randomizer function of M is a mapping φ : S → S (cid:48) where S (cid:48) is a new set of states.
The randomized MDP M φ =
( S φ , A φ , R φ , T φ , γ φ ) is deﬁned as, for s, s (cid:48) ∈ S , a ∈ A :
[Equation]
Given a policy π on MDP M and a randomization M φ , we also deﬁne the agent’s policy on M φ as π φ ( ·| s ) =
π ( ·| φ ( s )) .
Despite all randomized MDPs sharing the same underlying rewards and transitions, the agent’s policy can vary between domains.
For example, in policy-based algorithms (Williams, 1992), if there are several optimal policies then the agent may adopt different policies for different φ .
Furthermore, for value-based algorithms such as DQN (Mnih et al., 2015), two scenarios can lead to there being different policies for different φ .
First, the (unique) optimal Q-function may correspond to several possible policies.
Second, imperfect function approximation can lead to different value estimates for different randomizations and thus to different policies.
To compare the ways in which policies can differ between randomized domains, we introduce the notion of Lipschitz continuity of a policy over a set of randomizations.
Deﬁnition
We assume the state space is equipped with a distance metric.
A policy π is Lipschitz continuous over a set of randomizations { φ } if for all randomizations φ 1 and φ 2 in { φ } ,
[Equation]
is ﬁnite.
Here, D TV ( P (cid:107) Q ) is the total variation distance between distributions (given by 12 (cid:80) a ∈A | P ( a ) − Q ( a ) | when the action space is discrete).
The following inequality shows that this Lipschitz constant is crucial in quantifying the robustness of RL agents over a randomization space.
The smaller the Lipschitz constant, the less a policy is affected by different randomization parameters.
Informally, if a policy is Lipschitz continuous over randomized MDPs, then small changes in the background color in an environment will have a small impact on the policy.
Proposition 1 We consider an MDP M and a set of randomizations { φ } of this MDP.
Let π be a
K-Lipschitz policy over { φ } .
Suppose the rewards are bounded by r max such that ∀ a ∈ A , s ∈
S , | r ( s, a ) | ≤ r max .
Then for all φ 1 and φ 2 in { φ } , the following inequalities hold :
[Equation]
Where η i is the expected cumulative return of policy π φ i on MDP M φ i , for i ∈ { 1 , 2 } , and | φ 1 − φ 2 (cid:107) ∞
= sup s ∈S | φ 1 ( s ) − φ 2 ( s ) | .
Proof.
See appendix.
These inequalities shows that the smaller the Lipschitz constant, the smaller the maximum variations of the policy over the randomization space can be.
In the following, we present a regularization technique that produces low-variance policies over the randomization space by minimizing the Lipschitz constant of the policy.
P ROPOSED REGULARIZATION
We propose a simple regularization method to produce an agent with policies that vary little over randomized environments, despite being trained on only one environment.
We start by choosing one variation of the environment on which to train an agent with a policy π parameterized by θ , and during training we minimize the loss
[Equation]
where λ is a regularization parameter, L RL is the loss corresponding to the chosen reinforcement learning algorithm, the ﬁrst expectation is taken over the distribution of states visited by the current policy which we assume to be ﬁxed when optimizing this loss, and f θ is a feature extractor used by the agent’s policy.
In our experiments, we choose the output of the last hidden layer of the value or policy network as our feature extractor.
Minimizing the second term in this loss function minimizes the Lipschitz constant as deﬁned above over the states visited by the agent, and causes the agent to learn representations of states that ignore variations caused by the randomization.
Our method can be applied to many RL algorithms, since it involves simply adding an additional term to the learning loss.
In the following, we experimentally demonstrate applications to both value-based and policy-based reinforcement learning algorithms.
Implementation details can be found in the appendix, and the code will be made available online.
E XPERIMENTS 5. I LLUSTRATION ON A GRIDWORLD
We ﬁrst conduct experiments on a simple gridworld to illustrate the theory described above. Agent
[Figure]
probability Randomized 86% Regularized
[Figure]
% (ours)
The environment we use is the 3 × 3 gridworld shown in ﬁgure 2, in which two optimal policies exist.
The agent starts in the bottom left of the grid and must reach the goal while avoiding the ﬁre.
The agent can move either up or right, and in addition to the rewards shown in ﬁgure 2 receivesreward for invalid actions that would case it to leave the grid.
We set a time limit of 10 steps and γ = 1 .
We introduce randomization into this environment by describing the state observed by the agent as a tuple ( x, y, ξ ) , where ( x, y ) is the agent’s position and ξ is a randomization parameter with no impact on the underlying MDP.
For this toy problem, we consider only two possible values for ξ : +5 and − 5 .
The agents we consider use the REINFORCE algorithm (Sutton et al., 2000) with a baseline (see appendix), and a multi-layer perceptron as the policy network.
First, we observe that even in a simple environment such as this one, a randomized agent regularly learns different paths for different randomizations (ﬁgure 2).
An agent trained only on ξ =and regularized with our technique, however, consistently learns the same path regardless of ξ .
Although both agents easily solve the problem, the variance of the randomized agent’s policy can be problematic in more complex environments in which identifying similarities between domains and ignoring irrelevant differences is important.
Next, we compare the measured difference between the policies learned by regularized agents on the two domains to the smallest of our theoretical bounds in equation 1, which in this simple environment can be directly calculated.
For a given value of λ , we train a regularized agent on the reference domain.
We then measure the difference in returns obtained by this agent on the reference and on the regularized domain, and this return determines the agent’s position along the x axis.
We then numerically calculate the Lipschitz constant from the agent’s action distribution over all states, and use this constant to calculate the bound in proposition 1.
This bound determines the agent’s position along the y axis.
Our results for different random seeds and values of λ are shown in ﬁgure 2.
We observe that increasing λ does lead to decreases in both the empirical difference in returns and in the theoretical bound.
5. V ISUAL C ARTPOLE WITH DQN
We compare standard visual domain randomization to our regularization method on a more challenging visual environment, in terms of 1) training stability, 2) returns and variance of the learned policies, and 3) state representations learned by the agents.
5.2. E XPERIMENTAL S ETTING
To run domain randomization experiments, we use a visual Cartpole environment shown in ﬁgure 1, where the states consist of raw pixels of the images.
The agent must keep a pole upright as long as possible on a cart that can move left or right.
The episode terminates either after 200 time steps, if the cart leaves the track, or if the pole falls over.
The randomization consists of changing the color of the background.
Each randomized domain ξ ∈ Ξ corresponds to a color ( r, g, b ) , where 0 ≤ r, g, b ≤ 1 .
Our implementation of this environment is based on the OpenAI Gym (Brockman et al., 2016).
For training, we use the DQN algorithm with a CNN architecture similar to that used by Mnih et al.
In principle, such a value-based algorithm should learn a unique value function independently of the randomization parameters we consider.
However, as we will show function approximation errors cause different value functions to be learned for different background colors.
We compare the performance of three agents.
The Normal agent is trained on only one domain (with a white background).
The Randomized agent is trained on a chosen randomization space Ξ .
The
Regularized agent is trained on a white background using our regularization method with respect to randomization space Ξ .
The training of all three agents is done using the same hyperparameters, and over the same number of steps.
5.2. P
[Figure]
We ﬁrst compare the performance of our agents during training.
We train all three agents over two randomization spaces (environments with different background colors), having the following sizes :
• Ξ small = { ( r, g, b ) , 0 .
5 ≤ r, g, b ≤ 1 . }
=
[0 . 5 , 1] 3 : 18 of the unit cube.
Ξ big =
[0 , 1] × [0 . 5 , 1] × [0 , 1] : half the unit cube.
We obtain the training curves shown in ﬁgure 3.
We ﬁnd that the normal and regularized agents have similar training curves and are not affected by the size of the randomization space.
However, the randomized agent learns more slowly on the small randomization space Ξ small (left), and also achieves worse performance on the bigger randomization space Ξ big (right).
In high-dimensional problems, we would like to pick the randomization space Ξ to be as large as possible to increase the chances of transferring to the target domain.
We ﬁnd that standard domain randomization scales poorly with the size of the randomization space Ξ , whereas our regularization method is more robust to a larger randomization space.
5.2. G ENERALIZATION AND VARIANCE
We compare the returns of the policies learned by the agents in different domains within the randomization space.
We select a plane within Ξ big obtained by varying only the R and B channels but keeping G ﬁxed.
We plot the scores obtained on this plane in ﬁgure 4.
We see that despite having only been trained on one domain, the regularized agent achieves consistently high scores on the other domains.
On the other hand, the randomized agent’s policy exhibits returns with high variance between domains, which indicates that different policies were learned for different domains.
5.2. R EPRESENTATIONS LEARNED BY THE AGENTS
Agent
Standard
Deviations
Normal 10.
Randomized 6.
Regularized (ours) 3.
To understand what causes this difference in behavior between the two agents, we study the representations learned by the agents by analyzing the activations of the ﬁnal hidden layer.
We consider the agents trained on Ξ big , and a sample of states obtained by performing a greedy rollout on a white background (which is included in Ξ big ).
For each of these states, we calculate the representation corresponding to that state for another background color in Ξ big .
We then visualize these representations using t-SNE plots, where each color corresponds to a domain.
A representative example of such a plot is shown in ﬁgure 5.
We see that the regularized agent learns a similar representation for both backgrounds, whereas the randomized agent clearly separates them.
This result indicates that the regularized agent learns to ignore the background color, whereas the randomized agent is likely to learn a different policy for a different background color.
Further experiments comparing the representations of both agents can be found in the appendix.
To further study the effect of our regularization method on the representations learned by the agents, we compare the variations in the estimated value function for both agents over Ξ big .
Figure 5 shows the standard deviation of the estimated value function over different background colors, averaged over 10 training seeds and a sample of states obtained by the same procedure as described above.
We observe that our regularization technique successfully reduces the variance of the value function over the randomization domain.
[Figure]
To demonstrate the applicability of our regularization method to other domains and algorithms, we also perform experiments with the PPO algorithm (Schulman et al., 2017) on the CarRacing environment (Brockman et al., 2016), in which an agent must drive a car around a racetrack.
An example state from this environment and a randomized version in which part of the background changes color are shown in ﬁgure 6.
We start by training 3 agents on this domain: a normal agent on the original background, a randomized agent, and a regularized agent with λ = 50 .
Randomization in this experiment occurs over the entire RGB cube, which is larger than for the cartpole experiments.
Training curves are shown in ﬁgure 6.
We see that the randomized agent fails to learn a successful policy on this large randomization space, whereas the other agents successfully learn.
[Table] We also compare the generalization ability of these agents to other agents trained with different randomization and regularization methods. On the reference domain, we train a regularized agent with a smaller value of λ = 10 , and two agents respectively with dropout 0.1 and l2 weight decay of 10 − 4 , as in Cobbe et al. (2018) and Aractingi et al.
On the randomized domain, we train an agent with the EPOpt-PPO algorithm Rajeswaran et al.
(2016), where in our implementation the agent only trains on the randomized domains on which its score is worse than average.
Scores on both the reference domain and its randomizations are shown in 1.
These results conﬁrm that our regularization leads to agents that are both successful in training and successfully generalize to a wide range of backgrounds.
Moreover, a larger value of λ yields higher generalization scores.
We ﬁnd that none of the other techniques generalize well to the randomization space.
C ONCLUSION
In this paper we studied generalization to visually diverse environments in deep reinforcement learning.
We formalized the problem, illustrated the inefﬁciencies of standard domain randomization, and proposed a theoretically grounded method that leads to robust, low-variance policies that generalize well.
We conducted several experiments in different environments of differing complexities using both on-policy and off-policy algorithms to support our claims.
A P ROOF OF P ROPOSITION
The proof presented in the following applies to MDPs with a discrete action space.
However, it can straightforwardly be generalized to continuous action spaces by replacing sums over actions with integrals over actions.
The proof uses the following lemma :
Lemma
For two distributions p ( x, y ) =
p ( x ) p ( y | x ) and q ( x, y ) =
q ( x ) q ( y | x ) , we can bound the total variation distance of the joint distribution :
[Equation]
Proof of the Lemma .
We have that :
[Equation]
Proof of the proposition.
Let p tφ
i ( s, a ) be the probability of being in state φ
i ( s ) at time t
[Equation]
and executing action a , for i = 1 , 2 .
Since both MDPs have the same reward function, we have by deﬁnition that η i = (cid:80) s,a (cid:80) t γ t p t φ i ( s, a ) r t ( s, a ) , so we can write :
[Equation]
But p tφ 1 ( s, a ) =
p tφ 1 ( s ) π
φ 1 ( a | s ) and p tφ 2 ( s, a ) =
p tφ 2 ( s ) π
φ 2 ( a | s ) ,
Thus (Lemma 1) :
[Equation]
We still have to bound D TV
[Equation]
p tφ
[Equation] [Equation]
[Equation]
For s ∈ S we have that :
[Equation]
Summing over s we have that
[Equation]
But by marginalizing over actions : p φ 1 ( s | s (cid:48) )
= (cid:80) a π φ 1 ( a | s (cid:48) )
p φ 1 ( s | a, s (cid:48) ) , and using the fact that p φ 1 ( s | a, s (cid:48) )
=
T φ 1 ( s | a, s (cid:48) )
=
T φ 2 ( s | a, s (cid:48) )
=
p φ 2 ( s | a, s (cid:48) ) :
= p ( s | a, s (cid:48) ) , we have that
[Equation]
a
[Equation]
− π
[Equation] [Equation]
| s (cid:48) )
|
And using (cid:80) s p ( s | a, s (cid:48) )
=
1 we have that :
[Equation]
Thus, by induction, and assuming D TV ( p 0 φ 1 ( · ) ||
[Equation]
Plugging this into inequality 4, we get
[Equation]
We also note that the total variation distance takes values between 0 and
[Equation]
so we have
[Equation]
Plugging this into inequality 3 leads to our ﬁrst bound,
[Equation]
Our second, looser bound can now be achieved as follows,
[Figure]
B C OMPARISON OF OUR WORK WITH A RACTINGI ET AL . (2019)
Concurrently and independently of our work, Aractingi et al.
(2019) propose a similar regularization scheme on randomized visual domains, which they experimentally demonstrate with the PPO algorithm on the VizDoom environment Kempka et al.
(2016) with randomized textures.
As opposed to the regularization scheme proposed in our work in which we regularize the ﬁnal hidden layer of the network, they propose regularizing the output of the policy network.
Regularizing the last hidden layer as in our scheme more clearly separates representation learning and policy learning, since the ﬁnal layer of the network is only affected by the RL loss.
We hypothesized that regularizing the output of the network directly could lead to the regularization loss and the RL loss competing against each other, such that a tradeoff between policy performance and generalization would be necessary.
To test this hypothesis, we performed experiments on the visual cartpole domain with output regularization with different values of regularization parameter λ .
Our results are shown in ﬁgure 7.
We ﬁnd that increasing the regularization strength adversely affects training.
However, agents trained with higher values of λ do achieve more consistent results over the randomization space.
This shows that there is indeed a tradeoff between generalization and policy performance when regularizing the network output as in Aractingi et al.
In our experiments, however, we have found that changing the value of λ only affects generalization ability and not agent performance on the reference domain.
C E XPERIMENTAL DETAILS
All code used for our experiments will be made available online.
C. S TATE PREPROCESSING
For our implementation of the visual cartpole environment, each image consists of 84 × 84 pixels with RGB channels.
To include momentum information in our state description, we stack k =frames, so the shape of the state that is sent to the agent is 84 × 84 × 9 .
We note that because of this preprocessing, agents trained until convergence achieve average returns of about 175 instead of the maximum achievable score of 200.
Since the raw pixels do not contain momentum information, we stack three frames as input to the network.
When the environment is reset, two random actions are thus taken before the agent is allowed to make a decision.
For some initializations, this causes the agent to start in a situation it cannot recover from.
Moreover, due to the low image resolution the agent may sometimes struggle to correctly identify momentum and thus may make mistakes.
In CarRacing, each state consists of 96 × 96 pixels with RGB channels.
We introduce frame skipping as is often done for Atari games (Mnih et al.
(2015)), with a skip parameter of 5.
This restricts the length of an episode to 200 action choices.
We then stack 2 frames to include momentum information into the state description.
The shape of the state that is sent to the agent is thus 96 × 96 × 6 .
We note that although this preprocessing makes training agents faster, it also causes trained agents to not attain the maximum achievable score on this environment.
[Figure]
Given that regularized agents are stronger in interpolation over their training domain, it is natural to wonder what the performance of these agents is in extrapolation to colors not within the range of colors sampled within training.
For this purpose, we consider randomized and regularized agents trained on Ξ big , and test them on the set { ( x, x, x ) , 0 ≤ x ≤ 1 } .
None of these agents was ever exposed to x ≤ 0 .
5 during training.
Our results are shown in ﬁgure 8.
We ﬁnd that although the regularized agent consistently outperforms the randomized agent in interpolation, both agents fail to extrapolate well outside the train domain.
Since we only regularize with respect to the training space, there is indeed no guarantee that our regularization method can produce an agent that extrapolates well.
Since the objective of domain randomization often is to achieve good transfer to an a priori unknown target domain, this result suggests that it is important that the target domain lie within the randomization space, and that the randomization space be made as large as possible during training.
C.2. F URTHER STUDY OF THE REPRESENTATIONS LEARNED BY DIFFERENT AGENTS
We perform further experiments to demonstrate that the randomized agent learns different representations for different domains, whereas the regularized agent learns similar representations.
We consider agents trained on Ξ split =
[0 , 0 . 2] 3 ∪ [0 . 8 , 1] 3 , the union of darker , and lighter backgrounds.
We then rollout each agent on a single episode of the domain with a white background and, for each state in this episode, calculate the representations learned by the agent for other background colors.
We visualize these representations using the t-SNE plot shown in ﬁgure 9.
We observe that the randomized agent clearly separates the two training domains, whereas the regularized agent learns similar representations for both domains.
We are interested in how robust our agents are to unseen values ξ (cid:54)∈ Ξ split .
To visualize this, we rollout both agents in domains having different background colors : { ( x, x, x ) , 0 ≤ x ≤ 1 } , i.e ranging from black to white, and collect their features over an episode.
We then plot the t-SNEs of these features for both agents in ﬁgure 10, where each color corresponds to a domain.
We observe once again that the regularized agent has much lower variance over unseen domains, whereas the randomized agent learns different features for different domains.
This shows that the regularized agent is more robust to domain shifts than the randomized agent.
D F URTHER RELATED WORK : L IPSCHITZ CONTINUITY AND GENERALIZATION IN D EEP L EARNING
Lipschitz-sensitive bounds on the generalization abilities of neural networks have a long history (Bartlett (1997); Anthony & Bartlett (2009); Neyshabur et al.
Recently, Bartlett et al.
(2017) proved a generalization bound in terms of the norms of each layer, which is proportional to the Lipschitz constant of the network.
Oberman & Calder (2018) studied generalization through a general empirical risk minimization procedure with Lipschitz regularization, and provides generalization bounds.
Similarly, we show that the Lipschitz constant of the network with respect to the randomization parameters plays an important role in achieving zero-shot transfer to a target domain.
E A LGORITHMS
Algorithm
Initialize replay memory D to capacity N
Initialize action-value function Q with random weights θ
Initialize the randomization space Ξ , and a reference MDP M φ ref to train on.
Initialize a regularization parameter λ
Deﬁne a feature extractor f θ for episode = 1 , M do
Sample a randomizer function φ sampled uniformly from Ξ .
for t = 1 , T do
With probability (cid:15) select a random action a t otherwise select a t = arg max a Q ( φ ref ( s t ) , a ; θ )
Execute action a t in M φ ref , observe reward r t , the reference state, and the corresponding randomized state with the chosen visual settings: φ ref ( s t +1 ) , φ sampled ( s t +1 )
Store transition ( φ ref ( s t ) , φ sampled ( s t ) , a t , r t , φ ref ( s t +1 ) , φ sampled ( s t +1 )) in D
Sample random minibatch of transitions ( φ refj , φ sampled ( old ) j , a j , r j , φ refj +1 , φ sampled ( old ) j + ) from D , where φ sampled ( old )
j is the randomization that had been selected when the transition had been observed.
Set y j
= 
 r j for terminal φ refj +1 .
r j + γ max a (cid:48) Q ( φ refj +1 , a (cid:48) , θ ) otherwise.
Perform a gradient descent step on ( y j − Q ( φ refj , a j ; θ ))
+ λ (cid:107) f
θ ( φ refj ) − f θ ( φ sampled ( old ) j ) (cid:107)end for end for
Algorithm
Initialize policy network function π θ with random weights θ , baseline b θ
Initialize the randomization space Ξ , and a reference MDP M φ ref to train on.
Initialize a regularization parameter λ
Deﬁne a feature extractor f θ for episode = 1 , M do
Collect a set of trajectories ( s 0 , a 0 , r 1 , . . .
, s T − 1 , a T − 1 , r T ) by executing π θ on M φ ref .
For each trajectory, sample a randomizer function φ sampled uniformly from Ξ .
for t = 1 , T in each trajectory do
Compute the return R t =
(cid:80) T − 1 t (cid:48) =
t γ t (cid:48) − t r t (cid:48)
Estimate the advantage ˆ A t =
R t − b θ ( φ ref ( s t )) end for
Perform a gradient descent step on
T (cid:88) t = (cid:2) − ˆ A t log π θ ( a t | φ ref ( s t ))
+ (cid:107) R t − b θ ( φ ref ( s t ))
(cid:107)+ λ (cid:107) f θ ( φ ref ( s t )) − f θ ( φ sampled ( s t ))
(cid:107) 22 (cid:3) end for
Note that algorithm 2 can be straightforwardly adapted to several state of the art policy gradient algorithms such as PPO.
F A DYNAMICS RANDOMIZATION EXPERIMENT
We also perform an experiment to demonstrate that learning domain-invariant features can be helpful not only for visual randomization, but also for some instances of dynamics randomization.
We consider once again the Cartpole domain, where this time we randomize some physical dynamics of the environment.
Speciﬁcally, we choose to randomize the pole length l , and the gravity g .
The state for our reinforcement learning agent is the 6 -dimensional vector ( x, ˙ x, θ, ˙ θ, l, g ) , where x is the position of the cart, ˙ x its velocity, θ the angle between the pole and the horizontal plane, and ˙ θ the angular velocity.
We train all three agents (Normal, Regularized, Randomized) using only two randomized domains : l = 0 .
5 , g = 9 . 8 and l = 1 , g = 50 .
We use the DQN algorithm with a
Multi-Layer Perceptron architecture for this experiment.
We ﬁrst compare the training speed of the agents.
The training curves averaged over 100 seeds are plotted in ﬁgure 11.
We observe once again that the randomized agent is signiﬁcantly slower than the regularized one, and is more unstable.
Next, we examine the agents’ generalization ability.
We test the agents on environments having values of pole length l and gravity g unseen during training.
We plot their scores in ﬁgure 12.
The randomized agent clearly specializes on the two different training domains, corresponding to the two clearly distinguishable regions where high scores are achieved, whereas the regularized agents achieves more consistent scores across domain.
This result can be understood as follows.
Although the different dynamics between the two domains lead to there being different sets of optimal policies, our regularization method forces the agent to only learn policies that do not depend on the speciﬁc values of the randomized dynamics parameters.
These policies are therefore more likely to also work when those dynamics are different.
F. R EPRESENTATIONS LEARNED BY THE AGENT
We analyze the representations learned by each agent in our dynamics randomization experiment.
Once the agents are trained, we rollout their policies in both randomized environments with an (cid:15) - greedy strategy, where we use (cid:15) = 0 .
2 to reach a larger number of states of the MDP, oversteps.
We collect the representations (the activations of the last hidden layer) corresponding to the visited states.
These features are 100 -dimensional, so in order to visualize them, we use the t-SNE plots shown in ﬁgure 13.
We emphasize that although this ﬁgure corresponds to a single training seed, the general aspect of these results is repeatable.
The randomized agent learns completely different representations for the two randomized environments.
This explains its high variance during the training, since it tries to learn a different strategy for each domain.
On the other hand, our regularized agent has the same representation for both domains, which allows it to learn much faster, and to learn policies that are robust to changes in the environment’s dynamics.