Original: C ROSS -M ODAL D OMAIN A DAPTATION FOR R EINFORCEMENT L EARNING
Revised: C ROSS -M ODAL D OMAIN A DAPTATION FOR R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Domain adaptation is a promising direction for deploying RL agents in real-world applications, where vision-based robotics tasks constitute an important part.
Revised: To overcome the unbearable reinforcement training of agents in the real-world, the sim-to-real approach, i.e., training in simulators and adapting to target environments, is a promising direction.

Original: Current methods that train polices on simulated images not only require a delicately crafted simulator, but also add extra burdens to the training process.
Revised: However, crafting a delicately simulator can also be difﬁcult and costly. For example, to simulate vision-based robotics, simulators have to render high-ﬁdelity images, which can cost tremendous effort.

Original: In this paper, we propose a method that can learn a  mapping from high-dimensional images to low-level simulator states, allowing agents trained on the source domain of state input to transfer well to the target domain of image input.
Revised: This work aims at learning a cross-modal mapping between intrinsic states of the simulator and high-dimensional observations of the target environments. This cross-modal mapping allows agents trained on the source domain of state input to adapt well to the target domain of image input.

Original: By fully leveraging the sequential information in the trajectories and incorporating the policy to guide the training process, our method overcomes the intrinsic ill-posedness in cross-modal domain adaptation when structural constraints from the same modality are unavailable.
Revised: However, learning the cross-modal mapping can be ill-posed for previous same-modal domain adaptation methods, since the structural constraints no longer exist. We propose to leveraging the sequential information in the trajectories and incorporating the policy to guide the training process.

Original: Experiments on MuJoCo environments show that the policy, once combined with the mapping function, can be deployed directly in the target domain with only a small performance gap, while current methods designed for same-modal domain adaptation fail on this problem.
Revised: Experiments on MuJoCo environments show that the proposed crossmodal domain adaptation approach enables the agents to be deployed directly in the target domain with only a small performance gap, while previous methods designed for same-modal domain adaptation fail on this task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Reinforcement Learning (DRL) for vision-based robotic-control tasks has achieved remarkable success in recent years (Francis et al., 2020; Zhang et al., 2019; Zeng et al., 2018; Riedmiller et al., 2018; Levine et al., 2018).
Revised: Deep Reinforcement Learning (DRL) for vision-based robotic-control tasks has achieved remarkable success in recent years (Francis et al., 2020; Zhang et al., 2019; Zeng et al., 2018; Riedmiller et al., 2018; Levine et al., 2018).

Original: However, current RL algorithms necessitate a substantial number of interactions with the environment, which is costly both in time and money on real robots.
Revised: However, current RL algorithms necessitate a substantial number of interactions with the environment, which is costly both in time and money on real robots.

Original: An appealing alternative is to train policies in simulators, then transfer these policies onto real-world systems (Rao et al., 2020; James et al., 2019; Yan et al., 2017).
Revised: An appealing alternative is to train policies in simulators, then transfer these policies onto real-world systems (Rao et al., 2020; James et al., 2019; Yan et al., 2017).

Original: Due to inevitable differences between simulators and the real world, which is also known as the “reality gap” (Jakobi et al., 1995), applying policies trained in one domain directly to another almost surely fail, especially in visual-input tasks, due to the poor generalization of RL polices (Cobbe et al., 2019).
Revised: Due to inevitable differences between simulators and the real world, which is also known as the “reality gap” (Jakobi et al., 1995), applying policies trained in one domain directly to another almost surely fail, especially in visual-input tasks, due to the poor generalization of RL polices (Cobbe et al., 2019).

Original: Domain adaptation is a common way to improve transferability by mapping inputs from two domains to an aligned distribution.
Revised: Domain adaptation is a common way to improve transferability by mapping inputs from two domains to an aligned distribution.

Original: Although distribution alignment is difﬁcult with limited data, many recent works have adopted unsupervised visual domain adaptation (Hoffman et al., 2018;
Revised: Although distribution alignment is difﬁcult with limited data, many recent works have adopted unsupervised visual domain adaptation (Hoffman et al., 2018;

Original: Ganin et al., 2017; Yi et al., 2017; Kim et al., 2017) to learn the mapping function without a groundtruth pairing.
Revised: Ganin et al., 2017; Yi et al., 2017; Kim et al., 2017) to learn the mapping function without a groundtruth pairing.

Original: These adaptation methods exploit structural constraints (Fu et al., 2019) in two samemodal domains (i.e. learned on simulated images and deployed on real images ) to overcome the intrinsic ill-posedness of distribution matching as shown in Fig. 1(a) — mapping an instance in the target domain to anything of a similar probability in the source domain is “reasonable” if we only consider distribution matching.
Revised: These adaptation methods exploit structural constraints (Fu et al., 2019) in two samemodal domains (i.e., learned on simulated images and deployed on real images ) to overcome the intrinsic ill-posedness of distribution matching as shown in Fig. 1(a) — mapping an instance in the target domain to anything of a similar probability in the source domain is “reasonable” if we only consider distribution matching.

Original: However, training on simulated images introduces unwanted costs and difﬁculties, which are ignored in current works.
Revised: However, training on simulated images introduces unwanted costs and difﬁculties, which are ignored in current works.

Original: First, a rendering engine needs more human engineering and runs much slower (can be up to 20x slower according to Xia et al.
Revised: First, a rendering engine needs more human engineering and runs much slower (can be up to 20 × slower according to Xia et al.

Original: than a pure rigid body simulator, which adds considerable cost to the overall process.
Revised: than a pure rigid body simulator, which adds considerable cost to the overall process.

Original: Second, using RL methods to train a policy with image inputs is usually harder than training with state inputs (Kaiser et al., 2020; Tenenbaum, 2018), resulting in a sub-optimal simulation policy.
Revised: Second, using RL methods to train a policy with image inputs is usually harder than training with state inputs (Kaiser et al., 2020; Tenenbaum, 2018), resulting in a sub-optimal simulation policy.

Original: An ideal solution to avoid such problems is to train policies with simulated states and adapt the learned polices to real-world images.
Revised: An ideal solution to avoid such problems is to train policies with simulated states and adapt the learned polices to real-world images.

Original: However, all the structural constraints based on the modality consistency can not be used and the distribution alignment task by learning a mapping function becomes hard to solve.
Revised: However, all [Figure] 𝑡 ? [Figure]

Revised: 𝑠 𝑡 ′

Original: [Figure]
Revised: [Figure]

Original: 𝑜 𝑡 𝑠 𝑡 𝑠 𝑡 ′ 𝑜 𝑡− 𝑜 𝑡+ 𝑠 𝑡− 𝑠
Revised: 𝑜 𝑡 𝑠 𝑡 𝑠 𝑡 ′ 𝑜 𝑡− 𝑜 𝑡+ 𝑠 𝑡− 𝑠

Original: 𝑡+ ✓ ✗
Revised: 𝑡+ ✓ ✗

Original: [Figure]
Revised: [Figure]

Revised: instances, but only s t correct.

Revised: Since they are of similar probabilities, distribution matching may map o t to any of them.

Revised: In RL, the policy may output unreliable actions when taking these incorrectly mapped states as inputs.

Revised: In Fig. 1(b), a sequential structure can help rule out the wrong mapping based on trajectory contexts.

Revised: the structural constraints based on the modality consistency can not be used and the distribution alignment task by learning a mapping function becomes hard to solve.

Original: In this paper, we propose Cross-mOdal Domain Adaptation with Sequential structure (CODAS) that learns a mapping function from images in the target domain(real world) to states in the source domain (simulation).
Revised: In this paper, we propose Cross-mOdal Domain Adaptation with Sequential structure (CODAS) that learns a mapping function from images in the target domain to states in the source domain.

Original: The policy trained  in the simulator can be deployed in the real world directly with the help of the learned mapping function.
Revised: With the help of the learned mapping function, policies trained on states in the source domain can be deployed in the target domain of images directly.

Original: Speciﬁcally, based on the sequential nature of reinforcement learning problems, we formulate the cross-domain adaptation problem as a sequential variational inference problem and derive a series of solvable optimization objectives in CODAS.
Revised: Speciﬁcally, based on the sequential nature of reinforcement learning problems, we formulate the cross-domain adaptation problem as a sequential variational inference problem and derive a series of solvable optimization objectives in CODAS.

Original: It is worth noting that our work is different from recent works that learn state embeddings from image inputs, which map images to an arbitrary subspace in a low-dimensional vector space.
Revised: It is worth noting that our work is different from recent works that learn state embeddings from image inputs, which map images to an arbitrary subspace in a low-dimensional vector space.

Original: In CODAS, we embed the image space into a vector space with clear meanings (deﬁned in the state-based simulator), which improves the interpretability of the policy when deployed in the real world.
Revised: In CODAS, we embed the image space into a vector space with clear meanings (deﬁned in the statebased simulator), which improves the interpretability of the policy when deployed in the real world.

Original: We evaluate our method on 6 MuJoCo (Todorov et al., 2012) environments provided in OpenAI
Revised: We evaluate our method on 6 MuJoCo (Todorov et al., 2012) environments provided in OpenAI

Original: Gym (Brockman et al., 2016), where we treat states/rendered images as the source/target domain, respectively.
Revised: Gym (Brockman et al., 2016), where we treat states as the source domain, and rendered images as the target domain, respectively.

Original: We do the experiments in the scenario where only ofﬂine real data are available.
Revised: Experiments are conducted in the scenario where only ofﬂine real data are available.

Original: Experiment results show that the mapping function learned by our method can help transfer the policy to real-world images with a small performance degradation.
Revised: Experiment results show that the mapping function learned by our method can help transfer the policy to target domain images with a small performance degradation.

Original: Previous methods that use unaligned Generative Adversarial Networks (GANs) suffer from a severe performance degradation on this cross-modal transfer problem.
Revised: Previous methods that use unaligned Generative Adversarial Networks (GANs) suffer from a severe performance degradation on this cross-modal transfer problem.

Revised: The experiments provide an optimistic result which indicates cross-modal domain adaptation can serve as a low-cost Sim2Real approach.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: To our best knowledge, this work is the ﬁrst to address cross-modal domain adaptation in RL setting.
Revised: To our best knowledge, this work is the ﬁrst to address cross-modal domain adaptation in RL setting.

Original: We will discuss two research areas closely related to this topic, which are, (1) unsupervised visual domain adaptation in RL and (2) image-input representation learning in RL. 
Revised: We will discuss two research areas closely related to this topic, which are, (1) unsupervised visual domain adaptation in RL and (2) image-input representation learning in RL. 2. V ISUAL D OMAIN A DAPTATION IN RL

Original: Unsupervised visual domain adaptation aims to map the source domain and the target domain to an aligned distribution without pairing the data.
Revised: Unsupervised visual domain adaptation aims to map the source domain and the target domain to an aligned distribution without pairing the data.

Original: Prior methods fall into two major categories: feature-level adaptation, where domain-invariant features are learnt (Gopalan et al., 2011; Caseiro et al., 2015; Long et al., 2015; Ganin et al., 2017), and pixel-level adaptation, where pixels from a source image used to generate an image that looks like one from the target domain (Bousmalis et al., 2017; Yoo et al., 2016; Taigman et al., 2017; Hoffman et al., 2018).
Revised: Prior methods fall into two major categories: featurelevel adaptation, where domain-invariant features are learnt (Gopalan et al., 2011; Caseiro et al., 2015; Long et al., 2015; Ganin et al., 2017), and pixel-level adaptation, where pixels from a source image used to generate an image that looks like one from the target domain (Bousmalis et al., 2017; Yoo et al., 2016; Taigman et al., 2017; Hoffman et al., 2018).

Original: Pixel-level adaptation is challenging when data from two domains are unpaired.
Revised: Pixel-level adaptation is challenging when data from two domains are unpaired.

Original: Prior works tackle this problem by using GANs (Goodfellow et al., 2014) conditioned on simulated images to generate realistic images.
Revised: Prior works tackle this problem by using GANs (Goodfellow et al., 2014) conditioned on simulated images to generate realistic images.

Original: Gamrian & Goldberg (2019) transfers policies from Atari Games (Bellemare et al., 2013) to modiﬁed variants by training a GAN to map images from the target domain to the source domain.
Revised: Gamrian & Goldberg (2019) transfers policies from Atari Games (Bellemare et al., 2015) to modiﬁed variants by training a GAN to map images from the target domain to the source domain.

Original: GraspGAN (Bousmalis et al., 2018) addresses domain adaptation in robotic grasping by having the GAN reproduce the segmentation mask for the simulated image as an auxiliary task, including the robot arm, objects, and the bin.
Revised: GraspGAN (Bousmalis et al., 2018) addresses domain adaptation in robotic grasping by having the GAN reproduce the segmentation mask for the simulated image as an auxiliary task, including the robot arm, objects, and the bin.

Original: RCAN (James et al., 2019) adopts ideas from domain randomization by learning a mapping of images from randomized simulations to a canonical simulation and treating the real world just as one of the random simulations.
Revised: RCAN (James et al., 2019) adopts ideas from domain randomization by learning a mapping of images from randomized simulations to a canonical simulation and treating the real world just as one of the random simulations.

Original: RL-CycleGAN (Rao et al., 2020) uniﬁes the learning of a CycleGAN (Zhu et al., 2017) and an RL policy, claiming better performance by learning features that are most crucial to the Q-function in RL.
Revised: RL-CycleGAN (Rao et al., 2020) uniﬁes the learning of a CycleGAN (Zhu et al., 2017) and an RL policy, claiming better performance by learning features that are most crucial to the Q-function in RL.

Original: Image-to-image domain adaptation can somewhat bypass the ill-posedness for distribution matching (See Fig. 1(a)) since it often enjoys an implicit advantage that images differ locally,e.g. in color, textile, lighting, but resembles globally between two domains, while images and states differ essentially.
Revised: Image-to-image domain adaptation can somewhat bypass the ill-posedness for distribution matching (See Fig. 1(a)) since it often enjoys an implicit advantage that images differ locally, in color, textile, lighting, but resembles globally between two domains, while images and states differ essentially.

Original: Some works impose extra structural constraints (e.g. segmentation, geometry) (Fu et al., 2019;
Revised: Some works impose extra structural constraints (e.g., segmentation, geometry) (Fu et al., 2019;

Original: Bousmalis et al., 2018), but such tricks fail in image-to-state domain adaptation either.
Revised: Bousmalis et al., 2018), but such tricks fail in image-to-state domain adaptation either.

Original: In this work, we force the mapped states to follow transition consistency by using a recurrent structure (See
Revised: In this work, we force the mapped states to follow transition consistency by using a recurrent structure (See

Original: Fig. 1(b)) and to be able to recover the pre-learned policy.
Revised: Fig. 1(b)) and to be able to recover the pre-learned policy.

Original: We also employ a stochastic mapping function with the help of a variational encoder that is more robust to real-world data noise.
Revised: We also employ a stochastic mapping function with the help of a variational encoder that is more robust to target domain data noise.

Revised: 2. R EPRESENTATION L EARNING IN RL

Original: Representation learning aims to transform high-dimensional data into lower-dimensional vector representations, which suit RL better.
Revised: Representation learning aims to transform high-dimensional data into lower-dimensional vector representations, which suit RL better.

Original: It is widely accepted that learning policies from states (embeddings) is signiﬁcantly more sample-efﬁcient than learning from pixels, both empirically (Kaiser et al., 2020; Tenenbaum, 2018; Tassa et al., 2018) and theoretically (Jin et al., 2020).
Revised: It is widely accepted that learning policies from states (embeddings) is signiﬁcantly more sample-efﬁcient than learning from pixels, both empirically (Kaiser et al., 2020; Tenenbaum, 2018; Tassa et al., 2018) and theoretically (Jin et al., 2020).

Original: Sequential auto-encoder is a common network structure to learn state representations by minimizing reconstruction loss.
Revised: Sequential auto-encoder is a common network structure to learn state representations by minimizing reconstruction loss.

Original: Early works on DRL from images (Ha & Schmidhuber, 2018; Lange et al., 2012; Lange & Riedmiller, 2010) use a two-step learning process where an auto-encoder is ﬁrst trained to learn a low-dimensional representation, and subsequently a policy or model is learned based on this representation.
Revised: Early works on DRL from images (Ha & Schmidhuber, 2018; Lange et al., 2012; Lange & Riedmiller, 2010) use a two-step learning process where an auto-encoder is ﬁrst trained to learn a low-dimensional representation, and subsequently a policy or model is learned based on this representation.

Original: Later works on model-based RL improve representation learning by jointly training the encoder and the dynamics model end-to-end (Watter et al., 2015) – this has been proved effective in learning useful task-oriented representations.
Revised: Later works on model-based RL improve representation learning by jointly training the encoder and the dynamics model end-to-end (Watter et al., 2015) – this has been proved effective in learning useful task-oriented representations.

Original: PlaNet (Hafner et al., 2019) learns a hybrid of stochastic and deterministic latent state models using a reconstruction loss.
Revised: PlaNet (Hafner et al., 2019) learns a hybrid of stochastic and deterministic latent state models using a reconstruction loss.

Original: Solar (Zhang et al., 2019) combines probabilistic graphic models with a simple network structure to ﬁt local linear transitions.
Revised: Solar (Zhang et al., 2019) combines probabilistic graphic models with a simple network structure to ﬁt local linear transitions.

Original: Some recent works adopt advancements in unsupervised representation learning.
Revised: Some recent works adopt advancements in unsupervised representation learning.

Original: CURL (Laskin et al., 2020b) utilizes contrastive learning methods to capture essential information in an image that distinguishes from others, though later works (Laskin et al., 2020a; Kostrikov et al., 2020) point out that data augmentation may play the major role here.
Revised: CURL (Laskin et al., 2020b) utilizes contrastive learning methods to capture essential information in an image that distinguishes from others, though later works (Laskin et al., 2020a; Kostrikov et al., 2020) point out that data augmentation may play the major role here.

Original: Our work utilizes a sequential variational encoder structure to capture sequential information from trajectories.
Revised: Our work utilizes a sequential variational encoder structure to capture sequential information from trajectories.

Original: The main difference between our work and representation learning is whether the state space is predeﬁned.
Revised: The main difference between our work and representation learning is whether the state space is predeﬁned.

Original: We add extra supervised information to guide the training of the mapping by minimizing the distance between the distributions of the mapped states and the original states, and by enforcing the policy to recover the actions from the mapped states.
Revised: We add extra supervised information to guide the training of the mapping by minimizing the distance between the distributions of the mapped states and the original states, and by enforcing the policy to recover the actions from the mapped states.

Original: As a result, we successfully learn states that match the ground-truth simulator states well.
Revised: As a result, we successfully learn states that match the ground-truth simulator states well.

Original: The mapped states can be directly fed into the pre-trained policy network.
Revised: The mapped states can be directly fed into the pre-trained policy network.

Original:  C ROSS - MODAL D OMAIN A DAPTATION WITH S EQUENTIAL S TRUCTURE
Revised:  C ROSS - MODAL D OMAIN A DAPTATION WITH S EQUENTIAL S TRUCTURE

Original: Our work follows the problem setting similar to previous methods that tackle visual domain adaptation problems in RL.
Revised: Our work follows the problem setting similar to previous methods that tackle visual domain adaptation problems in RL.

Original: We have a policy π pre-trained in the source domain (state) and a dataset pre-collected in the target domain (image).
Revised: We have a policy π pre-trained in the source domain (state) and a dataset pre-collected in the target domain (image).

Original: The task is to learn a mapping q φ from images to states.
Revised: The task is to learn a mapping q φ from images to states.

Original: In the deployment, agents interact with a new policy ˜ π
Revised: In the deployment, agents interact with a new policy ˜ π

Original: ( o ) =
Revised: ( o ) =

Original: ( π ◦ q φ )( o ) , where ◦ denotes function composition.
Revised: ( π ◦ q φ )( o ) , where ◦ denotes function composition.

Original: During the training of the mapping function, only the source domain is accessible.
Revised: During the training of the mapping function, only the source domain is accessible.

Original: This section is organized as follows.
Revised: This section is organized as follows.

Original: Sec.
Revised: Sec.

Original: 3.1 formulates the cross-modal domain adaptation as a variational inference problem.
Revised: 3.1 formulates the cross-modal domain adaptation as a variational inference problem.

Original: Sec.
Revised: Sec.

Original: 3.2 decomposes the variational inference problem into several feasible optimization objectives.
Revised: 3.2 decomposes the variational inference problem into several feasible optimization objectives.

Original: Sec.
Revised: Sec.

Original: 3.3 proposes a residual network structure to handle the complex long-horizon training of the sequential structure. 
Revised: 3.3 proposes a residual network structure to handle the complex long-horizon training of the sequential structure. [Figure]

Original: 3. D OMAIN A DAPTATION AS V ARIATIONAL I NFERENCE
Revised: D OMAIN A DAPTATION AS V ARIATIONAL I NFERENCE

Original: The goal of domain adaptation is to ﬁnd a mapping from the target domain to the source domain, which is images to states in our case.
Revised: The goal of domain adaptation is to ﬁnd a mapping from the target domain to the source domain, which is images to states in our case.

Original: We ﬁrst formulate the generation process of the real world and simulation and the connection between them, as illustrated in Fig.
Revised: We ﬁrst model the generation process of the target domain and source domain respectively and the connection between them, as illustrated in Fig.

Original: The initial state follows distribution p ( s 1 ) .
Revised: The initial state follows distribution p ( s 1 ) .

Original: The transition function ˆ p ϕ ( s t | s t − 1 , a t − 1 ) , modeled as a feed-forward neural network with parameters ϕ , predicts the current state from the previous state and previous action.
Revised: The transition function ˆ p ϕ ( s t | s t − 1 , a t − 1 ) , modeled as a feed-forward neural network with parameters ϕ , predicts the current state from the previous state and previous action.

Original: The decoder p θ ( o t | s t , a t − 1 , o t − 1 ) , modeled as a deconvolution network with parameters θ , reconstructs the current observation from the current state, previous observation and previous action.In practice, [Figure] [Equation]
Revised: The decoder p θ ( o t | s t , a t − 1 , o t − 1 ) , modeled as a deconvolution network with parameters θ , reconstructs the current observation from the current state, previous observation and previous action.

Original: [Figure]

Original:  we model observations and states as multivariate Gaussian distributions.
Revised: In practice, we model these distributions as multivariate Gaussian distributions.

Original: We allow p θ ( o t | · ) dependent on o t − 1 so that some irrelevant patterns in images can be reconstructed in an auto-regressive manner.
Revised: We allow p θ ( o t | · ) dependent on o t − 1 so that some irrelevant patterns in images can be reconstructed in an auto-regressive manner.

Original: Such a conditional distribution is feasible in practice since we always have ground-truth o 0: t − 1 at timestep t in both training and deployment phases.
Revised: Such a conditional distribution is feasible in practice since we always have ground-truth o 0: t − 1 at timestep t in both training and deployment phases.

Original: Such a generation process enjoys an extra beneﬁt of solving Partially Observable Markov Decision Process (POMDP) since a single image generally does not reveal the full state of the environment.
Revised: A sequential generation process enjoys an extra beneﬁt of solving Partially Observable Markov Decision Process (POMDP) since a single image can not reveal the full state of the environment in general.

Original: The key point that distinguishes our method from conventional representation learning is the additional constraint that the mapped state trajectories should match the simulation state trajectories.
Revised: The key point that distinguishes our method from conventional representation learning is the additional constraint that the mapped state trajectories should match the simulation state trajectories.

Original: See rounded rectangle parts in Fig.
Revised: See rounded rectangle parts in Fig.

Original: Such a constraint may somewhat require that the underlying transition dynamics are the same in two domains.
Revised: Such a constraint may somewhat require that the underlying transition dynamics are the same in two domains.

Original: Introducing distributions can to some extent relax the assumption and increase robustness when there is a small mismatch between  source and target transition dynamics.
Revised: Introducing distributions can to some extent relax the assumption and increase robustness when there is a small mismatch between dynamics in source and target.

Original: [Equation]
Revised: [Equation]

Original: The overall optimization problem can be formulated as a variational inference problem described in
Revised: The overall optimization problem can be formulated as a variational inference problem described in

Original: Eq.2, where we want to learn a posterior distribution to approximate the ground-truth distribution.
Revised: Eq. 2, where we want to learn a posterior distribution to approximate the ground-truth distribution.

Original: [Equation]
Revised: [Equation]

Original: where τ is a trajectory, s and r indicate whether the trajectory is from the simulator or the real world, p ( · ) is the ground-truth distribution, q φ ( · ) is the mapping function we want to learn, and D KL computes the Kullback-Leibler divergence.
Revised: where τ is a trajectory, s and r indicate whether the trajectory is from the source or target domain, p ( · ) is the ground-truth distribution, q φ ( · ) is the mapping function we want to learn, and D KL computes the Kullback-Leibler divergence.

Original: Modeling the optimization as a trajectory distribution matching can naturally handle the stochasticity of environments and policies, and the possible noise in data collected in the real world.
Revised: Modeling the optimization as a trajectory distribution matching can naturally handle the stochasticity of environments and policies, and the possible noise in data collected in the real world.

Original: The Evidence Lower Bound (ELBO) of this variational problem can be formulated as:
Revised: The Evidence Lower Bound (ELBO) of this variational problem can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: The derivation of ELBO follows a common practice using Jensen inequality.
Revised: The derivation of ELBO follows a common practice using Jensen inequality.

Original: A detailed derivation can be found in Appendix A.
Revised: A detailed derivation can be found in Appendix A.

Original: The ﬁrst term maximizes the reconstruction probability, in order to enforce that the mapped states ˆ s can recover both observations o and action a . 
Revised: The ﬁrst term maximizes the reconstruction probability, in order to enforce that the mapped states ˆ s can recover both observations o and action a . The second term

Revised: Sample a batch of target domain trajectories τ r = { ( r a r o rT ) i }

Original: The second term enforces the alignment of the distributions of the mapped trajectories and the trajectories collected in the simulator.
Revised: Initialize RNN with a zero state. Infer the corresponding state trajectories τ s = { (ˆ s 1 , ..., ˆ s T ) i } via the mapping function f Rollout one with the oracle simulation dynamics p ( s | s a ) for each state-action pair in to construct the transition dataset enforces the alignment of the distributions of the mapped trajectories and the trajectories collected in the simulator.

Original: The policy π in source domains included in the second term does introduce a new assumption that real-world data are collected by a known behavioral policy.
Revised: The policy π in source domains included in the second term does introduce a new assumption that target domain data are collected by a known behavioral policy.

Original: This assumption is mild and is implicitly or explicitly introduced in previous works (Kim et al., 2020; Gamrian & Goldberg, 2019).
Revised: This assumption is mild and is implicitly or explicitly introduced in previous works (Kim et al., 2020; Gamrian & Goldberg, 2019).

Original: 3. D IFFERENTIABLE O PTIMIZATION O BJECTIVES
Revised: 3. D IFFERENTIABLE O PTIMIZATION O BJECTIVES

Original: The ELBO deﬁned in Eq.
Revised: The ELBO deﬁned in Eq.

Original: 3 contains terms involving distributions over the entire trajectory, and thus is impractical to solve directly.
Revised: 3 contains terms involving distributions over the entire trajectory, and thus is impractical to solve directly.

Original: Given the generation process deﬁned in the previous section, we can Initialize Sample a real-world trajectories τ r = { Initialize a state
Revised: Given the generation process deﬁned in the previous section, we can decompose the joint probability into the multiplication of one-step probabilities.

Original: Infer the state trajectories decompose the joint probability into the multiplication of one-step probabilities.

Original: Eq.
Revised: Eq.

Original: 4 shows the result after the decomposition.
Revised: 4 shows the result after the decomposition.

Original: For brevity, we will use ˆ s, ˆ o to denote s, o outputted by networks and omit the networks themselves.
Revised: For brevity, we will use ˆ s, ˆ o to denote s, o outputted by networks and omit the networks themselves.

Original: [Equation]
Revised: [Equation]

Original: A direct computation of the secondterm D KL  is intractable.
Revised: A direct computation of the second D KL term is intractable.

Original: Following the idea that the optimization process of GAN is equivalent to minimizing a certain distance measure between two distributions (Nowozin et al., 2016), we can use the optimization objective of GAN as the surrogate objective of minimizing the D KL [ q φ ( τ s | τ r ) ||
Revised: Following the idea that the optimization process of GAN is equivalent to minimizing a certain distance measure between two distributions (Nowozin et al., 2016), we can use the optimization objective of GAN as the surrogate objective of minimizing the D KL [ q φ ( τ s | τ r ) ||

Original: p π ( τ s )] .
Revised: p π ( τ s )] .

Original: For implementation simplicity and training stability, we still choose to optimize the original GAN optimization objective, which is  formulated as [Equation]
Revised: For implementation simplicity and training stability, we still choose to use the original GAN optimization objective, which is equivalent to minimizing JS divergence. The optimization objective is formulated in Eq.

Original: [Equation]
Revised: [Equation]

Original: Here, the discriminator takes state, rather than trajectory, as input for better practicality, where “real” samples are simulation states and “fake” samples are mapped states.
Revised: Here, the discriminator takes state, rather than trajectory, as input for better practicality, where “real” samples are simulation states and “fake” samples are mapped states.

Original: The latter term is the objective of the generator - q φ in our case.
Revised: The latter term is the objective of the generator - q φ in our case.

Original: The sequential structure is preserved in the generator.
Revised: The sequential structure is preserved in the generator.

Original: This part is different from  VAE-GAN (Larsen et al., 2016), which discriminates the reconstructed output of the variational encoder-decoder.
Revised: This part is different from previous works like VAE-GAN (Larsen et al., 2016) and Causal InfoGAN (Kurutach et al., 2018), that discriminate against the reconstructed output.

Original: Instead, we discriminate against the states after the variational encoder.
Revised: Instead, we discriminate against the states after the variational encoder.

Original: GAN merely acts as an optimizer to minimize an intractable divergence between the distributions here.

Original: The reconstruction of action a can be optimized in an end-to-end manner.
Revised: The reconstruction of action a can be optimized in an end-to-end manner.

Original: However, if a differentiable π b is available, we can replace p θ π with π , resulting an objective in Eq.
Revised: However, if a differentiable π of simulation states is available (which is the usual case), we can replace p θ π with π , resulting an objective in Eq.

Original: A ﬁxed π here can guide p θ s to output states that yields  similar a .
Revised: A ﬁxed π here can guide p θ s to output states that yields a similar a .

Original: There is a similar empirical observation that recovering a is helpful in improving the representation learning in RL (Schrittwieser et al., 2019).
Revised: Previous works (Schrittwieser et al., 2019) also discovered that recovering a is helpful in improving the representation learning in RL.

Original: [Equation]
Revised: ∼ D t = [log π ( a rt | ˆ s t ))] (6)

Original: Combining all the aforementioned losses together, i.e, reconstruction loss, policy loss and generation loss, the complete optimization objective of the mapping function is as follows:
Revised: Combining all the aforementioned losses together, i.e, reconstruction loss, policy loss and generation loss, the complete optimization objective of the mapping function is as follows:

Original: [Equation]
Revised: [Equation]

Original: Input: Simulator with oracle dynamics p ; Policy π ( a | s ) pre-trained in the simulation dynamics p

Original: s (cid:48) | s, a ) ; # Initialization.

Original: Pre-train DM t ϕ with D s = { ( s i , a i , s i +1 ) } .

Original: # Update per iteration. Update the the DM t ϕ ˆ p by minimizing Eq.8 using D ˆ s and D s ; Copy the new parameters ϕ of DM t ϕ to the embed-DM in q φ ; for m = 1 to M do Update the mapping function q φ by minimizing Eq.7; Update the reconstruction function p θ by minimizing the ﬁrst term in Eq.7; end for where λ D and λ π are hyper-parameters.
Revised: where λ D and λ π are hyper-parameters.

Original: Both the decoder and the policy are ﬁxed during the training process of the mapping function.
Revised: Both the decoder and the policy are ﬁxed during the training process of the mapping function.

Original: The loss function for p θ is just the ﬁrst term in Eq.7.
Revised: The loss function for p θ is just the ﬁrst term in Eq.

Original: Similarly, the mapping function q φ is ﬁxed during the training process of the reconstruction function p θ .
Revised: Similarly, the mapping function q φ is ﬁxed during the training process of the reconstruction function p θ .

Original: 3. S TATE I NFERENCE M ODEL WITH E MBEDDED D YNAMICS
Revised: 3. S TATE I NFERENCE M ODEL WITH E MBEDDED D YNAMICS

Original: Since a trajectory of practical RL problems often lasts hundreds or thousands timesteps, training
Revised: Since a trajectory of practical RL problems often lasts hundreds or thousands timesteps, training

Original: RNN on such a long-horizon trajectory is difﬁcult.
Revised: RNN on such a long-horizon trajectory is difﬁcult.

Original: The previous success of ResNet (He et al., 2016)empirically shows that a residual structure can simplify the learning target by predicting small residuals, resulting in a remarkable performance increase in learning ultra-deep neural networks.
Revised: The previous success of ResNet (He et al., 2016) shows that a residual structure can simplify the learning target by predicting small residuals, resulting in a remarkable performance increase in learning ultra-deep neural networks.

Original: [Figure]
Revised: RNN cell 𝑜 𝑡 Ƹ𝑠 𝑡− embed DM encoder 𝑎 𝑡− Δ𝑠 𝑡 Ƹ𝑠 𝑡 DM 𝑠 𝑡−1 𝑎 𝑡− 𝑠 𝑡 periodically update + ҧ𝑠 𝑡

Original: Adopting a similar idea, we incorporate a residual structure in the RNN to help stabilize the training process.
Revised: Adopting a similar idea, we incorporate a residual structure in the RNN to help stabilize the training process.

Original: We have a Dynamics Model ( DM ) trained independently using transition tuples collected in the simulator.
Revised: We have a Dynamics Model ( DM ) trained independently using transition tuples collected in the simulator.

Original: This structure further forces the mapped states to follow the simulation dynamics.
Revised: This structure further forces the mapped states to follow the simulation dynamics.

Original: Its parameters are periodically updated to embed-DM , in order to provide an “average” estimation of the next states ¯ s t (See Fig. 3).
Revised: Its parameters are periodically updated to embed-DM , in order to provide an “average” estimation of the next states ¯ s t (See Fig. 3).

Original: The job of the rest part, instead, is simpliﬁed to just output a correction.
Revised: The job of the rest part, instead, is simpliﬁed to just output a correction.

Original: With the proposed structure model, the mapped states follow the transition dynamics in the simulator better.
Revised: With the proposed structure model, the mapped states follow the transition dynamics in the simulator better.

Original: DM is ﬁrst trained using batches of transition tuples collected in the simulator.
Revised: DM is ﬁrst trained using batches of transition tuples collected in the simulator.

Original: During the training process of the mapping function, the dynamics model is trained online using D ˆ s = { (ˆ s t , a t , s t +1 ∼ p ( s (cid:48) | ˆ s t , a t ) } and is periodically updated to the mapping function.
Revised: During the training process of the mapping function, the dynamics model is trained online using D ˆ s = { (ˆ s t , a t , s t +1 ∼ p ( s (cid:48) | ˆ s t , a t ) } and is periodically updated to the mapping function.

Original: That is, we reset the simulator to the mapped states ˆ s t and then rollout with a one-step oracle simulation transition to get s t +1 .
Revised: That is, we reset the simulator to the mapped states ˆ s t and then rollout with a one-step oracle simulation transition to get s t +1 .

Original: The optimization objective of the dynamics model is an MSE loss in Eq. 8.
Revised: The optimization objective of the dynamics model is an MSE loss in Eq. 8.

Original: Algorithm 2 demonstrates a detailed training and updating procedure of the dynamics model and the entire mapping function.
Revised: Algorithm 2 demonstrates a detailed training and updating procedure of the dynamics model and the entire mapping function.

Original: If the simulation itself is differentiable, e.g. DiffTaichi (Hu et al., 2020), we can plug it into the embed-DM directly without extra learning and updating mentioned above.

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method in 6 MuJoCo environments from OpenAI Gym, namely InvertedPendulum,
Revised: We evaluate our method in 6 MuJoCo environments from OpenAI Gym, namely InvertedPendulum,

Original: InvertedDoublePendulum, HalfCheetah, Hopper, Swimmer and Walker2d.
Revised: InvertedDoublePendulum, HalfCheetah, Hopper, Swimmer and Walker2d.

Original: We deﬁne the rendered images as the target domain, and the original observations as the source domain.
Revised: We deﬁne the rendered images as the target domain, and the original observations as the source domain.

Original: The pre-collected dataset in the target domain contains 600 trajectories and is collected by a sub-optimal policy.
Revised: The pre-collected dataset in the target domain contains 600 trajectories and is collected by a sub-optimal policy.

Original: Please refer to Appendix C for a detailed experiment setting.
Revised: Please refer to Appendix C for a detailed experiment setting.

Original: We modify state-of-the-art methods in same-modal domain adaptation for comparison.
Revised: We modify state-of-the-art methods in same-modal domain adaptation for comparison, namely GAN and CycleGAN.

Revised: GAN uses the same model structure of the encoding network in CODAS, and is trained on pre-collected state and image datasets.

Revised: The state dataset is of the same size as the image dataset.

Revised: As described in Sec.

Revised: D.6, CycleGAN fails in the test tasks.

Original: We also compare with Behavioral Cloning (BC) considering target domain data are collected by a suboptimal policy.
Revised: We also compare with Behavioral Cloning (BC) considering target domain data are collected by a (sub)optimal policy.

Original: BC trains a policy in a supervised manner using ( o rt , a rt ) ∼ D r .
Revised: BC trains a policy in a supervised manner using ( o rt , a rt ) ∼ D r .

Original: To mitigate the partial observability issue mentioned in Sec. 3, we stack every 4 consecutive images as the new input to the BC algorithm.
Revised: To mitigate the partial observability issue mentioned in Sec. 3, we stack every 4 consecutive images as the new input to the BC algorithm.

Original: All methods are trained until convergence.
Revised: All methods are trained until convergence.

Original: Implementation details for all methods can be found in Sec.
Revised: Implementation details for all methods can be found in Sec.

Original: B. Due to some numerical instabilities  inside MuJoCo, we cannot get the oracle transition function in HalfCheetah and InvertedDoublePendulum.
Revised: B. Due to some numerical instabilities 1 inside MuJoCo, we cannot get the oracle transition function in HalfCheetah and InvertedDoublePendulum.

Original: CODAS in these two environments are implemented without embed-DM.
Revised: CODAS in these two environments are implemented without embed-DM.

Original: Experiment results in this section will answer the following three questions: (1) Does CODAS enable polices  transfer from states to images?
Revised: Experiment results in this section will answer the following three questions: 1) Does CODAS enable polices to transfer from states to images?

Original: Does CODAS outperform state-of-the-art methods? (2) Does every component in CODAS contribute to the overall performance?
Revised: Does CODAS outperform state-of-the-art methods? 2) Does every component in CODAS contribute to the overall performance?

Original: (3) Is CODAS robust to small mismatches in environment dynamics between two domains?
Revised: Is CODAS robust to small mismatches in environment dynamics between the source and target domains?

Revised: To focus on the performance of the adaptation process, we use reward ratio as the metric, which is deﬁned as r ratio =

Revised: rr ∗ , where r and r ∗ are the cumulative return of the adapted policy and the optimal policy trained on states respectively.

Revised: The quantitative performance of the optimal policy trained on states and rendered images using PPO is given in Sec.

Revised: D.1.

Original: 4. P ERFORMANCE
Revised: 4. P ERFORMANCE

Original: [Figure]

Original: Training curves of all methods are shown in Fig.
Revised: Training curves of all methods are shown in Fig.

Original: 4, x-axis being training iterations, y-axis being the ratio of discounted return after the adaptation of the original optimal policy.
Revised: 5, x-axis being training iterations, y-axis being the reward ratio.

Original: Each iteration, every method is updated using a batch  20 trajectories.
Revised: Each iteration, every method is updated using a batch of 20 trajectories.

Original: CODAS reaches an average of 70% reward ratio after adaptation, providing an optimistic result on future applications of cross-modal domain adaptation.
Revised: CODAS reaches an average of 70% reward ratio after adaptation, providing an optimistic result on future applications of cross-modal domain adaptation.

Original: BC performs well in simple environments (e.g. HalfCheetah and Inverted Pendulum), but poorly in rest environments, especially in those with  early termination.
Revised: BC performs well in simple environments (HalfCheetah and Inverted Pendulum), but poorly in rest environments, especially in those with an early termination.

Original: Surprisingly, GAN performs  worse than BC in most environments, which suggests mapping images to a predeﬁned interpretable and meaningful state space is more difﬁcult than end-to-end policy learning.
Revised: GAN performs even worse than BC in most environments, which suggests [Figure] [Figure]

Original: A visual illustration of theaccuracy of the mapped states  is shown in Fig.
Revised: A visual illustration of the mapped states accuracy is shown in Fig.

Original: Both reconstructed images and re-rendered images match the original ones well.
Revised: Both reconstructed images and re-rendered images match the original ones well.

Original: It is worth noting that re-rendered images can match the last falling frames, which are sparse in the dataset, well.
Revised: It is worth noting that re-rendered images can match the last falling frames, which are sparse in the dataset, well.

Original: Quantitative results on mapping states errors of GAN and our methods can be found in Sec.
Revised: Quantitative results on mapping states errors of GAN and our methods can be found in Sec.

Original: D.2.
Revised: D.

Original: [Figure]
Revised: [Figure]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original:  Since CODAS contains multiple ideas, we conduct additional experiments to understand the contribution of each component to the overall performance.
Revised: [Figure] Since CODAS contains multiple ideas, we conduct additional experiments to understand the contribution of each component to the overall performance.

Original: We conduct ablation studies to demonstrate (1) if sequential structure matters in the cross-modal domain adaptation in RL and (2) if embed-DM improves the long-horizon inference.
Revised: We conduct ablation studies to demonstrate (1) if sequential structure matters in the cross-modal domain adaptation in RL and (2) if embed-DM improves the long-horizon inference.

Original: Since the embed-DM is based on sequential structure, we only test on two ablated variants, i.e., CODAS w/o embed-DM and w/o both embed-DM and RNN.
Revised: Since the embed-DM is based on sequential structure, we only test on two ablated variants, i.e., CODAS w/o embed-DM and w/o both embed-DM and RNN.

Original: A comparison of all ablated variants is shown in Fig.
Revised: A comparison of all ablated variants is shown in Fig.

Original: Both axes are of the same meanings as above.
Revised: Both axes are of the same meanings as above.

Original: In all the environments, mapping functions with sequential structure yield better deployment performance.
Revised: In all the environments, mapping functions with sequential structure yield better deployment performance.

Original: In four environments where embed-DM is applicable, embed-DM helps further improve the performance.
Revised: In four environments where embed-DM is applicable, embed-DM helps further improve the performance.

Original: As we expect, the major purpose of incorporating embed-DM is to simplify the learning process.
Revised: As we expect, the major purpose of incorporating embed-DM is to simplify the learning process.

Original: If RNN itself can learn a reasonable mapping, the improvement of embed-DM will be marginal (e.g. in Walker2d).
Revised: If RNN itself can learn a reasonable mapping, the improvement of embed-DM will be marginal (Walker2d).

Original: The learning process is also a little unstable in InvertedDoublePendulum, where embed-DM is not available.[Figure]
Revised: The learning process is also a little unstable in InvertedDoublePendulum, where embed-DM is not available.

Original: 4. R OBUSTNESS TO D YNAMICS M ISMATCH
Revised: 4. R OBUSTNESS TO D YNAMICS M ISMATCH

Original: To analyze the CODAS’s robustness to dynamics mismatch [Figure] we modify the dynamics in the source domain.
Revised: To analyze the CODAS’s robustness to dynamics mismatch, we modify the dynamics in the source domain.

Original: Speciﬁcally [Figure] we reconﬁgure the friction coefﬁcient in Hopper.
Revised: Speciﬁcally, we reconﬁgure the friction coefﬁcient in Hopper.

Original: The result can be found in Figure 11.
Revised: The result can be found in Figure 10.

Original: The results in Hopper both with 105% and 120% friction still demonstrate a robust mapping.
Revised: The results in Hopper both with 110% and 120% friction still demonstrate a robust mapping.

Original: Besides [Figure] the mismatch of dynamics even not slow down the speed of learning.
Revised: Besides, the mismatch of dynamics even not slow down the speed of learning.

Original: The result suggests that the CODAS technique has the potential to be deployed in more realistic applications where the dynamics in two domains can not match exactly.
Revised: The result suggests that the CODAS technique has the potential to be deployed in more realistic applications where the dynamics in two domains can not match exactly.

Original: We leave this analysis in future works
Revised: We leave this analysis in future works.

Original: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work [Figure] we propose Cross-Modal Domain Adaptation with Sequential structure (CODAS).
Revised: In this work, we propose Cross-Modal Domain Adaptation with Sequential structure (CODAS).

Original: CODAS deﬁnes a new paradigm of Sim2Real - adapting policies trained on simulated states to realworld with image inputs.
Revised: CODAS enables a new paradigm of Sim2Real - adapting policies trained on simulated states to realworld with image inputs.

Original: We believe this setting is valuable in real-world applications of RL by exempting us from the tedious work of building and running rendering engines.
Revised: We believe this setting is valuable in real-world applications of RL by exempting us from the tedious work of building and running rendering engines.

Original: Previous methods that use GANs fail on this problem since global structural resemblance between two  domains no longer exists, while our method succeeds by fully leveraging the sequential structure and other auxiliary information provided by the  underlying RL problem.
Revised: Previous methods that use GANs fail on this problem since global structural resemblance between two same modal domains no longer exists, while our method succeeds by fully leveraging the sequential structure and other auxiliary information provided by the policy and dynamics underlying RL problem.

Original: We ﬁrst formulate the cross-modal domain adaptation problem as a variational inference problem and decompose it into several feasible optimization objectives.
Revised: We ﬁrst model the cross-modal domain adaptation problem as a variational inference problem and decompose it into several feasible optimization objectives.

Original: To better solve the complex long-horizon sequential mapping problem, we propose a residual network structure.
Revised: To better solve the complex long-horizon sequential mapping problem, we propose a residual network structure.

Original: We validate the proposed method by adapting policies from images to states on various MuJoCo environments.
Revised: We validate the proposed method by adapting policies from images to states on various MuJoCo environments.

Original: Our results further suggest that cross-modal domain adaptation is apromising ﬁeld for low-cost Sim2Real.
Revised: Our results provide an optimistic results of cross-modal domain adaptation as a low-cost Sim2Real approach.

Original: A D ERIVATION OF O PTIMIZATION O BJECTIVES
Revised: A D ERIVATION OF O PTIMIZATION O BJECTIVES

Original: Assume the distribution of trajectories in the simulator p π ( τ s ) is calculated from the real world trajectories τ r .
Revised: Assume the distribution of trajectories in the simulator p π ( τ s ) is calculated from the real world trajectories τ r .

Original: We want to match q πφ ( τ s | τ r ) to the ground-truth distribution p
Revised: We want to match q πφ ( τ s | τ r ) to the ground-truth distribution p

Original: πθ ( τ s | τ r ) , leading to the optimization objective:
Revised: πθ ( τ s | τ r ) , leading to the optimization objective:

Original: [Equation]
Revised: [Equation]

Original: To maximize the objective, we ﬁrst transform it into the Evidence Lower Bound (ELBO):
Revised: To maximize the objective, we ﬁrst transform it into the Evidence Lower Bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: The last term p πθ ( τ r ) is a constant with regards to the parameter θ , and thus can be ignored in the optimization process.
Revised: The last term p πθ ( τ r ) is a constant with regards to the parameter θ , and thus can be ignored in the optimization process.

Original: The minimization can be reduced to min .
Revised: The minimization can be reduced to min .

Original: E τ r
Revised: E τ r

Original: [ E q πφ ( τ s |
Revised: [ E q πφ ( τ s |

Original: τ r ) [log q πφ ( τ s | τ r ) − log p
Revised: τ r ) [log q πφ ( τ s | τ r ) − log p

Original: πθ ( τ r , τ s )]] .
Revised: πθ ( τ r , τ s )]] .

Original: Since the sampled trajectories can be considered as an i.i.d., the ﬁrst term of expectation can be further re-written as
Revised: Since the sampled trajectories can be considered as an i.i.d., the ﬁrst term of expectation can be further re-written as

Original: [Equation]
Revised: [Equation]

Original: Based on the generation process, we can decompose the ﬁrst term based on trajectories to multiplication of terms based on states.
Revised: Based on the generation process, we can decompose the ﬁrst term based on trajectories to multiplication of terms based on states.

Original: [Equation]
Revised: [Equation]

Original: A direct computation of the second term D KL is intractable.
Revised: A direct computation of the second term D KL is intractable.

Original: Following the idea that the optimization process of GAN is equivalent to minimizing a certain distance measure between two distributions (Nowozin et al., 2016), we can use the optimization objective of GAN as the surrogate objective of minimizing the D KL [ q φ ( τ s | τ r ) ||
Revised: Following the idea that the optimization process of GAN is equivalent to minimizing a certain distance measure between two distributions (Nowozin et al., 2016), we can use the optimization objective of GAN as the surrogate objective of minimizing the D KL [ q φ ( τ s | τ r ) ||

Original: p π ( τ s )] by introducing a new discriminator D ω parameterized by ω that maximizes Eq. 13,
Revised: p π ( τ s )] by introducing a new discriminator D ω parameterized by ω that maximizes Eq. 13,

Original: [Equation]
Revised: [Equation]

Original: For implementation simplicity and training stability, ( D ω ( s ) 1 − D ω ( s ) ) can become arbitrarily large, we still choose to optimize the original GAN optimization objective, which is formulated as Eq.14. [Equation]
Revised: For implementation simplicity and training stability, ( D ω ( s ) 1 − D ω ( s ) ) can become arbitrarily large, we still choose to optimize the original GAN optimization objective, which is formulated as Eq.

Original: [Equation]
Revised: [Equation]

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: B gives an illustration of the overall structure of our method.
Revised: 7 gives an illustration of the overall structure of our method.

Revised: π optimal is the optimal policy trained in the simulator.

Revised: π behavioral is a policy in simulator that mimics the behavior of the dataset collection policy (in terms of the cumulative returns).

Revised: In rare cases when π behavioral is not available, it can be replaced by a trainable network here.

Original: Detailed network structure and hyperparameters are listed in Sec.
Revised: Detailed network structure and hyperparameters are listed in Sec.

Original: B.1, all of which are the same for every environment.
Revised: B.1, all of which are the same for every environment.

Original: To help balance the inﬂuence of the residual ∆ s t , we introduce a new hyperparameter λ res , s.t,
Revised: To help balance the inﬂuence of the residual ∆ s t , we introduce a new hyperparameter λ res , s.t,

Original: [Equation]
Revised: [Equation]

Original: B. H YPERPARAMETERS AND N ETWORK S TRUCTURE
Revised: B. H YPERPARAMETERS AND N ETWORK S TRUCTURE

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C E XPERIMENT S ETTING
Revised: C E XPERIMENT S ETTING

Original: We evaluate our method in 6 MuJoCo environments from OpenAI Gym, namely InvertedPendulum,
Revised: We evaluate our method in 6 MuJoCo environments from OpenAI Gym, namely InvertedPendulum,

Original: InvertedDoublePendulum, HalfCheetah, Hopper, FixedSwimmer and Walker2d.
Revised: InvertedDoublePendulum, HalfCheetah, Hopper, FixedSwimmer and Walker2d.

Original: FixedSwimmer is based on modiﬁcations proposed in previous works (Wang et al., 2019) to avoid sub-optimal policies by changing one sensor position.
Revised: FixedSwimmer is based on modiﬁcations proposed in previous works (Wang et al., 2019) to avoid sub-optimal policies by changing one sensor position.

Original: We treat the simulation observations as low dimensional simulation states s and rendered images as real images.
Revised: We treat the simulation observations as low dimensional simulation states s and rendered images as real images.

Original: Images are collected by “track camera” in HalfCheetah, Hopper, FixedSwimmer and Walker2d and “default camera” in InvertedPendulum and InvertedDoublePendulum.
Revised: Images are collected by “track camera” in HalfCheetah, Hopper, FixedSwimmer and Walker2d and “default camera” in InvertedPendulum and InvertedDoublePendulum.

Original: All images are resized to [64 , 64 , 3] without any further pre-processing techniques in all environments.
Revised: All images are resized to [64 , 64 , 3] without any further pre-processing techniques in all environments.

Original: Examples of rendered images are shown in Fig.
Revised: Examples of rendered images are shown in Fig.

Original: Two polices are independently trained until convergence using PPO (Schulman et al., 2017) for every environment.
Revised: Two polices are independently trained until convergence using PPO (Schulman et al., 2017) for every environment.

Original: One of them is regarded  π real a human expert to collect the “real” image dataset.
Revised: One of them is regarded as π behavioral to collect the “real” image dataset.

Original: We use a stochastic policy to mimic the real data collection process.
Revised: π behavioral is a stochastic policy to mimic the real data collection process.

Original: The other is regarded as a pretrained simulator policy π sim .
Revised: The other is regarded as a pre-trained simulator policy π sim .

Original: The real-world dataset contains 600 episodes, each being truncated to a max length of 500.
Revised: The image dataset contains 600 episodes, each being truncated to a max length of 500.

Original: [Figure]
Revised: [Figure]

Original: D M ORE E XPERIMENT R ESULTS D. A BSOLUTE R ESULTS ON R EWARDS
Revised: D M ORE E XPERIMENT R ESULTS D. P ERFORMANCE OF O PTIMAL P OLICY

Revised: Table.

Revised: 3 shows the mean value and standard deviation of the un-discounted cumulative return oftrajectories by the optimal policy trained on states.

Revised: Trajectory length is 1000.

Original: [Table]
Revised: [Table]

Revised: Return

Revised: [Table]

Revised: Swimmer

Revised: [Table]

Revised: InvertedPendulum

Original:  D. Q UANTITATIVE R ESULTS OF S TATE M APPING E RROR
Revised: [Table] HalfCheetah [Table] D. A VERAGE REWARD RATIO OF ALL ENVIRONMENTS

Revised: [Table]

Revised: D. Q UANTITATIVE R ESULTS OF S TATE M APPING E RROR

Original: [Figure]
Revised: [Figure]

Original: D. T RAINING C URVES OF B EHAVIORAL C LONING
Revised: D. T RAINING C URVES OF B EHAVIORAL C LONING

Original: [Figure]
Revised: [Figure]

Original: D. R OBUSTNESS TO D YNAMICS M ISMATCH
Revised: D. R OBUSTNESS TO D YNAMICS M ISMATCH

Original: [Figure]
Revised: [Figure]

Original: r e w a r d r a t i o default 1.2x friction 
Revised: r e w a r d r a t i o default 1.2x friction 1.1x friction

Original: [Figure]

Original: .1x friction

Original: D. T RAINING C URVES OF C YCLE GAN
Revised: D. T RAINING C URVES OF C YCLE GAN

Original: We planned to use CycleGAN, one of the state-of-the-art methods in domain adaptation [Figure] for comparison.
Revised: We planned to use CycleGAN, one of the state-of-the-art methods in domain adaptation, for comparison.

Original: Concretely, we remove the identity loss and use the network structure as ours (See Visual
Revised: Concretely, we remove the identity loss and use the network structure as ours (See Visual

Original: Encoder/Visual Decoder in Sec.
Revised: Encoder/Visual Decoder in Sec.

Original: B) as the generator network structure.
Revised: B) as the generator network structure.

Original: However, the changes in the loss function and network structure may require a complete hyperparameter setting from image-toimage translation.
Revised: However, the changes in the loss function and network structure may require a complete hyperparameter setting from image-toimage translation.

Original: We fail to get a workable CycleGAN in most environments.
Revised: We fail to get a workable CycleGAN in most environments.

Original: The best results so far are shown in Fig.
Revised: The best results so far are shown in Fig.

Original: D.
Revised: [Figure] r [Figure] r

Original: [Figure]
Revised: [Figure]

Original:  i 
Revised: t i o

Original: [Figure]
Revised: [Figure]

Revised: E A N E XAMPLE OF F AILED D ISTRIBUTION M APPING

Revised: Fig.

Revised: 12 shows an example of wrong mapping using GAN.

Revised: Fig.

Revised: 12(a) and Fig.

Revised: 12(b) are two Gaussian

Revised: Mixture Models of 4 modes with a linear transformation between them.

Revised: The translucent circles denote the ground-truth correspondence between every mode in two domains and the dots denote the actual correspondence learned by a GAN.

Revised: Although GAN successfully matches the probability densities, it fails to map the target instances to their corresponding correct source instances.

Revised: Under such a wrong mapping function, a neural network trained on samples in the source domain cannot be transferred to the target domain directly.

Revised: [Figure]