C Y CADA: C YCLE -C ONSISTENT A DVERSARIAL D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain adaptation is critical for success in new, unseen environments.
Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difﬁcult to visualize and sometimes fail to capture pixel-level and low-level domain shifts.
Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at mapping images between domains, even without the use of aligned image pairs.
We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain
Adaptation model.
CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs.
Our model can be applied in a variety of visual recognition and prediction settings.
We show new state-of-the-art results across multiple adaptation tasks, including digit classiﬁcation and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains
I NTRODUCTION
Deep neural networks excel at learning from large amounts of data, but can be poor at generalizing learned knowledge to new datasets or environments.
Even a slight departure from a network’s training domain can cause it to make spurious predictions and signiﬁcantly hurt its performance (Tzeng et al., 2017).
The visual domain shift from non-photorealistic synthetic data to real images presents an even more signiﬁcant challenge.
While we would like to train models on large amounts of synthetic data such as data collected from graphics game engines, such models fail to generalize to real-world imagery.
For example, a state-of-the-art semantic segmentation model trained on synthetic dashcam data fails to segment the road in real images, and its overall per-pixel label accuracy drops from 93% (if trained on real imagery) to 54% (if trained only on synthetic data, see Table 6).
Feature-level unsupervised domain adaptation methods address this problem by aligning the features extracted from the network across the source (e.g. synthetic) and target (e.g. real) domains, without any labeled target samples.
Alignment typically involves minimizing some measure of distance between the source and target feature distributions, such as maximum mean discrepancy (Long & Wang, 2015), correlation distance (Sun & Saenko, 2016), or adversarial discriminator accuracy (Ganin & Lempitsky, 2015; Tzeng et al., 2017).
This class of techniques suffers from two main limitations.
First, aligning marginal distributions does not enforce any semantic consistency, e.g. target features of a car may be mapped to source features of a bicycle.
Second, alignment at higher levels of a deep representation can fail to model aspects of low-level appearance variance which are crucial for the end visual task.
Generative pixel-level domain adaptation models perform similar distribution alignment—not in feature space but rather in raw pixel space—translating source data to the “style” of a target domain.
Recent methods can learn to translate images given only unsupervised data from both domains (Bousmalis et al., 2016; 2017; Liu & Tuzel, 2016b; Shrivastava et al., 2017).
The results are visually compelling, but such image-space models have only been shown to work for small image sizes and limited domain shifts.
Furthermore, they also do not necessarily preserve content: while the translated image may “look” like it came from the right domain, crucial semantic information may be lost.
For example, a model adapting from line-drawings to photos could learn to make a line-drawing of a cat look like a photo of a dog.
[Figure] [Figure]
CycleGAN (Zhu et al., 2017)
[Figure]
(cid:88)
Feature Adapt (Ganin & Lempitsky, 2015; Tzeng et al., 2017) (cid:88) (cid:88)
Pixel Adapt (Taigman et al., 2017a; Bousmalis et al., 2017) (cid:88) (cid:88)
[Figure]
(cid:88) (cid:88) (cid:88) (cid:88)
How can we encourage the model to preserve semantic information in the process of distribution alignment?
In this paper, we explore a simple yet powerful idea: give an additional objective to the model to reconstruct the original data from the adapted version.
Cycle-consistency was recently proposed in a cross-domain image generation GAN model, CycleGAN (Zhu et al., 2017), which showed transformative image-to-image generation results, but was agnostic to any particular task.
We propose Cycle-Consistent Adversarial Domain Adaptation (CyCADA), which adapts representations at both the pixel-level and feature-level while enforcing local and global structural consistency through pixel cycle-consistency and semantic losses.
CyCADA uniﬁes prior feature-level (Ganin &
Lempitsky, 2015; Tzeng et al., 2017) and image-level (Bousmalis et al., 2016; Liu & Tuzel, 2016b) adversarial domain adaptation methods together with cycle-consistent image-to-image translation techniques (Zhu et al., 2017), as illustrated in Table 1.
It is applicable across a range of deep architectures and/or representation levels, and has several advantages over existing unsupervised domain adaptation methods.
We use a reconstruction (cycle-consistency) loss to encourage the cross-domain transformation to preserve local structural information and a semantic loss to enforce semantic consistency.
We apply our CyCADA model to the task of digit recognition across domains and the task of semantic segmentation of urban scenes across domains.
Experiments show that our model achieves state of the art results on digit adaptation, cross-season adaptation in synthetic data, and on the challenging synthetic-to-real scenario.
In the latter case, it improves per-pixel accuracy from 54% to 82%, nearly closing the gap to the target-trained model.
Our experiments conﬁrm that domain adaptation can beneﬁt greatly from cycle-consistent pixel transformations, and that this is especially important for pixel-level semantic segmentation with contemporary FCN architectures.
Further, we show that adaptation at both the pixel and representation level can offer complementary improvements with joint pixel-space and feature adaptation leading to the highest performing model for digit classiﬁcation tasks.
R ELATED W ORK
The problem of visual domain adaptation was introduced along with a pairwise metric transform solution by Saenko et al.
and was further popularized by the broad study of visual dataset bias (Torralba & Efros, 2011).
Early deep adaptive works focused on feature space alignment through minimizing the distance between ﬁrst or second order feature space statistics of the source and target (Tzeng et al., 2014; Long & Wang, 2015).
These latent distribution alignment approaches were further improved through the use of domain adversarial objectives whereby a domain classiﬁer is trained to distinguish between the source and target representations while the domain representation is learned so as to maximize the error of the domain classiﬁer.
The representation is optimized using the standard minimax objective (Ganin & Lempitsky, 2015), the symmetric confusion objective (Tzeng et al., 2015), or the inverted label objective (Tzeng et al., 2017).
Each of these objectives is related to the literature on generative adversarial networks (Goodfellow et al., 2014) and follow-up work for improved training procedures for these networks (Salimans et al., 2016b; Arjovsky et al., 2017).
The feature-space adaptation methods described above focus on modiﬁcations to the discriminative representation space.
In contrast, other recent methods have sought adaptation in the pixel-space using various generative approaches.
One advantage of pixel-space adaptation, as we have shown, is that the result may be more human interpretable, since an image from one domain can now be visualized in a new domain.
CoGANs
(Liu & Tuzel, 2016b) jointly learn a source and target representation through explicit weight sharing of certain layers while each source and target has a unique generative adversarial objective.
Ghifary et al.
(2016) uses an additional reconstruction objective in the target domain to encourage alignment in the unsupervised adaptation setting.
In contrast, another approach is to directly convert the target image into a source style image (or visa versa), largely based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).
Researchers have successfully applied GANs to various applications such as image generation (Denton et al., 2015;
Radford et al., 2015; Zhao et al., 2016), image editing (Zhu et al., 2016) and feature learning (Salimans et al., 2016a; Donahue et al., 2017).
Recent work (Isola et al., 2016; Sangkloy et al., 2016; Karacan et al., 2016) adopt conditional GANs (Mirza & Osindero, 2014) for these image-to-image translation problems (Isola et al., 2016), but they require input-output image pairs for training, which is in general not available in domain adaptation problems.
There also exist lines of work where such training pairs are not given.
Yoo et al.
(2016) learns a source to target encoder-decoder along with a generative adversarial objective on the reconstruction which is is applied for predicting the clothing people are wearing.
The Domain Transfer Network (Taigman et al., 2017b) trains a generator to transform a source image into a target image by enforcing consistency in the embedding space.
Shrivastava et al.
(2017) instead uses an L1 reconstruction loss to force the generated target images to be similar to their original source images.
This works well for limited domain shifts where the domains are similar in pixel-space, but can be too limiting for settings with larger domain shifts.
Bousmalis et al.
(2017) use a content similarity loss to ensure the generated target image is similar to the original source image; however, this requires prior knowledge about which parts of the image stay the same across domains (e.g. foreground).
Our method does not require pre-deﬁning what content is shared between domains and instead simply translates images back to their original domains while ensuring that they remain identical to their original versions.
BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2016) take an approach of simultaneously learning the transformations between the pixel and the latent space.
More recently, Cycle-consistent
Adversarial Networks (CycleGAN) (Zhu et al., 2017) produced compelling image translation results such as generating photorealistic images from impressionism paintings or transforming horses into zebras at high resolution using the cycle-consistency loss.
This loss was simultaneously proposed by Yi et al.
(2017) and Kim et al.
(2017) to great effect as well.
Our motivation comes from such ﬁndings about the effectiveness of the cycle-consistency loss.
Few works have explicitly studied visual domain adaptation for the semantic segmentation task.
Adaptation across weather conditions in simple road scenes was ﬁrst studied by Levinkov & Fritz (2013).
More recently, a convolutional domain adversarial based approached was proposed for more general drive cam scenes and for adaptation from simulated to real environments (Hoffman et al., 2016).
Ros et al.
(2016b) learns a multi-source model through concatenating all available labeled data and learning a single large model and then transfers to a sparsely labeled target domain through distillation (Hinton et al., 2015).
Chen et al.
(2017) use an adversarial objective to align both global and class-speciﬁc statistics, while mining additional temporal data from street view datasets to learn a static object prior.
Zhang et al.
(2017) instead perform segmentation adaptation by aligning label distributions both globally and across superpixels in an image.
[Figure]
C YCLE -C ONSISTENT A DVERSARIAL D OMAIN A DAPTION
We consider the problem of unsupervised adaptation, where we are provided source data X S , source labels Y S , and target data X T , but no target labels.
The goal is to learn a model f that can correctly predict the label for the target data X T .
We can begin by simply learning a source model f S that can perform the task on the source data.
For
K -way classiﬁcation with a cross-entropy loss, this corresponds to
[Equation]
where σ denotes the softmax function.
However, while the learned model f S will perform well on the source data, typically domain shift between the source and target domain leads to reduced performance when evaluating on target data.
To mitigate the effects of domain shift, we follow previous adversarial adaptation approaches and learn to map samples across domains such that an adversarial discriminator is unable to distinguish the domains.
By mapping samples into a common space, we enable our model to learn on source data while still generalizing to target data.
To this end, we introduce a mapping from source to target G S → T and train it to produce target samples that fool an adversarial discriminator D T .
Conversely, the adversarial discriminator attempts to classify the real target data from the source target data.
This corresponds to the loss function
[Equation]
This objective ensures that G S → T , given source samples, produces convincing target samples.
In turn, this ability to directly map samples between domains allows us to learn a target model f T by minimizing L task ( f T , G S
→ T ( X S ) , Y S ) .
However, while previous approaches that optimized similar objectives have shown effective results, in practice they can often be unstable and prone to failure.
Although the GAN loss in Equationensures that G S → T ( x s ) for some x s will resemble data drawn from X T , there is no way to guarantee that G S → T ( x s ) preserves the structure or content of the original sample x s .
In order to encourage the source content to be preserved during the conversion process, we impose a cycle-consistency constraint on our adaptation method (Zhu et al., 2017; Yi et al., 2017; Kim et al., 2017).
To this end, we introduce another mapping from target to source G T → S and train it according to the same GAN loss L GAN ( G T → S , D S , X S , X T ) .
We then require that mapping a source sample from source to target and back to the source reproduces the original sample, thereby enforcing cycle-consistency.
In other words, we want G T → S ( G S → T ( x s ))
≈ x s and G S → T ( G T → S ( x t ))
≈ x t .
This is done by imposing an L1 penalty on the reconstruction error, which is referred to as the cycle-consistency loss :
[Equation]
Additionally, as we have access to source labeled data, we explicitly encourage high semantic consistency before and after image translation.
We use the trained source task model f S as a noisy labeler by which we encourage an image to be classiﬁed in the same way after translation as it was before translation according to this classiﬁer.
Let us deﬁne the predicted label from a ﬁxed classiﬁer, f , for a given input X as p ( f, X ) = arg max( f ( X )) .
Then we can deﬁne the semantic consistency before and after image translation as follows:
[Equation]
This can be viewed as analogously to content losses in style transfer (Gatys et al., 2016), where the shared content to preserve is dictated by the source task model f S .
Taken together, these loss functions form our complete objective:
[Equation]
This ultimately corresponds to solving for a target model f T according to the optimization problem
[Equation]
We have presented this method as the addition of cycle-consistency and semantic constraints to standard adversarial adaptation approaches.
In this setting, typically adaptation is done in feature space, and X S and X T are feature representations such as intermediate layers of convolutional networks.
However, we can also consider the case where X S and X T are simply input images, such as in (Bousmalis et al., 2016; Liu & Tuzel, 2016b).
In this case, adaptation is being performed in image space, and together Equations 2 and 3 reproduce the CycleGAN objective.
Thus, as mentioned above, we can equivalently view CyCADA as CycleGAN augmented with an additional task loss, for the end goal of ultimately performing a discriminative task.
In this work, we apply CyCADA to the semantic segmentation task and implement G as a pixel-to-pixel convnet, f as an Fully-Convolutional
Net (FCN) and D as a convnet with binary outputs.
E XPERIMENTS
We evaluate CyCADA on several unsupervised adaptation scenarios.
We ﬁrst focus on adaptation for digit classiﬁcation using the MNIST (LeCun et al., 1998), USPS and Street View House Numbers (SVHN) (Netzer et al., 2011) datasets.
After which we present results for the task of semantic image segmentation, using the SYNTHIA (Ros et al., 2016a), GTA (Richter et al., 2016) and
CityScapes (Cordts et al., 2016) datasets.
4. D IGIT A DAPTATION
We evaluate our method across the adaptation shifts of USPS to MNIST, MNIST to USPS, and
SVHN to MNIST.
We report classiﬁcation accuracy for each shift in Table 2.
We ﬁnd that in the case of the small domain shifts between USPS and MNIST, the pixel space adaptation by which we train a classiﬁer using images translated using CycleGAN (Zhu et al., 2017), performs very well, outperforming or comparable to prior adaptation approaches.
Feature level adaptation offers a small beneﬁt in this case.
However, for the more difﬁcult shift of SVHN to MNIST, we ﬁnd that feature level adaptation outperforms the pixel level adaptation, and both may be combined to produce an overall model which outperforms all competing methods.
Additionally, the standard unsupervised
CycleGAN approach diverged when training SVHN to MNIST often suffering from random label ﬂipping.
Note that the recent pixel-da method by Bousmalis et al.
(2017) presents results for only the MNIST to USPS shift and reports 95.9% accuracy, while our method achieves 94.8% accuracy.
However, the pixel-da approach cross validates with some labeled data which is not an equivalent evaluation setting.
Figure 3 demonstrates an example where cycle constraints alone fail to have the
[Table]
desired behavior for our end task.
An SVHN image is mapped to a convincing MNIST type image and back to a SVHN image with correct semantics.
However,
MNIST-like image has mismatched semantics.
Our modiﬁed version, which uses the source labels to train a weak classiﬁcation model which can be used to enforce semantic consistency before and after translation, resolves this issue and produces strong performance.
4. S EMANTIC S EGMENTATION A DAPTATION
The task is to assign a semantic label to each pixel in the input image, e.g. road , building , etc.
We limit our evaluation to the unsupervised adaptation setting, where labels are only available in the source domain, but we are evaluated solely on our performance in the target domain.
For each experiment, we use three metrics to evaluate performance.
Let n ij be the number of pixels of class i predicted as class j , let t i = (cid:80) j n ij be the total number of pixels of class i , and let N be the number of classes.
Our three evaluation metrics are, mean intersection-over-union (mIoU), frequency weighted intersection-over-union (fwIoU), and pixel accuracy, which are deﬁned as follows:
[Equation]
Cycle-consistent adversarial adaptation is general and can be applied at any layer of a network.
Since optimizing the full CyCADA objective in Equation 5 end-to-end is memory-intensive in practice, we train our model in stages.
First, we perform image-space adaptation and map our source data into the target domain.
Next, using the adapted source data with the original source labels, we learn a task model that is suited to operating on target data.
Finally, we perform another round of adaptation between the adapted source data and the target data in feature-space, using one of the intermediate layers of the task model.
Additionally, we do not use the semantic loss for the segmentation experiments as it would require loading generators, discriminators, and an additional semantic segmentater into memory all at once for two images.
We did not have the required memory for this at the time of submission, but leave it to future work to deploy model parallelism or experiment with larger GPU memory.
For our ﬁrst evaluation, we consider the SYNTHIA dataset (Ros et al., 2016a), which contains synthetic renderings of urban scenes.
We use the SYNTHIA video sequences, which are rendered across a variety of environments, weather conditions, and lighting conditions.
This provides a synthetic testbed for evaluating adaptation techniques.
For comparison with previous work, in this work we focus on adaptation between seasons.
We use only the front-facing views in the sequences
[Table]
a
[Table]
s i d e
[Table]
a l k f e
[Table]
v e g e t a ti
[Table]
po l e ca r t r a f ﬁ c
s i
gn p
e d e s t
r
i a n b i c y c
l e l a n e m a r
k
i ng t r a f ﬁ c li gh t
m I o U f w I
o U
P i x e l a
[Table] [Table]
[Figure]
so as to mimic dashcam imagery, and adapt from fall to winter.
The subset of the dataset we use contains 13 classes and consists of 10,852 fall images and 7,654 winter images.
To further demonstrate our method’s applicability to real-world adaptation scenarios, we also evaluate our model in a challenging synthetic-to-real adaptation setting.
For our synthetic source domain, we use the GTA5 dataset (Richter et al., 2016) extracted from the game Grand Theft Auto V, which contains 24966 images.
We consider adaptation from GTA5 to the real-world Cityscapes dataset (Cordts et al., 2016), from which we used 19998 images without annotation for training and 500 images for validation.
Both of these datasets are evaluated on the same set of 19 classes, allowing for straightforward adaptation between the two domains.
4.2. I MPLEMENTATION DETAILS
Image-space adaptation also affords us the ability to visually inspect the results of the adaptation method.
This is a distinct advantage over opaque feature-space adaptation methods, especially in truly unsupervised settings—without labels, there is no way to empirically evaluate the adapted model, and thus no way to verify that adaptation is improving task performance.
Visually conﬁrming that the conversions between source and target images are reasonable, while not a guarantee of improved task performance, can serve as a sanity check to ensure that adaptation is not completely diverging.
This process is diagrammed in Figure 2.
For cycle-consistent image translation, we followed the network architecture and hyperparameters of
CycleGAN(Zhu et al., 2017).
All images were resized to have width of 1024 pixels while keeping the aspect ratio, and the training was performed with randomly cropped patches of size 400 by 400.
Also, due to large size of the dataset, we trained only 20 epochs.
Then the segmentation model was trained separately using the adapted source images and the ground truth labels of the source data.
We used VGG16-FCN8 architecture (Long et al., 2015), and trained for 60k iterations using SGD with learning rate 0.001.
4.2. C ROSS - SEASON ADAPTATION
We start by exploring the abilities of pixel space adaptation alone for the setting of adapting across seasons in synthetic data.
For this we use the SYNTHIA dataset and adapt from fall to winter weather conditions.
Typically in unsupervised adaptation settings it is difﬁcult to interpret what causes the
[Table]
[Figure]
performance improvement after adaptation.
Therefore, we use this setting as an example where we may directly visualize the shift from fall to winter and inspect the intermediate pixel level adaptation result from our algorithm.
In Figure 4 we show the result of pixel only adaptation as we generate a winter domain image (b) from a fall domain image (a), and visa versa (c-d).
We may clearly see the changes of adding or removing snow.
This visually interpretable result matches our expectation of the true shift between these domains and indeed results in favorable ﬁnal semantic segmentation performance from fall to winter as shown in Table 3.
We ﬁnd that CyCADA achieves state-of-the-art performance on this task with image space adaptation alone, however does not recover full supervised learning performance (train on target).
Some example errors includes adding snow to the sidewalks, but not to the road, while in the true winter domain snow appears in both locations.
However, even this mistake is interesting as it implies that the model is learning to distinguish road from sidewalk during pixel adaptation, despite the lack of pixel annotations.
Cycle-consistent adversarial adaptation achieves state-of-the-art adaptation performance.
We see that under the fwIoU and pixel accuracy metrics, CyCADA approaches oracle performance, falling short by only a few points, despite being entirely unsupervised.
This indicates that CyCADA is extremely effective at correcting the most common classes in the dataset.
This conclusion is supported by inspection of the individual classes in Table 3, where we see the largest improvement on common classes such as road and sidewalk .
4.2. S YNTHETIC TO REAL ADAPTATION
To evaluate our method’s applicability to real-world adaptation settings, we investigate adaptation from synthetic to real-world imagery.
The results of this evaluation are presented in Table 4 with qualitative results shown in Figure 5.
Once again, CyCADA achieves state-of-the-art results, recovering approximately 40% of the performance lost to domain shift.
CyCADA also improves or maintains performance on all 19 classes.
Examination of fwIoU and pixel accuracy as well as individual
[Figure]
class IoUs reveals that our method performs well on most of the common classes.
Although some classes such as train and bicycle see little or no improvement, we note that those classes are poorly represented in the GTA5 data, making recognition very difﬁcult.
We visualize the results of image-space adaptation between GTA5 and Cityscapes in Figure 6.
The most obvious difference between the original images and the adapted images is the saturation levels— the GTA5 imagery is much more vivid than the Cityscapes imagery, so adaptation adjusts the colors to compensate.
We also observe texture changes, which are perhaps most apparent in the road: in-game, the roads appear rough with many blemishes, but Cityscapes roads tend to be fairly uniform in appearance, so in converting from GTA5 to Cityscapes, our model removes most of the texture.
Somewhat amusingly, our model has a tendency to add a hood ornament to the bottom of the image, which, while likely irrelevant to the segmentation task, serves as a further indication that image-space adaptation is producing reasonable results.
C ONCLUSION
We presented a cycle-consistent adversarial domain adaptation method that uniﬁes cycle-consistent adversarial models with adversarial adaptation methods.
CyCADA is able to adapt even in the absence of target labels and is broadly applicable at both the pixel-level and in feature space.
An image-space adaptation instantiation of CyCADA also provides additional interpretability and serves as a useful way to verify successful adaptation.
Finally, we experimentally validated our model on a variety of adaptation tasks: state-of-the-art results in multiple evaluation settings indicate its effectiveness, even on challenging synthetic-to-real tasks.