DaDA: Distortion-aware Domain Adaptation for Unsupervised Semantic Segmentation
Abstract
Distributional shifts in photometry and texture have been extensively studied for unsupervised domain adaptation, but their counterparts in optical distortion have been largely neglected.
In this work, we tackle the task of unsupervised domain adaptation for semantic image segmentation where unknown optical distortion exists between source and target images.
To this end, we propose a d istortion- a ware d omain a daptation (DaDA) framework that boosts the unsupervised segmentation performance.
We ﬁrst present a r elative d istortion l earning (RDL) approach that is capable of modeling domain shifts in ﬁne-grained geometric deformation based on diffeomorphic transformation.
Then, we demonstrate that applying additional global afﬁne transformations to the diffeomorphically transformed source images can further improve the segmentation adaptation.
Besides, we ﬁnd that our distortion-aware adaptation method helps to enhance self-supervised learning by providing higher-quality initial models and pseudo labels.
To evaluate, we propose new distortion adaptation benchmarks, where rectilinear source images and ﬁsheye target images are used for unsupervised domain adaptation.
Extensive experimental results highlight the effectiveness of our approach over state-of-the-art methods under unknown relative distortion across domains.
Datasets and more information are available at https://sait-fdd.github.io/ .
Introduction
Recent years have witnessed dramatic improvements in semantic segmentation performance based on a massive amount of pixel-level annotations.
However, applying a segmentation model learned from the training dataset (source domain) directly to unseen test scenarios (target domain) often leads to signiﬁcant performance degradation, which is caused by the divergence of data distribution between source and target domains.
In addition, manually annotating target domain images with pixel-wise labels is extremely expensive and taxing.
To address such issues, many studies have proposed unsupervised domain adaptation methods to transfer the knowledge learned from the labeled source domain to the unlabeled target domain ( e.g., [33, 34, 27, 26, 40]).
Existing works on unsupervised segmentation adaptation methods have predominantly investigated domain shift problems mainly due to photometric and textural discrepancies between rectilinear image domains ( e.g., adapting synthetically generated images [29, 30] to real-world imagery [8]; or adapting images from/to different cities [7]).
In contrast, domain shifts in geometric and optical distortion have not been well explored, despite commonly appearing in many practical applications.
For example, wide-angle cameras ( e.g., ﬁsheye cameras) have been extensively used for complex
[Figure]
vision systems like autonomous vehicles [38] and surveillance [4] to obtain more information about the surrounding environment.
Fisheye images ( e.g., target image I T in Fig.1) have quite different geometric deformations ( e.g., radial distortion) compared to regular rectilinear images ( e.g., source image I S in Fig.1).
Such distortion variants pose even more challenging domain adaptation tasks.
For example, a state-of-the-art semantic segmentation model [6] only trained on rectilinear images fails to correctly predict pixels of vehicles and roads under optical distortion and its overall prediction accuracy drops from 67.02 % (trained on ﬁsheye images) to 32.39 % (trained only on rectilinear images, see Tab.1).
To alleviate this, we may rectify distortion at test time.
However, such a workaround inevitably leads to reduced ﬁeld-of-view ( e.g., over 30 % loss for ﬁsheye images [38]), resampling distortion artifacts at the image periphery, calibration errors in practice, and additional computing resources for the rectiﬁcation step at test time [19], which is against the original purpose of using wide-angle cameras.
This urges us to use native ﬁsheye images without needing rectiﬁcation.
Another motivation for our work is the scarcity and difﬁculty of constructing annotations for distorted ﬁsheye images ( e.g., Woodscape [38]), while we already have larger amounts of annotations for rectilinear images ( e.g., Cityscapes [8], GTAV [29]).
Remarkably, we have relatively fewer public datasets providing large-scale and ﬁnely annotated ﬁsheye images.
Woodscape [38] is currently the only real-world public dataset with segmentation annotations.
Such a lack of annotations for distorted images necessitates introducing optical distortion into unsupervised domain adaptation.
With these insights, we ﬁrst formulate an important but challenging unsupervised domain adaptation task for semantic segmentation where unknown optical distortion exists between the source and target domains.
To this end, we propose a novel distortion-aware domain adaptation (DaDA) framework, which provides a new perspective to minimizing domain gaps in geometric and optical distortion.
We ﬁrst present relative distortion learning (RDL), which is capable of modeling relative deformation between the source and target domain.
In particular, we build a deformation ﬁeld generator to transform the source image to a new image sharing similar distortion features of the target image.
To enable such challenging unsupervised and unpaired distortion learning, we exploit the properties of diffeomorphism , that is differentiable and has a differentiable inverse.
We directly integrate such properties into our distortion-aware losses to enforce the semantic quality of relative deformation ﬁelds at the image- and the prediction level.
We also observe that applying additional global afﬁne transformations ( e.g., rotation, shearing) to the diffeomorphically transformed source images can further improve the segmentation adaptation performance in most cases.
Owing to ﬁne-grained diffeomorphic and global afﬁne transformations, our framework provides distortionaware segmentation adaptation models and reliable pseudo labels, and thus ultimately improves the performance of self-supervised learning methods.
To validate, we propose new domain adaptation benchmarks, where a segmentation model trained on rectilinear images (Cityscapes [8] or GTAV [29]) is transferred to ﬁsheye images (Woodscape [38] or our in-house ﬁsheye driving scene dataset (FDD)).
With mean Intersection-over-Union (mIoU) as the evaluation metric, our framework achieves signiﬁcantly improved prediction performance compared to existing segmentation adaptation methods.
In essence, our key contributions can be summarized as: (1) a distortion-aware domain adaptation framework for boosting unsupervised semantic segmentation performance in the presence of unknown relative distortion; (2) distortion-aware losses that effectively enforce the unpaired and unsupervised relative distortion learning to transfer distortion style from target to source domain; (3) new unsupervised domain adaptation benchmarks posing challenging tasks, where source and target images have additional domain gaps in optical distortion.
Related Work 2.1 Domain Adaptive Semantic Segmentation
Recent advances in image-to-image translation and style transfer [42] have promoted the translation of source-domain images to mimic texture and appearance of the target-domain images.
CyCADA [24] applied CycleGAN [42] to preserve semantic consistency before and after image-level adaptation.
BDL [22] proposed a bidirectional learning method where image-to-image translation and segmentation adaptation is learned alternatively.
Similar methods have been proposed to adapt source images [35, 17] or to reconstruct the source image from the output-level representation [36].
Deviated from these adversarial approaches, FDA [37] proposed a method to align the low-level frequencies of source images and their counterparts in target images.
Ma et al.
[25] applied a photometric alignment method to minimize domain shifts in image- and category-level feature distributions.
More recently, self-supervised learning (SSL) approach has emerged in domain adaption tasks to further improve the segmentation adaptation performance.
SSL approaches try to rectify noisy pseudo labels with respect to uncertainties or conﬁdence scores [22, 17, 28, 27, 26, 13, 41, 40].
These methods inherently require good initial models, that are commonly trained on source or adapted on target beforehand.
To obviate the need of multi-stage and adversarial training, Araslanov and Roth [2] introduced a photometric augmentation method to SSL.
However, none of the prior works directly considers distortion-oriented domain shifts in the unsupervised segmentation adaptation tasks.
Moreover, it remains unknown how well existing adaptation methods address unknown relative distortion between domains.
As a pioneering work, we ﬁrst aim to evaluate prior arts on the adaption task in the presence of unknown optical distortion across domains.
For this, we noted that many of current state-of-the-art methods rely on SSL ( e.g., [17, 27, 26, 41, 40]) and have adopted AdaptSeg [33] or AdvEnt [34] as for their based segmentation adaptation.
Thus, we take these adversarial adaptation approaches as our baseline methods and further evaluate SSL approaches [26, 27, 40] in the presence of optical distortion shifts across domains.
To evaluate, we newly formulate segmentation adaptation benchmarks, i.e., transferring from real or synthetic rectilinear images to real ﬁsheye images.
2.2 Diffeomorphic Deformation Networks
Spatial transformation networks (STN [15]) have been used in various contexts since it is capable of learning differentiable deformation ﬁeld.
STN primarily allows simple linear transformations ( e.g., afﬁne, translations, scaling, and rotations) and can be extended to more ﬂexible mappings such as thin plate spline transformations [23].
However, optical distortion involves complex nonlinear transformation and STNs are limited to support such transformations.
Instead Detlefsen et al.
[12] proposed to exploit diffeomorphisms in spatial transformation, which can model complex nonlinear deformation and address optimization divergence issues in learning spatial transformation.
A diffeomorphism is a globally one-to-one continuous and smooth mapping, which has a differentiable inverse.
Diffeomorphic transformations have been largely used for image registration [16, 10, 32] and shape analysis [39].
However, traditional diffeomorphic deformation methods demand high computational costs and difﬁcult to implement; and thus make it difﬁcult to be incorporated into deep neural networks.
To address such issue, Dalca et al.
[11] proposed a probabilistic generative model to generate diffeomorphic deformation ﬁeld for medical image registration task.
However, their work depends on a paired set of 3D brain images depicting the same contexts.
Inspired by [11], we propose a new relative distortion generator, which takes a set of unpaired source- and target-domain image, to transform the source image to a new image sharing similar distortion style of the target image.
2.3 Radial Distortion Rectiﬁcation
Traditional distortion rectiﬁcation approaches require images captured with speciﬁc camera calibration parameters [20] and thus are not ﬂexible.
The other work exploits the principle that a straight line should be projected into a straight line in a calibrated image [1], but such method mainly depends on the accuracy of line detection.
More recently, deep neural networks have been used for robust and efﬁcient rectiﬁcation performance [5, 21].
However, these works primarily aim to rectify distorted images and do not support distortion style transfer for unpaired and unsupervised domain adaptation tasks.
In contrast, we propose to transfer distortion style between unpaired source and target images.
Method
Let S be the source-domain data with segmentation labels Y S and T be the target-domain data with no labels.
We aim to train a scene segmentation network M performing satisfactory pixel-wise prediction on T by minimizing domain gaps between S and T .
Here we formulate more challenging adaptation tasks where distributional shifts include not only visual domain gaps ( e.g., texture, lighting, contrast) but also geometric and optical distortions ( e.g., radial distortion).
Such domain gaps make
M difﬁcult to learn transferable knowledge in both visual and geometric domains.
3.1 Relative Distortion Learning
To minimize the discrepancy between source and target domain, prior works proposed pixel-level image translation methods based on cycle-consistency [14, 17, 35, 42] or photometric alignment [2, 25].
However, these methods do not consider domain gaps in geometric distortion and thus fail to transfer distortion style from I T to I S (see Fig.4).
To address this, we propose a relative distortion learning (RDL) method, which predicts a relative deformation ﬁeld to transfer the distortion style between domains.
Given a source-domain image I S and a target-domain image I T , we aim to transform I S to a new image I S
→ T based on a relative deformation ﬁeld Φ S → T , where I S → T shares a similar distortion style of I T .
We achieve this transformation through a grid-based sampling operation I S ◦ Φ S
→ T = I S
→ T .
Ultimately, the transformed source image I S
→ T aim to mitigate the domain shift in optical distortion at a ﬁne-grained level.
However, it is not trivial to predict relative deformation ﬁelds since input images are not paired ( i.e., dissimilar image contents); and the relative distortion involves nonlinear geometric deformation, which is hard to be parameterized without knowing optical features ( e.g., lens distortion model, focal length).
To learn such a challenging geometric relationship, we exploit diffeomorphic transformation in the unpaired and unsupervised distortion translation task.
Diffeomorphic Transformation.
Let u ∈ R 2 × w ×
h be a ﬂow ﬁeld which is constant over time.
Then, we describe a differential equation of evolution of deformation φ ( t ) by
[Equation]
where t is time and φ (0) the identity transformation.
By integrating u ( φ ( t ) ) over t ∈ [0 , 1] , we get a diffeomorphic deformation ﬁeld Φ :=
φ (1) , which maps the coordinates from one image to another image.
For fast and differentiable integration, we use an exponentiated integration technique, so-called scaling-and-squaring [3].
This integration method starts from φ (1 / 2 T ) =
φ (0) + u/ 2 T and iteratively computes deformation ﬁelds of next time steps via φ (1 / 2 t − 1 ) =
φ (1 / 2 t ) ◦ φ (1 / 2 t ) over
T times, where ◦ indicates a grid-based resampling operation.
Then, we obtain an approximate deformation ﬁeld Φ .
The inverse deformation ﬁeld Φ −  can be achieved by integrating the negative ﬁeld φ (1 / 2 T )
=
φ (0) − u/ 2 T via the same integration method.
To generate a ﬁeld of Φ S → T , which deﬁnes relative distortion between I S and I T , we propose a deformation ﬁeld generator G , which takes both I S and I T ; and their ﬁrst-order gradient ( i.e., Sobel ﬁlter) ∇ I S and ∇ I T as input.
Here we use the image gradients to provide rich geometric features inspired by the distortion rectiﬁcation method [1].
Then, G generates deformation ﬁeld Φ S → T and its inverse ﬁeld Φ T → S .
For example, in Fig.2, G generates a ﬂow ﬁeld u to construct a Φ S → T (red-colored grid in (e)) from a pair of I T (a) and
I S (b).
Finally, a new transformed image I S
→ T (e) is generated via I S ◦ Φ S → T .
Similarly, negative integration of the ﬂow ﬁeld u yields an inverse deformation ﬁeld Φ T → S , which generate a reconstructed image (d) via I (cid:48)
S =
I S
→ T ◦ Φ T → S .
To supervise the generator G for learning the relative distortion in unpaired and unsupervised settings, we exploit the properties of diffeomorphic transformations: topology preserving, invertible , and differentiable .
Such diffeomorphic constraints are directly built into relative distortion learning to enforce desirable deformation ﬁeld outputs.
In particular, we propose distortion-aware losses: distortion reconstruction loss and semantic distortion-consistent loss.
These losses evaluate the cycleconsistency of the relative deformation ﬁeld at the image- and the prediction-level, while enforcing the semantic quality of the relative deformation ﬁeld.
Distortion reconstruction loss ( L recon ) ensures that G generates convincing ﬂow ﬁelds to reconstruct the source image I S from the transformed image I S
→ T by the inverse deformation ﬁeld Φ T → S , and vice versa:
[Equation]
1 , where I (cid:48) S =
[Equation]
Φ
[Equation]
S ,
[Equation]
Semantic distortion-consistent loss ( L sem ) enforces semantic consistency between the source image
I S and its transformed pair I S
→ T at the pixel-wise prediction level.
The same constraint can be applied to between I T and
I T
→ S .
That is, an ideal deformation ﬁeld generator G should be able to generate Φ S → T and Φ T → S , that reconstruct structural information from the pixel-wise prediction outputs of the distorted input images.
With these semantic constraints, the relative distortion learning can be further improved by the segmentation model M ; and also the segmentation adaptation can beneﬁt from the improved quality of relative deformation ﬁelds.
The loss function can be written as:
[Equation]
To minimize domain gaps in optical distortion between the transformed source images I S
→ T and the target-domain images I T , we introduce a distortion-aware discriminator D G , which aims to discriminate distortion style between I T and I S
→ T .
Similar to G , we again use the ﬁrst-order image gradient, ∇ I T and ∇ I
S
→ T , as input to D G along with the images.
The loss function is deﬁned as:
[Equation]
We also calculate the adversarial loss using D G :
[Equation]
where the deformation generator G tries to produce the relative deformation ﬁeld to fool the discriminator D G in distinguishing distortion style.
Therefore, the total loss function for the relative distortion learning is deﬁned as:
[Equation]
where β 1 , β 2 , and β 3 are constants controlling the effect of corresponding losses.
3.2 Distortion-aware Adversarial Adaptation
Relative distortion learning aims to adapt optical distortion shifts at both pixel- and feature-level.
Based on the adapted representations, we further apply adversarial segmentation adaptation.
As our baseline adaptation methods, we take AdaptSeg [33] and AdvEnt [34], that have been extensively used in many prior work [26–28, 40] as for their based segmentation adaptation.
Typically, the segmentation loss uses the cross-entropy for the transformed source image I S
→ T and its corresponding label Y S
→ T :
[Equation]
where Y S → T is obtained by transforming ground-truth annotations from the source domain by
Y S ◦ Φ S → T .
Entropy minimization loss is further introduced by AdvEnt [34]:
[Equation]
which tries to directly minimize pixel-wise entropies to enhance prediction certainty in the target domain.
As a common practice, a domain discriminator D M is used to minimize the difference between the transformed source and target prediction probabilities:
[Equation]
The adversarial loss function for segmentation adaptation can be written as:
[Equation]
where the segmentation model M is trained to fool the discriminator D M .
Equipped with the relative distortion learning and the adversarial adaptation, we train the adaptive segmentation network with the following total loss:
[Equation]
where γ is 0 for AdaptSeg [33] and 1 for AdvEnt [34].
Experimental Results
We present extensive experimental results to validate our distortion-aware domain adaptation (DaDA) framework for semantic segmentation in the presence of both visual and geometric domain shifts.
For this, we formulate new domain adaptive segmentation benchmarks, i.e., transferring from real-world or synthetic rectilinear images to real ﬁsheye images.
4.1 Datasets
In the experiments, we ﬁrst show evaluations of the model trained on real-world rectilinear images from the Cityscapes [8] and test the adapted model on real ﬁsheye images from Woodscape [38] or our in-house ﬁsheye driving dataset (FDD).
Then, we introduce a more challenging adaptation task, where the model trained on synthetic dataset GTAV [29] is transferred to real ﬁsheye images (Woodscape or FDD) without annotations.
The Cityscapes dataset contains 2 , 975 training images of high-quality driving scene with the resolution of 2048 × 1024 .
The GTAV dataset contains 24 ,synthesized images with the resolution of 1914 × 1052 .
The Woodscape dataset consists of 8 ,ﬁsheye images with the resolution of 1280 × 966 , where the images are captured by ﬁsheye cameras ( 190 ◦ F.O.V) looking at four different directions of the vehicle.
We use front and rear camera scenes containing 4 , 023 images in our experiments.
The images are randomly split into a training set with 3 , 023 images and a validation set with 1 , 000 images.
We report the results on 17 classes aligning mismatched classes between Cityscapes and GTAV; and Woodscape.
Our in-house ﬁsheye driving dataset (FDD) includes 3 , 897 of fully annotated images with the resolution of 1920 × 1080 captured by ﬁsheye cameras ( 200 ◦ F.O.V) at front- and rear-side of the vehicle.
We randomly pulled 974 of validation images and remaining 2 , 923 images are used for the training.
For FDD, we use 12 classes, where incompatible classes in Cityscapes and GTAV are merged or excluded similar to Woodscape (see Appendix B.4 for detailed class information of Woodscape and FDD).
4.2 Experimental Details Diffeomorphic and Afﬁne Transformation
Fig.2 depicts an example of diffeomorphic and afﬁne transformations applied to a source- and a target-domain image.
First, both original target and source images are randomly cropped and resized to (a) and (b) followed by randomized horizontal ﬂipping and photometric jittering.
For the ﬁne-grained diffeomorphic transformation, both target ( I T in (a)) and source image ( I S in (b)) are used to generate a transformed image
I S
→ T in (e) via the relative deformation ﬁeld generator G .
Additional global afﬁne transformation is applied to I S
→ T to generate an image in (f), which includes both ﬁne-grained diffeomorphic and global afﬁne deformations.
For the afﬁne transformation, we adopted RandAugment (RA) [9] excluding preceded photometric jittering.
RA is one of the immediately probable
[Table]
and applicable augmentation methods including a series of afﬁne transformations ( i.e., rotation, shear-x, shear-y, trans-x, and trans-y with 0.5 probability of application).
For example, a rotation transformation is applied to the source image I S in (c).
In our experiments, we tested transformed images generated by either one of afﬁne-only (c), diffeomorphic-only (e), or both transformations (f).
Implementation Details.
We trained all networks with the Adam [18] solver with a batch size of 4.
The learning rate is 0 .
2 × 10 − 5 for M and D M and 0 .
1 × 10 − 6 for G and D G .
We set the weight factors of losses in Eq.(6) as: β 1 = 100 . 0 , β 2 = 10 . 0 , β 3 = 10 . 0 for Cityscapes → Woodscape (or
FDD); and β 1 = 100 . 0 , β 2 = 1 . 0 , β 3 = 100 . 0 for GTAV → Woodscape (or FDD).
Further details on the implementation as well as our hyperparameter selections are provided in Appendix A and B.
4.3 Comparisons with State-of-the-Art Methods
We ﬁrst evaluate the effect of our distortion-aware adaptation framework when applied to the based adaptation methods: AdaptSeg [33] and AdvEnt [34].
We use DeepLab-V2 [6] with ResNet101 backbone as the base semantic segmentation architecture M .
We test the different spatial transformations ( +RDL : applying a diffeomorphic transformation learned by RDL, +RA : applying an afﬁne transformation via RandAugment [9], +RA+RDL : applying both diffeomorphic and afﬁne transformations) on source images in the adaptation tasks.
Woodscape as target domain (Tab.1).
Compared to the based adaptation methods, applying either one of ﬁne-grained diffeomorphic ( +RDL ) or global afﬁne transformations ( +RA ) achieves clear improvements on the all adaptation tasks.
Remarkably, applying both ﬁne-grained diffeomorphic and global afﬁne transformations ( +RA+RDL ) achieves signiﬁcant improvements of +7 .
38% and +2 .
92% on Cityscapes → Woodscape and GTAV → Woodscape tasks, respectively.
Note that the performance improvement of the distortion-aware adaptation comes from object classes such as person ( +20 .
07% ), car ( +14 . 49% ), bus ( +14 . 61% ), and truck ( +12 .
61% ) as well as background such as sidewalk ( +13 . 09% ) (see class-wise iou( % ) in Appendix B.4).
Optical distortion gradually
[Table]
increases when objects and backgrounds appear closer to the image periphery.
We hypothesize that our distortion adaptation method rectiﬁes erroneous predictions in such distorted regions.
These improvements are also conﬁrmed in Fig.1-(b), Fig.3, and Fig.5 (See Section 4.4).
FDD as target domain (Tab.1).
Here, the results are consistent with the previous adaptation tasks.
Again, +RA+RDL clearly shows superior prediction performance to the based methods by up to +3 .
45% and +3 .
62% on Cityscapes → FDD and GTAV → FDD, respectively.
Notably, +RDL contributes up to 3 .
56% (AdvEnt+RDL on Cityscapes → FDD) of improvements, which also consistently achieves higher performance gain than +RA ( +2 .
Such results are also echoed in the Cityscapes → CityscapesFishEye task in Appendix B.2.
Note that +RDL always leads to improvements in segmentation adaptation, regardless of domain shift, throughout our experiments ( e.g., based methods vs. +RDL , and +RA vs. +RA+RDL ) as presented in Tab.1 and Tab.7 in Appendix
B.2.
In contrast, the randomized afﬁne augmentation (RA) leads to degraded segmentation adaptation results upon the geometric distributional shifts between source and target domains ( e.g., +RDL vs.
+RDL+RA in Cityscapes → CityscapesFisheye and Cityscapes → FDD).
Thus, we may conclude that our learnable diffeomorphic transformation (RDL) plays an important role in aligning the domain gap of geometric deformation.
Relationship to SSL (Tab.2).
We evaluate the effect of our distortion-aware domain adaptation (DaDA), as a “ warm-up ” phase, for state-of-the-art adaptation methods using self-supervised learning:
IAST [26], IntraDA [27], and ProDA [40].
In particular, we train the initial segmentation model with +RA+RDL distortion adaptation; and use AdaptSeg-based adaptation models for IAST [26] and
ProDA [40]; and AdvEnt-based models for IntraDA [27].
Tab.2 shows the effectiveness of DaDA, where it further improves the self-supervised learning by providing higher-quality initial models and pseudo labels.
DaDA attains up to +6 .
82% improvement against the baseline SSL methods.
This implies that satisfactory distortion-aware adaptation cannot be achieved only by relying on SSL.
Comparisons with Image-to-Image Translation.
We compare our relative distortion learning (RDL) with an existing image-to-image translation method CycleGAN [42].
Unpaired rectilinear source images (Cityscapes and GTAV) and ﬁsheye target images (Woodscape) are given to both RDL and CycleGAN.
The input images are randomly cropped and resized to 512 × 512 .
Fig.4 shows that
CycleGAN fails to generate transformed images from source images mimicking the distortion style of
[Table]
target images.
This is obvious to observe since CycleGAN-like existing translation approaches do not have devices ( e
[Table]
g., diffeomorphic transformer) to geometrically transform source images.
In contrast,
RDL enables modeling relative distortion across domains and generates transformed source images alike the distortion style of target images.
In Fig.4, buildings and vehicles are distorted replicating counterparts in target images via RDL, while CycleGAN focuses on translating texture and color of images.
Fig.4 also demonstrates that RDL is able to reconstruct source images ( I S ) from the distortion-translated images ( I S → T ) via an inverse relative deformation ﬁeld Φ T → S .
Overall, these results imply that our distortion-aware losses ( L recon , L sem ) are effective in guiding the generator G to produce convincing relative deformation ﬁelds
[Table]
4.4 Ablation Studies
To better understand the effect of our distortion-aware adaptation approach
[Table]
we conduct ablation studies on the distortion-aware losses and the competence in predicting distorted image regions.
Effect of Distortion-aware Losses (Tab.3).
We evaluate the effect of the distortion-aware losses on segmentation performance.
Tab.3 depicts the improvement of prediction performance compared to the based methods, by adding individual or all of the proposed distortion-aware losses in Eq.(11).
Only adding the adversarial loss (+ L adv _ G ) contributes to the adaptation performance up to +3 .
28% of improvements.
Progressive introduction of the distortion-aware losses consistently improves prediction accuracy (+ L sem , + L recon ).
Ultimately, utilizing all losses together with segmentation adaptation achieves +5 .
and +1 .
69% of improvements for Cityscapes → Woodscape and
GTAV → Woodscape, respectively.
Here we observe a relatively smaller improvement in GTAV → Woodscape task, which exhibits both severe visual and geometric distributional misalignment.
We believe that introducing additional texture-aware translation methods along with our distortion adaptation approach might lead to further improvement in such a synthetic-to-real adaptation task.
Overall, the relative distortion learning supervised by the distortion-aware losses effectively reduces domain shifts in optical distortion, and thus improves the prediction performance.
Distortion-aware mIoU. To quantitatively demonstrate the effectiveness of
DaDA on predicting distorted image areas, we propose a distortion-aware mIoU( % ) metric.
The image coordinates are normalized to [-1.0,1.0] and we gradually mask label where (cid:112) i 2 + j 2 of pixel at ( i, j ) is smaller than a certain distance threshold ( dist ).
In Fig.5 (bottom row), dist = 0 . 0 shows that the original label is used for mIoU( % ) calculation, while dist = 0 .
8 indicates that large areas of undistorted regions are masked out in the label so that we evaluate the models on distorted regions.
Plots in Fig.5 (top row) show that the performance gain ( ∆ mIoU) achieved by adding DaDA increases as dist increases.
This indicates that DaDA effectively addresses domain shifts in distortion and improves the prediction performance for the distorted image regions.
Discussion and Conclusion
In this paper we proposed a novel distortion-aware domain adaptation (DaDA) framework that is capable of modeling domain shifts in geometric deformation based on a relative distortion learning (RDL) method.
Besides, we demonstrated that our distortion adaptation approach further improves self-supervised learning by providing higher-quality initial models and pseudo labels.
Extensive experimental results proved that our method minimizes domain shifts in optical distortion, and thus signiﬁcantly improves the segmentation adaptation performance under unknown relative distortion across domains.
In the future we will investigate the interplay between the texture-oriented and the distortion-oriented domain shifts to further improve the unsupervised domain adaptation.
While we ﬁrst tackle adapting existing semantic segmentation models trained on rectilinear images to unlabeled ﬁsheye images, various set-ups of domain adaptation tasks among distorted and rectilinear images can be further considered.
For example, we may use distorted images as source and rectilinear images as a target, or both source and target domains include distorted images.
Applying relative distortion learning to such extended adaptation tasks could be another interesting direction for future work.
We hope our work provides a solid baseline and new perspectives on distortion-aware domain adaptation
Disclosure of Funding
We received no third-party funding for this work.