EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
Abstract
Score-based diffusion models (SBDMs) have achieved the SOTA FID results in unpaired image-to-image translation (I2I).
However, we notice that existing methods totally ignore the training data in the source domain, leading to sub-optimal solutions for unpaired I2I. To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I. Building upon two feature extractors, we carefully design the energy function such that it encourages the transferred image to preserve the domain-independent features and discard domain-specific ones.
Further, we provide an alternative explanation of the EGSDE as a product of experts, where each of the three experts (corresponding to the SDE and two feature extractors) solely contributes to faithfulness or realism.
Empirically, we compare EGSDE to a large family of baselines on three widely-adopted unpaired I2I tasks under four metrics.
EGSDE not only consistently outperforms existing SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance.
Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the realism results further (e.g., FID of 51.04 in
Cat → Dog and FID of 50.43 in Wild → Dog on AFHQ) by tuning hyper-parameters.
The code is available at https://github.com/ML-GSAI/EGSDE.
Introduction
Unpaired image-to-image translation (I2I) aims to transfer an image from a source domain to a related target domain, which involves a wide range of computer vision tasks such as style transfer, super-resolution and pose estimation [35].
In I2I, the translated image should be realistic to fit the style of the target domain by changing the domain-specific features accordingly, and faithful to preserve the domain-independent features of the source image.
Over the past few years, generative adversarial networks [12] (GANs)-based methods [10, 60, 54, 36, 3, 57, 44, 19, 17, 26, 10] dominated this field due to their ability to generate high-quality samples.
In contrast to GANs, score-based diffusion models (SBDMs) [48, 16, 34, 49, 2, 31] perturb data to a Gaussian noise by a diffusion process and learn the reverse process to transform the noise back to the data distribution.
Recently, SBDMs achieved competitive or even superior image generation performance to GANs [9] and thus were naturally applied to unpaired I2I [7, 32], which have achieved
[Figure]
npu t
O u t
[Figure]
t
[Figure]
Output realistic expert realistic 𝜆 !
=
𝜆 !
=faithful 𝜆 " =𝜆 " =faithful expert
[Figure]
the state-of-the-art FID [13] and KID [4] results empirically.
However, we notice that these methods did not leverage the training data in the source domain at all .
Indeed, they trained a diffusion model solely on the target domain and exploited the test source image during inference (see details in
Sec. 2.2).
Therefore, we argue that if the training data in the source domain can be exploited together with those in the target domain, one can learn domain-specific and domain-independent features to improve both the realism and faithfulness of the SBDMs in unpaired I2I.
To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained across the two domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I. Formally, EGSDE defines a valid conditional distribution via a reverse time SDE that composites the energy function and the pretrained SDE.
Ideally, the energy function should encourage the transferred image to preserve the domain-independent features and discard domain-specific ones.
To achieve this, we introduce two feature extractors that learn domainindependent features and domain-specific ones respectively, and define the energy function upon the similarities between the features extracted from the transferred image and the test source image.
Further, we provide an alternative explanation of the discretization of EGSDE in the formulation of product of experts [15].
In particular, the pretrained SDE and the two feature extractors in the energy function correspond to three experts and each solely contributes to faithfulness or realism.
Empirically, we validate our method on the widely-adopted AFHQ [8] and CelebA-HQ [20] datasets including Cat → Dog, Wild → Dog and Male → Female tasks.
We compare to a large family of baselines, including the GANs-based ones [36, 60, 17, 26, 3, 10, 57, 58] and SBDMs-based ones [7, 32] under four metrics (e.g., FID).
EGSDE not only consistently outperforms SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance.
Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the FID further (e.g., 51.04 in Cat → Dog and 50.43 in Wild → Dog ) by tuning hyper-parameters.
EGSDE can also be extended to multi-domain translation easily.
Background 2.1 Score-based Diffusion Models
Score-based diffusion models (SBDMs) gradually perturb data by a forward diffusion process, and then reverse it to recover the data [49, 2, 47, 16, 9].
Let q ( y 0 ) be the unknown data distribution on R D .
The forward diffusion process { y t } t ∈ [0 ,T ] , indexed by time t , can be represented by the following forward SDE:
[Equation]
where w ∈ R D is a standard Wiener process, f ( · , t ) :
R D → R D is the drift coefficient and g ( t ) ∈ R is the diffusion coefficient.
The f ( y , t ) and g ( t ) is related into the noise size and determines the perturbation kernel q t |
( y t | y 0 ) from time 0 to t .
In practice, the f ( y , t ) is usually affine so that the the perturbation kernel is a linear Gaussian distribution and can be sampled in one step.
Let q t ( y ) be the marginal distribution of the SDE at time t in Eq.
Its time reversal can be described by another SDE [49]:
[Equation]
where w is a reverse-time standard Wiener process, and d t is an infinitesimal negative timestep.
[49] adopts a score-based model s ( y , t ) to approximate the unknown ∇ y log q t ( y ) by score matching, thus inducing a score-based diffusion model (SBDM), which is defined by a SDE:
[Equation]
There are numerous SDE solver to solve the Eq.
(3) to generate images.
[49] discretizes it using the
Euler-Maruyama solver.
Formally, adopting a step size of h , the iteration rule from s to t =
s − h is:
[Equation]
2.2 SBDMs in Unpaired Image to Image Translation
Given unpaired images from the source domain X ⊂ R D and the target domain Y ⊂ R D as the training data, the goal of unpaired I2I is to transfer an image from the source domain to the target domain.
Such a process can be formulated as designing a distribution p ( y 0 | x 0 ) on the target domain
Y conditioned on an image x 0 ∈ X to transfer.
The translated image should be realistic for the target domain by changing the domain-specific features and faithful for the source image by preserving the domain-independent features.
ILVR [7] uses a diffusion model on the target domain for realism.
Formally, ILVR starts from y T ∼ N ( 0 , I ) and samples from the diffusion model according to Eq.
(4) to obtain y t .
For faithfulness, it further refines y t by adding the residual between the sample y t and the perturbed source image x t through a non-trainable low-pass filter
[Equation]
where Φ( · ) is a low-pass filter and q t | 0 ( ·|· ) is the perturbation kernel determined by the forward SDE in Eq.
Similarly, SDEdit [32] also adopts a SBDM on the target domain for realism, i.e., sampling from the
SBDM according to Eq.
For faithfulness, SDEdit starts the generation process from the noisy source image y M ∼ q M |
( y M | x 0 ) , where M is a middle time between 0 and T , and is chosen to preserve the original overall structure and discard local details.
We use p r 1 ( y 0 | x 0 ) to denote the marginal distribution defined by such SDE conditioned on x 0 .
Notably, these methods did not leverage the training data in the source domain at all and thus can be sub-optimal in terms of both the realism and faithfulness in unpaired I2I.
Method
To overcome the limitations of existing methods [7, 32] as highlighted in Sec. 2.2, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pre-trained across the two domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I (see Fig. 2).
EGSDE defines a valid conditional distribution p ( y 0 | x 0 ) by compositing a pretrained SDE and a pretrained energy function under mild regularity conditions 2 as follows:
[Equation]
[Figure]
where w is a reverse-time standard Wiener process, d t is an infinitesimal negative timestep, s ( · , · ) : R D × R → R D is the score-based model in the pretrained SDE and E ( · , · , · ) : R D × R D × R → R is the energy function.
The start point y M is sampled from the perturbation distribution q M |
( y M | x 0 ) [32], where M = 0 .
5 T typically.
We obtain the transferred images by taking the samples at endpoint t =
following the SDE in Eq.
Similar to the prior work [7, 32], EGSDE employs an SDE trained solely in the target domain as in Eq.
(2), which defines a marginal distribution of the target images and mainly contributes to the realism of the transferred samples.
In contrast, the energy function involves the training data across both the source and target domain, making EGSDE distinct from the prior work [7, 32].
Notably, although many other possibilities exist, we carefully design the energy function such that it (approximately) encourages the sample to retain the domain-independent features and discard the domain-specific ones to improve both the faithfulness and realism of the transferred sample.
Below, we formally formulate the energy function.
3.1 Choice of Energy
In this section, we show how to design the energy function.
Intuitively, during the translation, the domain-independent features (pose, color, etc .
on Cat → Dog) should be preserved while the domain-specific features (beard, nose, etc .
on Cat → Dog) should be changed accordingly.
Motivated by this, we decompose the energy function E
( y , x , t ) as the sum of two log potential functions [5]:
[Equation]
where E i ( · , · , · ) : R D × R D
×
R
→
R
and
E s ( · , · , · ) : R D × R D × R → R are the log potential functions, x t is the perturbed source image in the forward SDE, q t | 0 ( ·|· ) is the perturbation kernel from timeto time t in the forward SDE, S s ( · , · , · ) : R D × R D × R → R and S i ( · , · , · ) : R D × R D × R → R are two functions measuring the similarity between the sample and perturbed source image, and λ s ∈ R > 0 , λ i ∈ R > 0 are two weighting hyper-parameters.
Note that the expectation w.r.t.
q t | 0 ( x t | x ) in Eq.
(7) guarantees that the energy function changes slowly over the trajectory to satisfy the regularity conditions in Appendix A.1.
To specify S s ( · , · , · ) , we introduce a time-dependent domain-specific feature extractor E s ( · , · ) : R D × R → R C × H × W , where C is the channel-wise dimension, H and W are the dimension of height and width.
In particular, E s ( · , · ) is the all but the last layer of a classifier that is trained on both domains to predict whether an image is from the source domain or the target domain.
Intuitively,
E s ( · , · ) will preserve the domain-specific features and discard the domain-independent features for accurate predictions.
Building upon it, S s ( · , · , · ) is defined as the cosine similarity between the features extracted from the generated sample and the source image as follows:
[Equation]
where E hws ( · , · ) ∈ R C denote the channel-wise feature at spatial position ( h, w ) .
Here we employ the cosine similarity since it preserves the spatial information and helps to improve the FID score empirically (see Appendix C.1 for the ablation study).
Intuitively, reducing the energy value in Eq.
(7) encourages the transferred sample to discard the domain-specific features to improve realism.
To specify S i ( · , · , · ) , we introduce a domain-independent feature extractor E i ( · , · ) :
R D × R → R D , which is a low-pass filter.
Intuitively, E i ( · , · ) will preserve the overall structures (i.e., domainindependent features) and discard local information like textures (i.e., domain-specific features).
Building upon it, S i ( · , · , · ) is defined as the negative squared L 2 distance between the features extracted from the generated sample and source image as follows:
[Equation]
Here, we choose negative squared L 2 distance as the similarity metric because it helps to preserve more domain-independent features empirically (see Appendix C.1 for the ablation study).
Intuitively, reducing the energy value in Eq.
(7) encourages the transferred sample to preserve the domainindependent features to improve faithfulness.
In this paper we employ a low-pass filter for its simpleness and effectiveness while we can train more sophisticated E i , e.g., based on disentangled representation learning methods [42, 6, 14, 23, 28], on the data in the two domains.
In our preliminary experiment, alternative to Eq.
(7), we consider a simpler energy function that only involves the original source image x as follows:
[Equation]
which does not require to take the expectation w.r.t.
x t .
We found that it did not perform well because it is not reasonable to measure the similarity between the noise-free source image and the transferred sample in a gradual denoising process.
See Appendix C.2 for empirical results.
3.2 Solving the Energy-guided Reverse-time SDE
Based on the pretrained score-based model s ( y , t ) and energy function E ( y , x , t ) , we can solve the proposed energy-guided SDE to generate samples from conditional distribution p ( y 0 | x 0 ) .
There are numerical solvers to approximate trajectories from SDEs.
In this paper, we take the Euler-Maruyama solver following [32] for a fair comparison.
Given the EGSDE as in Eq.
(6) and adopting a step size h , the iteration rule from s to t =
s − h is:
[Equation]
The expectation in E ( y s , x 0 , s ) is estimated by the Monte Carlo method of a single sample for efficiency.
For brevity, we present the general sampling procedure of our method in Algorithm 1.
In experiments, we use the variance preserve energy-guided SDE (VP-EGSDE) [49, 16] and the details are explained in Appendix A.3, where we can modify the noise prediction network to ˜ ϵ ( y , x 0 , t ) = ϵ ( y , t ) + (cid:112) ¯ β t ∇ y E ( y , x 0 , t ) and take it into the sampling procedure in DDPM [16].
Following SDEdit [32], we further extend this by repeating the Algorithm 1 K times (see details in Appendix A.2).
Further, we explain the connection with classifier guidance[9] in Appendix A.5.
3.3 EGSDE as Product of Experts
Inspired by the posterior inference process in diffusion models [46], we present a product of experts [15] explanation for the discretized sampling process of EGSDE, which formalizes our motivation in an alternative perspective and provides insights on the role of each component in EGSDE.
We first define a conditional distribution ˜ p ( y t | x 0 ) at time t as a product of experts:
[Equation]
Require: the source image x 0 , the initial time M , denoising steps N , weighting hyper-parameters λ s , λ i , the similarity function S s ( · , · , · ) , S i ( · , · , · ) , the score function s ( · , · ) y ∼ q M | 0 ( y | x 0 )
# the start point h
=
MN for i
= N to 1 do s ← ih x ∼ q s | 0 ( x | x 0 )
# sample perturbed source image from the perturbation kernel
E ( y , x , s ) ← λ s S
s
( y , x , s )
− λ
i S i
( y , x , s )
# compute energy with one Monte Carlo y ← y − [ f ( y , s ) − g
( s )
( s ( y , s ) − ∇ y E
( y , x , s ))]
h # the update rule in Eq.
(12) z ∼ N ( 0 , I ) if i > 1 , else z =y ← y + g ( s ) √ h z end for y 0 ← y return ywhere
Z t is the partition function, p e ( y t | x 0 )
∝ exp( −E ( y t , x 0 , t )) and p r 1 ( y t | x 0 ) is the marginal distribution at time t defined by SDEdit based on a pretrained SDE on the target domain.
To sample from ˜ p
( y t | x 0 ) , we need to construct a transition kernel ˜ p ( y t | y s ) , where t = s − h
and h is small.
Following [46], using the desirable equilibrium ˜ p ( y t | x 0 ) =
(cid:82) ˜ p ( y t | y s )
˜ p ( y s | x 0 )
d y s , we construct the ˜ p ( y t | y s ) as follows:
[Equation]
where ˜ Z t ( y s ) is the partition function and p ( y t | y s )
=
N ( µ ( y s , h ) , Σ( s, h ) I ) is the transition kernel of the pretrained SDE in Eq.
(4), i.e., µ ( y s , h ) =
y s − [ f ( y s , s ) − g ( s ) 2 s ( y s , s )] h and Σ( s, h ) =
g ( s ) 2 h .
Assuming that E ( y t , x 0 , t ) has low curvature relative to Σ( s, h ) − 1 , it can be approximated using Taylor expansion around µ ( y s , h ) and further we can obtain
[Equation]
More details about derivation are available in Appendix A.4.
We can observe the transition kernel ˜ p ( y t | y s ) in (15) is equal to the discretization of our EGSDE in Eq.
Therefore, solving the energy-guided SDE in a discretization manner is approximately equivalent to drawing samples from a product of experts in Eq.
Note that E ( y t , x 0 , t ) =
λ s E s ( y t , x 0 , t )
+ λ i E i
( y t , x 0 , t ) , the ˜ p ( y t | x 0 ) can be rewritten as:
[Equation]
[Equation]
where p r 2 ( y t | x 0 ) ∝ exp( − λ s E s ( y t , x 0 , t )) , p f ( y t | x 0 ) ∝ exp( − λ i E i ( y t , x 0 , t )) .
In Eq.
(16), by setting t = 0 , we can explain that the transferred samples approximately follow the distribution defined by the product of three experts, where p r 1 ( y t | x 0 ) and p r 2 ( y t | x 0 ) are the realism experts and p f ( y t | x 0 ) is the faithful expert , corresponding to the score function s ( y , t ) and
the log potential functions E s ( y , x , t ) and E i ( y , x , t ) respectively.
Such a formulation clearly explains the role of each expert in EGSDE and supports our empirical results.
Related work
Apart from the prior work mentioned before, we discuss other related work including GANs-based methods for unpaired I2I and SBDMs-based methods for image translation.
GANs-based methods for Unpaired I2I.
Although previous paired image translation methods have also achieved remarkable performances [18, 51, 50, 37, 55, 59], we mainly focus on unpaired image translation in this work.
The methods for two-domain unpaired I2I are mainly divided into two classes: two-side and one-side mapping [35, 57].
In the two-side framework [60, 54, 25, 29, 27, 24, 11, 1, 53, 56, 21], the cycle-consistency constraint is the most widely-used strategy such as in [Figure] I [Figure]
CycleGAN [60], DualGAN [54] and DiscoGAN [25].
The key idea is that the translated image should be able to be reconstructed by an inverse mapping.
More recently, there are numerical studies to improve this such as SCAN [27] and U-GAT-IT [24].
Specifically, U-GAT-IT [24] applies an attention module to let the generator and discriminator focus on more important regions instead of the whole regions through the auxiliary classifier.
Since such bijective projection is too restrictive, several studies are devoted to one-side mapping [36, 3, 10, 60, 57, 38, 19].
One representative approach is to design some kind of geometry distance to preserve content [35].
For example, DistanceGAN [3] keeps the distances between images within domains.
GCGAN [10] maintains geometry-consistency between input and output.
CUT [36] maximizes the mutual information between the input and output using contrastive learning.
LSeSim
[57] learns spatially-correlative representation to preserve scene structure consistency via self-similarities.
SBDMs-based methods for Image Translation.
Several studies leveraged SBDMs for image translation due to their powerful generative ability and achieved good results.
For example, DiffusionCLIP [22] fine-tune the score network with CLIP [39] loss, which is applied on text-driven image manipulation, zero-shot image manipulation and multi-attribute transfer successfully.
GLIDE [33] and SDG [30] has achieved great performance on text-to-image translation.
As for I2I, SR3 [41] and Palette [40] learn a conditional SBDM and outperform state-of-art GANs-based methods on super-resolution, colorization and so on, which needs paired data.
For unpaired I2I, UNIT-DDPM [43] learns two SBDMs and two domain translation models using cycle-consistency loss.
Compared with it, our method only needs one SBDM on the target domain, which is a kind of one-side mapping.
ILVR [7] and SDEdit [32] utilize a SBDM on the target domain and exploited the test source image to refine inference, which ignored the training data in the source domain.
Compared with these methods, our method employs an energy function pretrained across both the source and target domains to improve the realism and faithfulness of translated images.
Experiment
Datasets.
We validated the EGSDE on following datasets, where all images are resized to 256 × 256 : (1) CelebA-HQ [20] contains high quality face images and is separated into two domains: male and female.
Each category has 1000 testing images.
We perform Male → Female on this dataset.
(2) AFHQ [8] consists of high-resolution animal face images including three domains: cat, dog and wild, which has relatively large variations.
Each domain has 500 testing images.
We perform
Cat → Dog and Wild → Dog on this dataset.
We also perform multi-domain translation on AFHQ dataset and the experimental results are reported in Appendix D.
Implementation.
The time-dependent domain-specific extractor E s ( x , t ) is trained based on the backbone in [9].
The resize function including downsampling and upsampling operation is used as low-pass filter and is implemented by [45].
For generation process, by default, the weight parameter λ s , λ i is set 500 and 2 respectively.
The initial time M and denoising steps N is set 0 .
5 T and 500 by default.
More details about implementation are available in Appendix B.
Evaluation Metrics.
We evaluate translated images from two aspects: realism and faithfulness.
For realism, we report the widely-used Frechet Inception Score (FID) [13] between translated images and the target dataset.
To quantify faithfulness, we report the L 2 distance, PSNR and SSIM [52] between each input-output pair.
To quantify both faithfulness and realism, we leverage Amazon Mechanical
Turk(AMT) human evaluation to perform pairwise comparisons between the baselines and EGSDE.
More details is available in Appendix B.6.
5.1 Two-Domain Unpaired Image Translation
In this section, we compare EGSDE with the following state-of-the-art I2I methods in three tasks:
SBDMs-based methods including ILVR [7] and SDEdit [32], and GANs-based methods including
CUT [36], which are reproduced using public code.
On the most popular benchmark Cat → Dog, we also report the performance of other state-of-the-art GANs-methods , where StarGAN v2 [8] is evaluated by the provided public checkpoint and the others are public results from CUT[36] and
ITTR [58].
We provide more details about reproductions in Appendix B.7.
The quantitative comparisons and qualitative results are shown in Table 1 and Figure 3.
We can derive several observations.
First , our method outperforms the SBDMs-based methods significantly in almost all realism and faithfulness metrics, suggesting the effectiveness of employing energy function pretrained on both domains to guide the generation process.
Especially, compared with the most direct competitor, i.e., SDEdit, with a lower L 2 distance at the same time, EGSDE improves the FID score by 8.35, 8.76 and 7.5 on Cat → Dog, Wild → Dog and Male → Female respectively.
Second , EGSDE † outperforms the current state-of-art GANs-based methods by a large margin on the challenging AFHQ dataset.
For example, compared with CUT [36], we achieve an improvement of
FID score with 25.17 and 42.51 on the Cat → Dog and Wild → Dog tasks respectively.
In addition, the human evaluation shows that EGSDE are preferred compared to all baselines (> 50 % ).
The qualitative results in Figure 3 agree with quantitative comparisons in Table 1, where our method achieved the results with the best visual quality for both realism and faithfulness.
We show more qualitative results and select some failure cases in Appendix C.6.
5.2 Ablation Studies
The function of each expert.
We validate the function of realistic expert E s ( y , x , t ) and faithful expert E i ( y , x , t ) by changing the weighting hyper-parameter λ s and λ i .
As shown in Table 3 and
Figure 1, larger λ s results in more realistic images and larger λ i results in more faithful images.
More results is available in Appendix C.5.
The choice of initial time M .
We explore the effect of the initial time M of EGSDE.
As shown in
Figure 4, the larger M results in more realistic and less faithful image.
More results is available in
Appendix C.3.
Repeating K Times.
Following SDEdit [32], we show the results of repeating the Algorithm 1 K times.
The quantitative and qualitative results are depicted in Table 2 and Figure 4.
The experimental results show the EGSDE outperforms SDEdit in each K step in all metrics.
With the increase of K , the SDEdit generates more realism images but the faithful metrics decrease sharply, because it only
[Table] 423 ± 0. EGSDE 65.82 ± 0. 47.22 ± 0.
19.31 ± 0. 0.415 ± 0.
EGSDE † 51.04 ± 0. 62.06 ± 0. 17.17 ± 0. 0.361 ± 0.
Wild → Dog
CUT [36] 92.
ILVR [7] 75.33 ± 1.
63.40 ± 0.
16.85 ± 0.
0.287 ± 0.
SDEdit [32] 68.51 ± 0.
55.36 ± 0.
17.98 ± 0.
0.343 ± 0.
EGSDE 59.75 ± 0.
54.34 ± 0.
18.14 ± 0.
0.343 ± 0.
EGSDE † 50.43± 0.
66.52± 0.
16.40± 0.
0.300± 0.
Male → Female
CUT [36] 31. ILVR [7] 46.12 ± 0. 52.17 ± 0. 18.59 ± 0.
0.510 ± 0.
SDEdit [32] 49.43 ± 0.
43.70 ± 0.
20.03 ± 0.
0.572 ± 0.
EGSDE 41.93 ± 0.
42.04 ± 0. 20.35 ± 0. 0.574 ± 0. EGSDE † 30.61 ± 0.
53.44 ± 0. 18.32 ± 0. 0.510 ± 0. - utilizes the source image at the initial time M .
As shown in Figure 4, when K =3, SDEdit discard the domain-independent information of the source image (i.e., color and background) while our method still preserves them without harming realism.
Conclusions and Discussions
In this paper, we propose energy-guided stochastic differential equations (EGSDE) for realistic and faithful unpaired I2I, which employs an energy function pretrained on both domains to guide the generation process of a pretrained SDE.
Building upon two feature extractors, we carefully design the energy function to preserve the domain-independent features and discard domain-specific ones of the source image.
We demonstrate the EGSDE by outperforming state-of-art I2I methods on three widely-adopted unpaired I2I tasks.
One limitation of this paper is we employ a low-pass filter as the domain-independent feature extractor for its simpleness and effectiveness while we can train more sophisticated extractor, e.g. based on disentangled representation learning methods [42, 6, 14, 23, 28], on the data in the two domains. We Source M = 0.3T
M = 0.7T
[Figure] S D E d it K= K=
K=
K=
K=
K=
[Figure]
[Table]
[Table]
leave this issue in future work.
In addition, we must take care to exploit the method to avoid the potential negative social impact (i.e., generating fake images to mislead people).
Acknowledgement
We thank Cheng Lu, Yuhao Zhou, Haoyu Liang and Shuyu Cheng for helpful discussions about the method and its limitations.
This work was supported by the National Key Research and Development Program of China (2020AAA0106302); NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); Beijing NSF Project (No. JQ19016);
Beijing Outstanding Young Scientist Program NO.
BJJWZYJH012019100020098; a grant from Tsinghua Institute for Guo Qiang; the High Performance Computing Center, Tsinghua University; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13).
Part of the computing resources supporting this work, totaled A100 GPU hours, were provided by High-Flyer AI.
(Hangzhou High-Flyer AI Fundamental Research Co., Ltd.).
J.Z was also supported by the XPlorer Prize.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[-MN y 
-]
[Yes] See Section 6.
[-( s ) 
 ( s ( y , s ) − ∇ y E ( y , x , s ))] h z ∼ N ( 0 , I ) if i > 1 , else z =y ← y + g ( s ) √ h z end for y 
-](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[-← y end for y 
-]
[Yes] 2.
[-The perturbation kernel q t | 
-]
[-( y t | y 0 ) is N ( y 
 e − 12 (cid:82) t 0 β ( s )d s , (1 − e − (cid:82) t 0 β ( s ) d s ) I ) and β ( t ) = β min + t ( β max − β min ) in practice.-]If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sectionand Appendix B (b) Did you specify all the [-update rule in Eq. (18) z ∼ N ( 0 , I ) if i > 1 , else z =y ← y + (cid:112) β ( s ) h z end for y 
 ← y return y-]training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B and Appendix C (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5 and Appendix B (b) Did you mention the license of the assets?
[Yes] See Appendix B (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Appendix B (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use the public datasets.
[-λ s = 0 , λ i = 
-]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We use the public datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]