K ERNEL OF C YCLE GAN AS A P RINCIPLE HOMOGENEOUS SPACE
A BSTRACT
Unpaired image-to-image translation has attracted signiﬁcant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data.
It is known that the
CycleGAN problem might admit multiple solutions, and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions.
We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions.
We examine the case of zero ‘pure’ CycleGAN loss ﬁrst in its generality, and, subsequently, expand our analysis to approximate solutions for ‘extended’
CycleGAN loss where identity loss term is included.
In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist.
Furthermore, we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results.
We conclude that ﬁnding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result utterly useless.
I NTRODUCTION
Machine learning methods for image-to-image translation are widely studied and have applications in several ﬁelds.
In medical imaging, the CycleGAN has found an important application for translating one modality to another, for instance in MR to CT translation (Han, 2017; Sj¨olund et al., 2015;
Wolterink et al., 2017).
Classically, these methods are trained in a supervised setting making their applications limited due to the a lack of good paired data.
Similar issues appear in e.g. transferring the style of one artist to another (Gatys et al., 2015) or adding snow to sunny California streets (Liu et al., 2017).
Unpaired image-to-image translation models such as CycleGAN (Zhu et al., 2017) promise to solve this issue by only enforcing a relationship on a distribution level, thus removing the need for paired data.
However, given their widespread use, it is paramount to gain more understanding of their dynamics, to prevent unexpected things from happening, e.g., (Cohen et al., 2018).
As a step in that direction, we explore the solution space of the CycleGAN in the subsequent sections of this paper.
The general task of unpaired domain translation can be informally described as follows:
given two probability spaces X and Y which represent our domains, we seek to learn a mapping G : X → Y [Figure] [Equation]
[Figure] [Equation] Y
[Equation]
[Figure] Y F G y y
[Figure]
y x
[Figure]
X
[Equation]
Y
[Equation]
[Figure]
X
[Figure]
such that a sample x ∈ X is mapped to a sample G ( x ) ∈ Y where
G ( x
[Equation]
∈ Y is the best representative of x in Y .
The mapping G is typically approximated by a neural network G θ parametrized by θ .
Without paired data, directly solving this is impossible but on a distribution level it is easily seen if G solves eq.
(1) then the distribution of G ( x ) as x is sampled from X is equal to that of Y .
Mathematically, if X =
( X, X , µ ) and Y =
( Y, Y , ν ) are probability spaces with probability measures µ and ν respectively, this can be written as
[Equation]
for all A ∈ Y , (2)
Or in words, the probability measure ν equals the push-forward measure G ∗ µ .
By Jensen’s equality we can relate this to the ﬁxed f-divergence D f :
[Equation]
only
[Equation]
[Equation]
While adversarial adversarial optimization techniques such as GANs can in principle solve problem eq.
(3), they remain under-constrained thus not giving a reasonable solution to the original problem eq.
The idea behind the cycle consistency condition from (Zhu et al., 2017) is to enforce additional constraints by introducing another function F : Y → X , which is also approximated by a neural network and tries to solve the inverse task: for each y ∈ Y ﬁnd F ( y )
∈
X that would be the best translation of y to X .
Similar to the reasoning above, this condition would imply that
[Equation]
The goal is to enforce that F ( G ( x ))
≈
x for all x ∈ X and, similarly, that G ( F ( y ))
≈ y for all y ∈ Y , i.e. to minimize the following cycle consistency loss
[Equation]
where typically the L 1 norm is chosen, but in principle any norm can be chosen.
Zhu et al.
(Zhu et al., 2017) also suggested that an adversarial loss could in principle have been used here as well, but they did not note any performance improvement.
Combining these losses, we arrive at the CycleGAN loss deﬁned as
[Equation]
where the factor α cyc > 0 determines the weight of the cycle consistency term.
We illustrate the
CycleGAN model in ﬁg. 1.
Precautions with generative models have been addressed before, for example, unpaired image to image translation can hallucinate features in medical images (Cohen et al., 2018).
Furthermore, it was already noted in (Zhu et al., 2017) that the CycleGAN might admit multpiple solutions and that the issue of tint shift in image-to-image translation arises due to the fact that for a ﬁxed input image x ∈ X multiple images y 1 , . . .
, y n ∈ Y with different tints might be equally plausible.
Adding identity loss term was suggested in (Zhu et al., 2017) to alleviate the tint shift issue, i.e., the extended
CycleGAN loss is deﬁned as
[Equation]
F ( y ) − y (cid:107) + E x ∼ X (cid:107) G ( x ) − x (cid:107) ) , where the factor α id ≥ 0 determines the weight of the identity loss term.
In general, to properly deﬁne the identity loss one needs to represent both X and Y as being the supported on the same manifold, which is limiting if the distributions are substantially different.
The goal of this work is to study the kernel, or null space, of the CycleGAN loss, which is the set of solutions ( G, F ) which have zero ‘pure’ CycleGAN loss, and to give a perturbation bounds for approximate solutions for the case of extended CycleGAN loss.
We do the theoretical analysis in section 2.
We show that under certain assumptions on the probability spaces X , Y the kernel has symmetries which allow for multiple possible solutions in Proposition 2.1.
Furthermore, we show in
Proposition 2.2 and the following remarks that the kernel admits a natural structure of a principle homogeneous space with the automorphism group Aut(X) of X acting on the set of solutions freely and transitively.
Next, we expand our analysis to the case of approximate solutions for the extended
CycleGAN loss by proving perturbation bounds in Proposition 2.3 and Corollary 2.1.
We discuss the existence problem of automorphism in Proposition 2.4 and Proposition 2.6.
We proceed in section 3 by showing that unexpected symmetries can be learned by a CycleGAN.
In particular, when translating the same domain to itself CycleGAN can learn a nontrivial automorphism of the domain.
In appendix A, we brieﬂy explain the measure-theoretic language we use heavily in the paper for those readers who are more used to working with distributions, and also remind the reader of some basic notions from differential geometry which we use as well.
T HEORY 2. C YCLE GAN KERNEL AS A PRINCIPLE HOMOGENEOUS SPACE
The notions of isomorphism of probability spaces and of probability space automorphisms are central to this paper.
Intuitively speaking, an isomorphism f : X → Y of probability spaces X and Y is a bijection between X and Y
such that the probability of an event
A ⊂
Y equals the probability of event { x : F ( x ) ∈ A } ⊂ X .
An isomorphism of a probability space to itself is called a probability space automorphism.
For example, if our probability space consists of samples from n -dimensional spherical Gaussian distribution, then any rotation in SO( R n ) is a probability space automorphism.
For a precise deﬁnition we refer the reader to appendix A.
Firstly, we prove that if at least one of the probability spaces X , Y admits a nontrivial probability automorphism, then any exact solution in the kernel of CycleGAN can be altered giving a different solution.
Proposition 2.1 (Invariance of the kernel) .
Let X =
( X, X , µ ) , Y =
( Y, Y , ν ) be probability spaces and ϕ : X → X be a probability space automorphism.
Let G : X → Y and F : Y → X be measurable maps satisfying
[Equation]
Then F, G are probability space isomorphisms and
[Equation]
If, furthermore, ϕ (cid:54) = id X , 1 then
[Equation]
Proof.
Since ϕ is a probability space automorphism, its inverse ϕ − 1 is an automorphism as well.
In particular, it is measure-preserving since
[Equation]
We note that by eq.
(2) and the positivity of the norms eq.
(6) implies that
[Equation]
and
[Equation]
Therefore both F and G are isomorphisms.
By deﬁnition of L ,
[Equation]
Since ( G ◦ ϕ ) ∗ µ = G ∗ ( ϕ ∗ µ ) and ϕ is measure-preserving, eq.
(9) implies that ( G ◦ ϕ ) ∗ µ = ν .
Similarly, ( ϕ − 1 ◦ F ) ∗ ν = µ since ϕ − 1 is measure-preserving as well.
This shows that
[Equation]
Using eq.
(10) and the fact that ϕ − 1 ◦ ϕ =
ϕ ◦ ϕ − 1 = id X almost everywhere, we conclude that
[Equation]
and
[Equation]
Combining these observations together, we deduce that
[Equation]
and the proof of eq.
(7) is complete.
To prove eq.
(8), ﬁrst note that there exists a set A ∈ X such that µ ( A ) > 0 and ϕ ( [Equation]
[Equation] = x for all x ∈ A, since we assume that ϕ essentially differs from the identity mapping.
If G ◦ ϕ
=
G µ -a.e.
, then
F ◦ G ◦ ϕ
=
F ◦ G µ -a.e.
as well, which implies that ϕ ( x ) = x for µ -almost every x , which is a contradiction.
In a similar way one can show that ϕ − 1 ◦ F essentially differs from F .
We provide the following converse to Proposition 2.1.
Proposition 2.2 (Kernel as a principle homogeneous space) .
Let X =
( X, X , µ ) , Y =
( Y, Y , ν ) be probability spaces.
Let F : X → Y, G : Y → X and F (cid:48) :
X → Y, G (cid:48) :
Y → X be measurable maps satisfying
[Equation]
Then there exists a unique probability space automorphism ϕ :
X → X such that
[Equation]
For the proof it sufﬁces to take ϕ :=
G ◦ F (cid:48) .
Combined with Proposition 2.1, this allows us to say that the group
Aut(X) of probability space automorphisms of X acts freely and transitively on the set of isomorphisms Iso(X , Y) when the latter set is nonempty.
This amounts to saying that the space of solutions of CycleGAN is a principle homogeneous space.
It can be helpful to view this result from the abstract category theory point of view, that is, if C is a category and X ∈ C is any ﬁxed object, then for any object Y ∈ C
the automorphism group Aut( X ) acts on the set of homomorphisms
Hom( X, Y ) on the right by composition, i.e. we deﬁne α ( φ ) :
= φ ◦ α for all φ ∈ Hom( X, Y ) , α ∈ Aut( X ) .
This action leaves the space of isomorphisms Iso( X, Y ) ⊆
Hom( X, Y ) invariant, and this restricted action is transitive if Iso( X, Y ) is nonempty, and, furthermore, free, i.e. α ( φ ) (cid:54) =
φ for all α (cid:54) =
id X and all φ ∈ Iso( X, Y ) .
To proceed with our analysis for case of approximate solutions for extended CycleGAN loss, we ﬁrst formulate a useful ‘push-forward property’ for general f -divergences between distributions on R n 2 .
The proof is provided in appendix A.
Lemma 2.1 (Push-forward property for f -divergences) .
Let p, q be distributions on R n and ϕ : R n → R n be a diffeomorphism.
Then for any f -divergence D f we have
[Equation]
[Equation]
We are now ready to prove the perturbation bounds for approximate solutions.
Proposition 2.3 (Perturbation bound) .
Let X , Y be probability spaces with probability densities p X ,
p Y ∈
L 1 ( R n ) and let ϕ ∈
Aut(X) be a diffeomorphic probability space automorphism.
Assume that ϕ − 1 is C ϕ -Lipshitz, where C ϕ > 0 is some positive constant.
Let G : R n → R n
and
F : R n → R n be measurable maps.
Then the following perturbation bound holds for extended
CycleGAN loss:
[Equation]
Proof.
The proof is an adaptation of the proof of Proposition 2.1.
By deﬁnition of L ext ,
[Equation]
Firstly, since ϕ is measure-preserving
[Equation]
D f
(( G ◦ ϕ ) ∗ p X (cid:107) p Y )
=
D f
( G ∗ p X (cid:107) p Y ) .
Using Lemma 2.
and the fact that ϕ is measure-preserving again [Equation] we see that
[Equation]
Secondly,
[Equation]
where the equality ( ∗ ) uses the fact that ϕ is measure-preserving.
As in before, E
[Equation]
since ϕ ◦ ϕ − 1 = id X almost everywhere.
Finally
[Equation]
since ϕ is a probability space automorphism and ϕ − 1 is C ϕ -Lipshitz
[Equation]
we conclude that
[Equation]
and that
[Equation]
Combining all these estimates together [Equation] we deduce that
[Equation]
and the proof is complete.
Corollary 2.1 (Asymptotic perturbation bound) .
In the setting of Proposition 2.3, let G
i
R n → R n and F i : R n → R n for i ≥ 1 be a sequence of measurable maps such that the ‘pure’ CycleGAN loss converges to zero, i [Equation] e.,
[Equation]
and let
L id
[Equation]
Then the following asymptotic perturbation bound holds for the ‘extended’ CycleGAN loss:
[Equation]
Corollary 2.1 has a direct practical implication.
When using a CycleGAN model for translating substantially different distributions (such as different medical imaging modalities) one would be forced to pick a small value for α id in order for the model to produce reasonable results.
Furthermore, since the distributions are substantially different, we can expect that L id (cid:29) 2 · E x ∼ X (cid:107) ϕ
( x ) − x (cid:107) for many nontrivial automorphism ϕ .
Therefore, the asymptotic perturbation bound automatically implies that the approximate solution space admits a lot of symmetry, potentially leading to undesirable results.
2. E XISTENCE OF AUTOMORPHISMS
By Proposition 2.1 we see that if either space admits a nontrivial probability automorphism, then the
CycleGAN problem has multiple solutions.
However, for this to be a problem in practice there must actually exist such probability automorphisms, which we shall now show is the case.
First of all, we state the following proposition, which says that we can transfer automorphism from an isomorphic copy of X to X itself.
Lemma 2.2.
Let f : Z → X be an isomorphism of probability spaces and T : Z → Z be an automorphism of Z .
Then S :
= f ◦ T ◦ f − 1 is an automorphism of X and the diagram
[Equation]
commutes.
Furthermore, if Z ⊂ R n , X ⊂ R m are submanifolds and f , T are diffeomorphisms, then
S is a diffeomorphism as well.
Proof.
The ﬁrst claim follows from invertibility of f and T .
The second claim follows from the deﬁnition of a diffeomorphism between submanifolds, see appendix A.
An important notion in probability theory is that of a Lebesgue probability space.
Many probability spaces which emerge in practice such as [0 , 1] n ⊂ R n with the Lebesgue measure or R n with a
Gaussian probability distribution, both deﬁned on the respective σ -algebras of Lebesgue measurable sets, are instances of Lebesgue probability spaces.
Deﬁnition 2.1.
A probability space X is called a Lebesgue probability space if it is isomorphic as a measure space to a disjoint union ([0 , c ] , λ ) , where λ is the Lebesgue measure on the σ -algebra of
Lebesgue measurable subsets of the interval [0 , c ] , and at most countably many atoms of total mass 1 − c .
Informally speaking, this deﬁnition says that Lebesgue probability spaces consist of a continuous part and at most countably many Dirac deltas (=atoms).
First of all, we provide an abstract result about existence of nontrivial probability space automorphisms in Lebesgue probability spaces which are either ‘not purely atomic’ or have at least two atoms with equal mass.
‘Not purely atomic’ means that the sum of the probabilities of all atoms is strictly less than 1 .
Proposition 2.4.
Let X be a Lebesgue probability space such that at least one of the assumptions
X not purely atomic; 2.
there exist at least two atoms a j , a k in X with equal mass
holds.
Then X admits nontrivial automorphisms.
Proof.
If the space X is not purely atomic, we have X (cid:39) [0 , c ] (cid:116) (cid:70)
i ≥ 1 a i for some c > 0 , where [0 , c ] is the continuous part and (cid:70)
i ≥
a
i is the atomic part of the probability measure µ .
Interval [0 , c ] admits at least one nontrivial automorphism, namely the transformation x
(cid:55)→ c − x (leaving the atoms ﬁxed), hence so does X by Lemma 2.2.
In fact, there are inﬁnitely many other automorphisms, which can be obtained by exchanging nonoverlapping subintervals ( a, a + d ) , ( b, b + d ) ⊂
[0 , c ] of the same length.
If there exist two atoms a j , a k in X with equal mass, then a transformation which transposes a j with a k and keeps the rest of X ﬁxed is a nontrivial automorphism.
Probability spaces of images which appear in real life typically have a continuous component which would correspond to continuous variations in object sizes, lighting conditions, etc.
Therefore, they admit some probability space automorphisms.
However, such abstract automorphisms can be highly discontinuous, which would make it questionable if neural networks can learn them.
We would like to show that there are also automorphisms which are smooth, at least locally.
For this, we ﬁrst state the following technical claim.
The proof is provided in appendix A.
Proposition 2.5.
Let µ be a Borel probability measure on R n and f : R n → R
m be a continuous injective function.
Then f : ( R n , B ( R n ) , µ ) → ( R m , B ( R m ) , f ∗ µ ) is an isomorphism of probability spaces, where f ∗ µ denotes the push-forward of measure µ to R m .
Finally, we show the existence of smooth automorphisms under the assumption that our data manifold D ⊂ R m can be generated by embedding R n with standard Gaussian measure into R m as a submanifold.
We write γ n for the standard Gaussian probability measure on the space R n .
Proposition 2.6.
Let Z :=
( R n , B ( R n ) , γ n ) be an n -dimensional standard Gaussian distribution.
Let f : R n → R m be a manifold embedding.
Denote by X the probability space ( R m , B ( R m ) , f ∗ γ n ) .
Then the following assertions hold:
f is an isomorphism of probability spaces when viewed as a map Z → X ; 2.
every rotation T ∈ SO( R n ) is a probability space automorphism and a diffeomorphism of Z .
T induces a probability space automorphism of X which is, additionally, a diffeomorphism when restricted to Im f ⊂ R m .
Proof.
The ﬁrst claim follows directly from Proposition 2.5.
For the second part, it is clear that rotations in SO( R n ) preserve isotropic Gaussian distribution, and the rest follows from Lemma 2.2.
The connection with generative models is clear if we take f to be an invertible generative model such as RealNVP Dinh et al.
or Glow Kingma & Dhariwal (2018).
The assumption of manifold embedding in the proposition can be seen as too limiting in general, and we explain how to ‘bypass’ it in Lemma A.2 for the interested readers.
In conclusion, if we assume that the distributions we are working with could be represented by an invertible generative model, then there exists a rich space of automorphisms.
Given the success of e.g. Glow, this assumption seems to be valid for natural images.
N UMERICAL RESULTS
Since we have established that the existence of automorphisms can negatively impact the results of CycleGAN, we now demonstrate how this can happen by considering a toy case with a known solution and demonstrating that CycleGAN can and does learn a nontrivial automorphism.
The toy experiment which we perform is translation of MNIST dataset to itself.
That is, at training time we pick two minibatches batch A and batch B from MNIST at random and use these as samples from X and Y respectively.
The generator neural network in this case is a convolutional autoencoder with residual blocks, fully connected layer in the bottleneck and no skip connections from encoder to decoder.
We also train a simple CNN for MNIST classiﬁcation in order to classify CycleGAN outputs.
The networks were trained using SGD.
The ‘natural’ transformation in this case is, of course, the identity mapping and we expect the classiﬁcation of the inputs and outputs to stay the same.
But we shall see that this is not the case.
In ﬁg. 2a–ﬁg.
2h we show some examples for the generated fake samples and the reconstruction on test set.
In ﬁg. 3a–ﬁg.
3b we provide the confusion matrices for the A2B and B2A generators respectively.
We use these matrices to understand if e.g. the class of transformed image for A2B translation equals the source class, or if is a random variable independent of the source class, or if we can spot some deterministic permutation of classes.
We have observed that in practice the identity mapping is not learned.
Instead, the network leans towards producing a certain permutation of digits, rather than identity or a random assignment of classes independent of the source label.
One explanation would be as follows.
Suppose that we can perfectly disentangle class and style in latent digit representation Makhzani et al. (2015).
Then any permutation in S 10 , acting on the class part of the latent code, determines a probability space automorphism on the space of digits, which can be learned by a neural network.
Further investigation of confusion matrices reveals that the networks introduce short cycles , e.g., mapping 2 to 6 and vice versa.
We provide additional experiments on BRATS2015 dataset in appendix B, where we show that in the absense of identity loss the pure CycleGAN loss demonstrates noticeable symmetry, while the PSNR is clearly not invariant.
Increasing the weight of the identity loss term reduces the symmetry, but does not necessarily result in a similar PSNR improvement.
src fake reco (a) src fake reco (b) src fake reco (c) src fake reco (d) src fake reco (e) src fake reco (f) src fake reco (g) src fake reco (h)
[Figure]
D ISCUSSION AND FUTURE WORK
We have shown theoretically that under mild assumptions, the kernel of the CycleGAN admits nontrivial symmetries and has a natural structure of a principle homogeneous space.
To show empirically that such symmetries can be learned, we have trained a CycleGAN on the task of translating a domain to itself.
In particular, we show that on the MNIST2MNIST task, in contrast to the expected identity, the CycleGAN learns to permute the digits.
We have therefore effectively shown, that it is not the CycleGAN loss which prevents this from occurring more often, but hypothesize that the network architecture also has major inﬂuence.
We advocate against the usage of CycleGAN when translating between substantially different distributions in critical tasks such as medical imaging, given the theoretical results in Corollary 2.1 which suggest ambiguity of solutions, even in the presence of the identity loss term.
We would like to point out that some work has been done recently extending the CycleGAN.
For example, in Na et al.
the authors argue that many image-to-image translation tasks are ‘multimodal’ in a sense that there are multiple equally plausible outputs for a single input image, therefore, one should explicitly model this uncertainty in the model.
To address this issue, the authors design a network which has two ‘style’ encoders E X : X → Z X , E Y :
Y → Z Y , two discriminators for each domain, two conditional encoders for each direction E XY :
X × Z Y → Z XY , E Y X :
Y × Z X
→ Z Y X and two generators for each direction G XY :
Z XY → Y , G Y X : Z Y X
→ X .
The style encoders serve to extract the ‘style’ of the image, which is present in both domains, e.g., in case of the ‘female-to-male’ task on CelebA dataset the style would correspond to coarsely represented facial features.
The loss term forces the mutual information between the style vector of the translated image and the input style to the conditional encoder to be maximized.
This allows the network to roughly preserve the style in the translation.
While we leave full analysis of this approach for the future work, we expect that such loss would reduce ambiguity in the solution space to those isomorphisms which differ by automorhpishs from the set
[Equation]
leaving the style ﬁxed, since replacing G Y X with ϕ ◦ G Y X and E XY with E XY ◦ ϕ − 1 does not change the loss value for such ϕ .
Therefore, the reduction in uncertainty of our solution depends on capacity of the encoder E X , and, ideally, should be quantiﬁed.
In particular, one might still need to enforce additional problem-speciﬁc features in the encoder E X to guarantee that important image style content is preserved.
A B ACKGROUND
Firstly, we very brieﬂy explain the probability theory language we use in this article, and we refer the reader to (Eisner et al., 2015; Bogachev, 2007) for more details.
Formally, a measurable space ( X, X ) is a pair of a set X and a σ -algebra X of subsets of X .
Given a topological space X with topology U , there exists the smallest σ -algebra B ( X ) , which contains all open sets in U .
This σ -algebra is called
Borel σ -algebra of X and its elements are called Borel sets.
A probability space X =
( X, X , µ ) is a triple of a set X , a sigma algebra X of subsets of X and a probability measure µ deﬁned on the sigma-algebra X .
Given a probability space ( X, X , µ ) , a measurable set A ∈ X is called an atom if µ ( A ) > 0 and for all measurable B ⊂
A such that µ ( B ) < µ ( A ) we have µ ( B ) = 0 .
Given measurable spaces ( X, X ) and ( Y, Y ) , we say that a mapping φ : X → Y is measurable if for any
A ∈ Y we have φ − 1 ( A ) ∈ X .
If X =
( X, X , µ ) and Y =
( Y, Y , ν ) are probability spaces and φ : X → Y is a measurable map, we say that φ is measure-preserving if for all A ∈ Y we have µ ( φ − 1 ( A )) =
ν ( A ) .
An approximation argument easily shows that a measurable transformation φ : X → X is measure-preserving if and only if for all nonnegative measurable functions f on X we have
[Equation]
Given a probability space X , a measurable space ( Y, Y ) and a measurable map φ : X → Y , we deﬁne the push-forward measure φ ∗ µ on Y by setting ( φ ∗ µ )( A ) :
= µ ( φ − 1 ( A )) for all A ∈ Y .
Let ( X, X , µ ) and ( Y, Y , ν ) be probability spaces and f : X → Y be a measure-preserving map.
A measurable map g : Y → X is called an essential inverse of f if f ◦ g = id Y for ν -almost every y ∈ Y and g ◦ f = id X for µ -almost every x ∈ X .
One can show that essential inverse is measure preserving and uniquely deﬁned up to equality almost everywhere.
We say that f is an isomorphism if it admits an essential inverse.
An isomorphism f : X → X is called an automorphism .
Lemma A.1 (Push-forward property for f -divergences) .
Let p, q be distributions on R n and ϕ : R n → R n be a diffeomorphism.
Then for any f -divergence D f we have
[Equation]
[Equation]
Proof.
First of all, change of variables formula for the integral implies that
[Equation]
for [Equation] y
[Equation]
Therefore,
[Equation]
Applying change of variables formula with x = ϕ ( y ) , we get
[Equation]
f
[Equation]
p
[Equation]
x
[Equation]
where the equality in ( ∗ ) uses a general property of Jacobians of smooth invertible maps that ∂ϕ − 1 ∂ x ◦ ϕ
= (cid:16) ∂ϕ∂
y (cid:17) − 1 .
Hence D f
( ϕ ∗ p (cid:107) q ) =
D f ( p (cid:107) ( ϕ − 1 ) ∗ q ) , which completes the proof.
We remind the reader that a Polish space is a separable completely metrizable topological space.
A
Borel probability space is a Polish space endowed with a probability measure µ on its Borel σ -algebra, and we will also say that µ is a Borel probability measure .
The basic examples of Borel probability spaces would be e.g. the spaces [0 , 1] n ⊂ R n with its Borel σ -algebra B ( R n ) , endowed with Lebesgue measure λ n .
A Borel σ -algebra of the space [0 , 1] n endowed with Lebesgue measure λ n can be extended by adding all λ n -measurable sets, leading to the σ -algebra of Lebesgue-measurable sets .
For the proof of Proposition 2.5 we need the following theorem, see Kechris (1995), Theorem 15.1.
Theorem A.1 (Lusin-Souslin theorem) .
Let X, Y be Polish spaces and f : X → Y be continuous.
If A ⊂ X is Borel and f | A is injective, then f ( A ) is Borel.
Proof of Proposition 2.5.
Denote the image f ( R n ) ⊂ R m by Im f .
Then Im f ⊂ R m is a Borel subset, since R n is a countable union of a compact sets and f is continuous.
Furthermore, from
Lusin-Souslin theorem (theorem A.1)
it follows that for every Borel subset A ⊂ R n its image f
( A ) ⊂ R m is Borel as well.
Pick a point x 0 ∈ R n which is not an atom of µ .
We want to deﬁne an almost everywhere inverse ˜ f of f .
Deﬁne a function ˜ f : R m → R n by
[Equation]
Using the remark above it is easy to see that ˜ f is Borel measurable and that ( f ∗ µ )( ˜ f − 1 ( A ))
=
µ ( A ) for every Borel A .
It follows from the deﬁnition that ˜ f ◦ f = id R n and that
[Equation]
Since ( f ∗ µ )(Im f ) = 1 , ˜ f is an almost everywhere inverse to f .
We conclude that f is a probability space isomorphism.
Secondly, we remind the reader of a couple of notions from differential geometry which we use in the text, and we refer the reader to e.g. (Warner, 1983) for more details.
Given a subset X of a manifold
M and a subset Y of a manifold N , a function f : X → Y is said to be smooth if for all
p ∈
X there is a neighborhood U ⊂ M of p and a smooth function g : U → N such that g extends f , i.e., the restrictions agree g | U ∩ X = f | U ∩ X .
f is said to be a diffeomorphism between X and Y if it is bijective, smooth and its inverse is smooth.
Let M and N be smooth manifolds.
A differentiable mapping f : M → N is said to be an immersion if the tangent map d p f : T p M → T f ( p ) N is injective for all p ∈ M .
If, in addition, f is a homeomorphism onto f ( M ) ⊂ N , where f ( M ) carries the subspace topology induced from N , we say that f is an embedding .
If M ⊂ N and the inclusion map ı : M → N is an embedding, we say that M is a submanifold of N .
Thus, the domain of an embedding is diffeomorphic to its image, and the image of an embedding is a submanifold.
We close this section with a small lemma, explaining how one can weaken the embedding assumption for generative models in Proposition 2.6.
Lemma A.2.
Let f : R n → R m be an injective manifold immersion.
Let B R ⊂ R n be an open ball of radius R > 0 in R n and B R be its closure.
Then f : B R → f ( B R ) is a manifold embedding.
Proof.
Since B R is compact and f is continuous, image of every closed subset A ⊆ B R is compact and hence closed.
This shows that f − 1 : f ( B R ) → B R is continuous and thus f : B R → f ( B R ) is a homeomorphism.
Restricting to the open ball B R ⊂ B R , we conclude that f : B R → f ( B R ) is a homemorphism and thus a manifold embedding.
As a consequence, for our example with spherical Gaussian latent vector one can take sufﬁciently large ball of radius R > 0 in the latent space, truncating the latent distribution to ‘sufﬁciently likely’ values.
This ball remains invariant under rotations, thus leading to a differentiable automorphism on the submanifold of ‘sufﬁciently likely’ images.
B BRATS2015 EXPERIMENTS
We present some additional results on the BRATS2015 dataset.
For this experiment Unet-based generators with residual connections were used.
The number of downsampling layers was 4 for both generators, and skip connections were preserved.
We trained all models for 20 epochs with
Adam optimizer and learning rate 0 . 0002 .
We trained 4 models with α id ∈ { 0 . 0 , 10 . 0 , 20 . 0 , 40 . 0 } .
No data augmentation was used so as to avoid creating any additional symmetries.
All images were normalized by dividing by the 95% -percentile, as is common in medical imaging when working with
MR data.
We hypothesize that ﬂipping images horizontally is a distribution symmetry.
We measure the ﬁnal test loss for both the network output ( Loss ) and its ﬂipped version ( Loss (f) ), as well as the PSNR for both translation directions without ( PSNR T1-Fl , PSNR Fl-T1 ) and with horizontal ﬂips ( PSNR
T1-Fl (f) , PSNR Fl-T1 (f) ).
We summarize these results in table 1.
We observe that in the absense of identity loss the pure CycleGAN loss demonstrates noticeable symmetry, while the PSNR is clearly not invariant.
Increasing the weight of the identity loss term reduces the symmetry, but does not always result in a similar PSNR improvement.
We present some samples from the model with α id = 0 in ﬁg. 4a, ﬁg. 4b.
[Table] [Figure]