On Translation and Reconstruction Guarantees of the Cycle-Consistent Generative Adversarial Networks
Abstract
The task of unpaired image-to-image translation has witnessed a revolution with the introduction of the cycle-consistency loss to Generative Adversarial Networks (GANs).
Numerous variants, with Cycle-Consistent Adversarial Network (CycleGAN) at their forefront, have shown remarkable empirical performance.
The involvement of two unalike data spaces and the existence of multiple solution maps between them are some of the facets that make such architectures unique.
In this study, we investigate the statistical properties of such unpaired data translator networks between distinct spaces, bearing the additional responsibility of cycleconsistency.
In a density estimation setup, we derive sharp non-asymptotic bounds on the translation errors under suitably characterized models.
This, in turn, points out sufficient regularity conditions that maps must obey to carry out successful translations.
We further show that cycle-consistency is achieved as a consequence of the data being successfully generated in each space based on observations from the other.
In a first-of-its-kind attempt, we also provide deterministic bounds on the cumulative reconstruction error.
In the process, we establish tolerable upper bounds on the discrepancy responsible for ill-posedness in such networks.
Introduction
The overwhelming number of variants GANs [1] have inspired while catering to its vast application domains is a testament to its versatility.
One such family of progenies having remarkable accolades of its own owes its genesis to the cycle-consistency constraint.
Possibly the most influential one belonging to this group is CycleGAN [2].
It offers an unsupervised image-to-image (I2I) translation framework for unpaired observations hailing from unrelated data spaces.
In terms of the architecture, both DualGAN [3] and DiscoGAN [4] are immediate relatives to CycleGAN.
In the chassis of such networks lie two concurrent adversarial generation processes, commonly termed translations , regularized by a cyclic loss.
This penalization ensures the reconstruction of input data from either space post-translation.
In addition, models such as DTN [5] and UNIT [6] assume the existence of a shared latent space between the domains.
This allows the restructuring of the model without altering the objective.
By stacking multiple translator networks, SCAN [7] promises significant performance improvement, especially for high-resolution images.
Some members of the family ([2], U-GAT-IT[8], [9]) also deploy an additional identity loss to remove tilt-shift in generated images.
We call this broad class of I2I translation machines ‘cycle-consistent networks’.
The constraint of cycle-consistency should be primarily credited for the masterly generative capability of such models, from which tasks like style transfer, object transfiguration [2], and data augmentation [10] benefit immensely.
In this study, we intend to rise above the empirical evidences by providing a statistical backbone to the fact that cycle-consistent networks can simultaneously translate data both ways without losing the capacity to reconstruct.
We call the two maps, operating in opposite directions, Translators .
Both underlying distributions portraying purposeful image data, in the absence of conventional latent laws, call for such transformations to differ from usual generators used in vanilla GANs.
Unlike GANs, there may exist non-unique solution maps bringing about ‘zero’ realized loss in this case [9].
As such, searching for translators that minimize the error is not sufficient.
This fact motivates us to study the desirable regularities of the maps that facilitate the statistical convergence of output measures, which in turn define our notion of consistency .
Theoretically, the concept of ‘cycle-consistency’ is analogous to ‘regeneration’ [11] in case of Variational Autoencoders [12].
In such inverse problems, maps reconstructing the input signal become sensitive to slight perturbations due to noise.
A noisy output from earlier translations contributes to this ambiguity in the inverse generation process, formally known as ‘ill-posedness’ [13].
Theoretical insights regarding the source and admissible error margins of ill-posedness remain absent to date.
Confronted with such challenges, this paper provides a fresh perspective on the theoretical machinery of the cycle-consistent adversarial networks.
Our contributions can be summarized in the following way:
• We show that translators, based on deep ReLU networks, prevent the information provided by input empirical laws from dissipating during generation cycles [Proposition (1) and Remark (3)].
• In Theorem (1) and Corollary (1), we prove that the same translators not only achieve zero generation loss asymptotically but the generated sequence of distributions also converge to the target density almost surely.
• Under Sobolev-smooth input laws, we establish that the uses of L 1 norm and 1 -Wasserstein distance in the cyclic loss are equivalent, attesting to Zhu et al. ’s
[2] observation that the latter does not improve performance [Theorem (3)].
Furthermore, we prove that a network deploying the aforementioned translators achieves cycleconsistency as a consequence of translation consistency in both directions.
[Theorem (3), (5)]
Related work
Playing catch-up to earlier empirical success, theoretical scrutiny of GANs fostered a series of notable works in recent years.
Liu et al.
[14] characterized the objective functions of several GAN architectures (f-GAN [15], WGAN [16], etc.)
as adversarial divergences .
This allowed them to analyze the convergence of generated distributions towards the target law in a unified framework.
Meanwhile, Arora et al.
[17] explored the expressiveness of generator networks and the generalization performance of GANs under the neural net distance .
In a later work, however, we observed Zhang et al.
[18] show improved results over both.
Convergence and related asymptotic properties of the density estimates in a GAN setup can also be found in the parametric approach of [19].
On the other hand, error decomposition of the GAN-objective under both parametric and non-parametric regimes may lead to non-asymptotic concentration bounds.
Several works followed this approach with various smoothness assumptions on the data distributions and the transformations involved [20, 21, 22].
A recent study of the same nature also focused on learning from low-dimensional latent laws using smooth maps [23].
One may also come across several GAN variants inspiring similar pursuits.
Biau et al.
[24] presented a comprehensive study of the convergence and related asymptotic properties of the parametric density estimates in a WGAN setup.
From a non-parametric viewpoint, Haas et al.
[25] derived deterministic upper bounds on the expected WGAN loss, under both conditional and unconditional generation processes.
Lately, a non-asymptotic approach of a similar spirit has been utilized to establish risk bounds on the realized Bidirectional-GAN (BiGAN) error [26].
Cycle-consistent networks, despite marking a triumph in deep generative modeling, have not received such independent attention yet.
This scarcity makes the existing attempts even more meaningful.
Moriakov et al.
[9] proved that multiple solutions to the CycleGAN problem exist, as a consequence of the existence of nontrivial automorphisms in either data space.
Tiao et al.
[27] pointed out that the cycle-consistency loss boils down to an expected posterior log-likelihood, in a Bayesian setup.
On the contrary, the CycleGAN objective can also be recognized as the Unbalanced Gromov-Monge
Divergence (UGMD), when the transformations are assumed to be isometric [28].
However, all the above studies refrain from exploring the statistical guarantees a cycle-consistent translator aims to provide by following its concurrent objectives.
Our current work is a humble attempt to fill this gap.
Preliminaries 3.1 Notations
The two data spaces involved X and Y , equipped with respective distances c and c ′ , are considered to be Polish (i.e., separable and completely metrizable).
A simple characterization of the same might be R d for d ≥ 1 .
Let P ( X ) denote the space of probability measures defined on X .
We refer to the set of measurable functions mapping X to Y as F ( X , Y ) .
The ‘forward’ and ‘backward’ translator maps between the spaces are denoted by F and G respectively.
Observe that, a probabilistic forward translator belongs to F ( X , P ( Y )) .
Similarly, G ∈ F ( Y , P ( X )) .
The two discriminator networks at both ends induce functions D X and D Y , which play the role of critics in the two simultaneous adversarial games.
For non-negative real sequences { a n } n ∈ N and { b n } n ∈ N , the notation a n ≾ b n , or equivalently a n = O ( b n ) , means that there exists a constant C > 0 , such that lim sup n →∞ a n b n ≤ C .
The total variation distance between measures P, Q is represented by ∥ P − Q ∥ TV .
We also denote max { x, y } as x ∨ y .
Let us now introduce some concepts that we frequent in the upcoming discussion.
Definition 1 (Pushforward Measure) .
Let µ ∈ P ( X ) .
For a measurable map f : X → Y , we denote the pushforward measure of µ by f # µ ∈ P ( Y ) , defined as f # µ ( ω ) =
µ ( f − 1 ( ω )) , where ω is a measurable set ⊂ Y .
Definition 2 (Wasserstein Distance) .
For a metric c : X × X → R ≥ 0 and measures P, Q ∈ P ( X ) , the r th Wasserstein Distance between P and Q is defined as
[Equation]
where Γ( P, Q ) =
(cid:8) γ ∈ P ( X × X ) : (cid:82) X γ ( x, y ) dy
=
P, (cid:82) X γ ( x, y ) dx
=
Q (cid:9) is the set of all measure couples between P and Q ; r ∈ [1 , ∞ ) .
Remark 1.
In our analysis, we make extensive use of a particular case of this discrepancy measure, namely when r = 1 .
We also reiterate the fact that W 1 c can be written as W 1 c ( P, Q ) =
sup l ∈ L 1 c (cid:110)(cid:82) X l ( x ) dP ( x ) − (cid:82) X l ( x ) dQ ( x ) (cid:111) , where L 1 c := class of 1 -Lipschitz functions with respect to c
[Remark 6.5 in [29]].
However, we adopt the notation d L 1 c instead to maintain consistency with the other Integral Probability Metrics (IPMs).
3.2 Problem setup
Throughout our discussion, we denote the distributions at both ends by µ ∈ P ( X ) and ν ∈ P ( Y ) respectively.
The adversarial loss that the backward generation process ( µ G ←− ν ) tries to minimize is given by,
[Equation]
The same convention leads to the forward generation ( µ F −→ ν ) loss, L D Y ( ν, µ, F ) .
The string tying these two processes together comes in the form of the cyclic loss.
Based on our notations, it can be written as
[Equation]
where ∥·∥ 1 represents the L 1 norm.
We point out that this specific choice of the norm is based on the recommendation of Zhu et al.
According to them, the usage of an adversarial loss instead does not improve regenerated image quality.
Through the following illustration (Figure 1), we offer the reader a glimpse of our vision of concurrent translations and reconstructions.
A typical CycleGAN [2], or equivalently DiscoGAN [4] formulation, carries out the following optimization task:
[Equation] [Equation]
[Figure]
where L X and L Y are classes of discriminator functions and the maps F , G are sculpted using translator networks.
Also, λ 1 , λ 2 > 0 .
Observe that, (1) can be rewritten as
[Equation]
given that d F ( P, Q ) =
sup f ∈F (cid:8) E P [ f ] − E Q [ f ] (cid:9) .
The only feature that differentiates the DualGAN [3] objective from (2) is the employment of the conventional generation technique based on noises.
Furthermore, in an attempt to preserve the colour composition in images, models such as the extended
CycleGAN [2, 9], U-GAT-IT [8] impose a constraint that penalizes the translators’ tendency to move away from the identity.
It is given by,
[Equation]
Observe that, such regularization is feasible only when the two data distributions are equi-dimensional.
The map F is built with the fundamental motivation of transforming µ into ν .
As such, the discrepancy (cid:13)(cid:13) µ − F # µ (cid:13)(cid:13) 1 should not be minimized beyond the difference between µ and ν .
We provide a detailed discussion on the same in the supplementary material.
Theoretical analysis 4.1 Data distributions
Depicting real images as observations from probability distributions is a completely theoretical construct.
The representation provides practitioners with a refined view of the problem.
Moreover, the transformed objective of density estimation has its benefits.
Perhaps this is the idea that inspired the genesis of ‘Roundtrip’, a CycleGAN progeny [30].
In our study, we consider X ≡ R d and Y ≡ R k ; d, k ∈ N + .
The two dimensions need not be equal in general.
The consequences of the special case of equality will be discussed at a later stage.
Let us now introduce some particular classes of functions that determine the nature of the data distributions under consideration.
We denote by L p ( R d ) the space of p -fold Lebesgue-integrable functions, equipped with the norm ∥ .
∥ p , p ∈ [1 , ∞ ) .
The notation for the space of uniformly continuous functions is C u ( R d ) .
Also, ∥ .
∥ ∞ stands for the uniform norm.
Definition 3 (Sobolev Space [31]) .
Let α =
( α 1 , α 2 , ..., α d ) , α i ∈ N + ∪{ 0 } such that | α | = (cid:80) d i =
α i .
For x ∈ R d , the mixed partial weak differential operator of order | α | is given by D α = ∂ | α | ∂x α 11 ... ∂x αdd .
Based on these notations, for p ∈ [1 , ∞ ) and radius L ∈ R ≥
W m,pL ( R d ) =
(cid:110)
f ∈
L p ( R d ) : D
α f ∈ L p ( R d ) ∀| α | ≤ m : ∥ f ∥ W m,p ≡ ∥ f ∥ p + (cid:88) | α | = m ∥
D α f ∥ p < L (cid:111) defines the L p -Sobolev Space of order m .
Remark 2.
Here, we also mention an extended class of functions, mainly when p = ∞ .
If f : R d → R is differentiable at x , we write D α f = f ( α ) .
We define
W m, ∞
L ( R d ) =
(cid:110) f ∈ C u ( R d ) :
f ( α ) ∈ C u ( R d ) ∀|
α | ≤ m : ∥ f ∥
W m ≡∥ f ∥ ∞ + (cid:88) | α | = m (cid:13)(cid:13)(cid:13) f
( α ) (cid:13)(cid:13)(cid:13) ∞ <
L (cid:111) .
The generalization of W s, ∞ for a non-integer s results in the space of Hölder-Zygmund type functions.
For a detailed exposition of such functions, one may turn to [31].
We consider µ and ν to have corresponding densities p µ and p ν , with respect to Lebesgue measures, in their respective spaces.
The following assumption provides coherence to their characterization.
Assumption 1.
(Regularity of distributions)
There exists m x , m y ∈ N + such that p µ ∈ W m x ,p L (Ω x ) and p ν ∈ W m y ,q L (Ω y ) , where the supports Ω x ⊆ R d and Ω y ⊆
R k are both compact, p, q ∈ [1 , ∞ ) .
Feature extracted image data, in vectorized form, tend to hail from bounded domains in each of its coordinates.
Our characterization of input laws having compact support complements this fact.
Perhaps it is the very reason that motivates [21] and [20] to assume the same, contextually.
4.2 Class of discriminator functions
Functional classes L X , L Y are characterized based on their ability to tell apart real and generated observations.
Some of the notable choices of the same include functions defined over a Reproducing
Kernel Hilbert Space (RKHS) [32], Lipschitz [16], and Sobolev functions [33].
In our work, we concentrate on two families, namely L 1 c and W m, ∞ .
Our first choice is motivated by the heightened generative quality the Wasserstein distance brings along to deep models.
It also offers a pathway to fend off mode collapse and vanishing gradients.
On the other hand, the latter class of critics enables us to study the effect of improved smoothness on translation and regeneration.
So far, we have only discussed the Lagrangian formulation of the optimization problem at hand.
In fact, (2) is the embodiment of the exact Lagrange dual function .
The forthcoming analysis, however, relies on the ‘constrained version’ [Chapter 5 of [34]] given as follows:
[Equation]
where t 1 , t 2 ≥ 0 .
Solutions from (2) turn out to be lower bounds to that derived from (3), a fact that inspires the forthcoming theory.
We say ‘simultaneous successful translations have taken place’ only when the constraints in (3) are met with.
The immediate inquiry that follows involves checking the feasibility of an architecture to achieve cycle-consistency.
As supporting evidence for both phenomena, we produce deterministic upper bounds on the respective errors along with convergence guarantees of distributions.
4.3 Translation guarantees
Let us concentrate on the backward translation ( µ G ←− ν ) first.
Observe that, a realized sample counterpart of the objective turns out to be d L X (ˆ µ n 1 , G # ˆ ν n 2 ) , where ˆ µ n 1 = 1 n 1 (cid:80) n 1 i =
δ X i is the empirical distribution corresponding to µ , based on n 1 ∈ N + i.i.d.
samples { X i } n 1 i
=
Similarly, ˆ ν nstands for the same in case of ν , given n 2 ∈ N + samples.
As a consequence, any backward translator
G ∈ F ( Y , P ( X )) should be recognized as G ( n 1 , n 2 ) .
Non-uniqueness of the members residing in the kernel of CycleGAN loss is a well-known fact [9].
Our goal is to prescribe real architectures that induce maps satisfying the first constraint in the sample version of (3).
Such recommendations rely on the next definition.
Definition 4 (ReLU Neural Network) .
Given L ∈ N + , a L -deep Neural Network (NN) is defined as the collection of maps ϕ : R N 0 −→ R N L +1 , { N i } L +1 i =0 ∈ N + given by
[Equation]
where A i ( y )
=
M i y
+ b i ; M i ∈
R N i +
× N i and b i ∈
R N i +1 , i = 0 , ..., L is an affinity.
The activation σ ( y ) =
y ∨ 0 , y ∈ R .
Under this setup, we call W = ∨ Li
=
1 N i the width of the network.
Denote this collection by Φ( W, L ) N L +1 N.
It is fair to say that ReLU is the most commonly used activation function in modern deep NNs.
It is simple to use and also speeds up training, much to practitioners’ delight.
Moreover, it is superior at dealing with vanishing gradients compared to sigmoid or tanh .
However, we draw inspiration from the remarkable approximation capability that ReLU-based networks offer [35, 36], especially towards smooth functions [37].
Theorem 1.
There exist backward translators ϕ , based on ReLU neural network Φ( W, L ) dk with width W ≥ 7 d + 1 and depth L ≥ 3 , such that whenever n 1 ≤ W − d − 1⌊ W − d − 1 6 d
[Equation]
L 2 ⌋ + 2 , we have
[Equation]
Proof of Theorem 1.
Let us begin by fragmenting the translation error as the following,
[Equation]
Before moving forward, denote the set of discrete probability measures based on at most n ∈ N + points in R d by, P n ( d ) :
= (cid:110) (cid:80) ni
=1 a i δ x i : a i ≥ 0 , (cid:80) ni
=1 a i = 1 , { x i } ni =1 ∈ R d (cid:111) .
The following lemma allows us to show that the first term on the right-hand side of (4) can be made arbitrarily small.
Lemma 1 ([38]) .
Let p be an absolutely continuous univariate distribution and π ∈ P n ( d ) .
There exists ϕ ∈
Φ( W, L ) d 1 with W ≥ 7 d +1 and L ≥ 2 , such that whenever n ≤ W − d − 1⌊
[Equation]
2 ⌋ + d L 1 c ( π, ϕ # p ) ≤ ϵ, given ϵ > 0 .
Observe that, ˆ µ n 1 ∈ P n 1 ( d ) .
Now, choose ϕ ∈ Φ( W, L ) dk such that the first layer deploys an additional linear map A that projects ν to a one-dimensional absolutely continuous distribution first.
This can always be done due to the absolute continuity of ν itself.
For the resultant map ϕ , we notice L ≥ 3 , and the specifications of W, n 1 remain as directed by lemma (1).
As a result, d L 1 c (ˆ µ n 1 , ϕ # ν ) ≤ ϵ for arbitrary ϵ > 0 .
The second term in (4) is the portion of the density estimation error from the base domain that translates on to the target space.
To get control over such a discrepancy we exploit the regularity of the transformation ϕ .
Observe that, the activation function σ ≡ ReLU is 1-Lipschitz.
The transformation carrying signal y from i th layer to the next is of the form A i
( y ) =
M i y
+ b i ;
M i ∈ R N i +
× N i and b i ∈
R N i +1 , i = 0 , ..., L .
The matrix M i can be constructed such that ∥ M i ∥ p = sup ∥ y ∥ p =
∥ M i y ∥
p ≤ k i , for some constant k i > 0 .
For cases p = 2 or ∞ , [39] present exact techniques to ensure ∥ M i ∥ p = 1 .
Under such a framework, A i ’s become k i -Lipschitz transforms.
Since Lipschitz functions are closed under composition, we can expect ϕ to behave similarly, with a constant k ∗ dependent on { k 0 , k 1 , ..., k L } .
However, deep ReLU networks are much more expressive and are capable of approximating a vast array of smooth functions.
Let us denote the class of L G -Lipschitz functions mapping (Ω y , c ′ )
→ (Ω x , c ) as G Lip , where L G > 0 .
The following two results encapsulate our idea precisely.
Lemma 2.
For α,
[Equation]
P
[Equation] [Equation]
α, β ) .
Lemma 3 ([40]) .
Let g ∈ G Lip .
Also, ϕ is the ReLU NN-induced function as given in Lemma (1), having width O ( W ) and depth O ( L ) .
Then
[Equation]
where C 1 > 0 is a constant, dependent on L G , √ k, and diam (Ω y ) .
Using both lemmas, we obtain d L 1 c (ˆ µ n 1 , ϕ # ˆ ν n 2 ) ≤ ϵ + L G d L 1 c ′ ( ν, ˆ ν n 2 ) + O ( C 1 W − 2 k L − 2 k ) .
The sole task remaining is to upper bound the statistical estimation error in the base space.
To that end, by applying Corollary 2.1 of [41] for k ≥ 2 , we get E ν [ d L 1 c ′
( ν, ˆ ν n 2 )] ≤ O (( k 2 n 2 ) − 1 k ) .
This result enables us to formally present what we mean by ‘translation guarantee’.
The next corollary can be seen as an embodiment of the same idea.
Corollary 1 (Translation consistency) .
As min ( n 1 , n 2 ) → ∞ , we have d L 1 c (ˆ µ n 1 , ϕ # ˆ ν n 2 ) a.s. −→ 0 .
In other words, given sufficient information from both the distributions, the backward translation method governed by a map ϕ satisfies the constraint in the sample version of (3).
The corollary is an asymptotic statement that ensures the error eventually shrinks below any given t 1 > 0 .
A crucial observation in this context is that for m ≥ 1 , W m, ∞is a sub-family of bounded Lipschitz functions.
In our case, since the supports of the distributions are taken to be bounded, one may equivalently say
W m, ∞⊂ L 1 c , c ≡ L 1 .
As such, W m, ∞playing the role of the critic should produce results similar to Theorem (1).
From this point onward, all proofs are placed in the supplementary material.
Theorem 2.
For a backward translator ϕ of width W and depth L , as specified in Theorem (1)
[Equation]
where n 1 ≤ W − d −
[Equation]
One might wonder what makes Lipschitz transformations so relevant to this context.
The first rather evident observation is that it restricts any further amplification of the distance between laws posttranslation.
The next reason, a particular consequence of the former, brings us to the concept of
Information preserving transformations (IPT) [11].
A contextual definition of the same is as follows
Definition 5 (IPT [11]) .
A map I ∈ F
( Y , P ( X )) is said to be an Information preserving transformation of degree r ≥ 1 under distance metric d , if there exist constants k 1 , k 2 ≥ 0 , such that
[Equation]
Here, (cid:92) ( I # ν ) n is an empirical counterpart of the translated law
I # ν based on n ∈ N + samples.
As such, IPTs are maps that ensure the error committed while replacing I # ˆ ν n with (cid:92) ( I # ν ) n (information dissipated) remains arbitrarily small, with a high probability.
The next result suggests that Lipschitzregular transforms behave as IPT when the target class of distributions is not too ‘complex’.
Definition 6 (Yatracos family [42]) .
Given a class of functions F : Ω → R , the Yatracos family associated to it is defined as,
[Equation]
Proposition 1 (Information preservation of Lipschitz translators) .
Let the Vapnik–Chervonenkis (VC) dimension of Y ( P ( X )) be finite.
Then for any g ∈ G Lip , there exists a constant C 2 > 0 such that
[Equation]
Remark 3.
Observe that, maps induced by ReLU feed-forward networks can similarly pose as IPT, incurring an additional approximation error of order
O ( W − 2 k L − 2 k )
[lemma (2)].
This near-perfect behaviour makes the choice of ReLU-NNs, as suitable translators, rather inevitable.
However, neural networks based on tanh [43], sigmoid [44], and GroupSort [45] activations have also been shown to approximate Lipschitz functions with high precision.
We feel, a comparative analysis of the activations based on their effectiveness in the face of information dissipation may lead to improved prescriptions.
Remark 4 (Forward translation) .
We stress the fact that so far in our discussion, the data dimensions d, k have no restrictions put on them.
As such, the same arguments hold true for the forward generation process ( µ F −→ ν ) as well.
A forward translator map ψ ∈ Φ( W, L )
kd can be similarly constructed that achieves translation consistency.
In other words, one may easily check that the second constraint in (3) is also satisfied for arbitrary values of t 2 .
Cycle-consistent networks find themselves under the obligation to reconstruct the input signal following their translation.
A major obstacle in the process, however, that often mars the quality of regenerated observations is ill-posedness.
It stems from a translation belonging to the feasible set of solutions that results in noisy output, devoid of sufficient information to aid the reconstruction.
Theoretically, the remedy to ill-posedness lies in the formation of a ‘perfect’ transport map between the measures.
Having IPT (Lipschitz) as a reference, NN-based transports tend to overcome this issue asymptotically
[Corollary (1)].
However, measurable maps, in general, lack such approximation capability.
Our following discussion sheds light on the same.
For this section, let us assume the dimensions of the two data domains to be equal, i.e., d = k .
This occasion, in particular, has interesting consequences.
Given that p µ and p ν have finite variance,
Brenier’s theorem [46] ensures the existence of a unique solution γ
= ( Id × T ) # ν to the Kantotovich
Optimal Transport (OT) problem: W 1 c ≡ L 2 .
In other words, we get hold of a map T ∈ F ( Y , P ( X ))
such that T # ν = µ .
It is clear that any function aspiring to subdue ill-posedness should lie in an ϵ -envelope of T , ϵ > 0 being as small as possible.
The larger the deviation, the more is the extent of degradation in reconstructed image quality.
Drawing inspiration from this fact, [47, 13] deploy
OT-based regularizers to guide the solution map toward T .
However, the regularity properties of T can only be determined under very specific assumptions on the data domains [48, 49].
Moreover, we only have access to an empirical counterpart of the target law in the sample version of the problem.
As a result, approximations of the transport map are likely to be noisy.
Our next result aims at pointing out the tolerable error margin due to ill-posedness in a sample backward translation.
Lemma 4.
For a discriminator class L X , and a backward translator G
[Equation]
where E 1 B x := (cid:13)(cid:13)
ˆ µ n 1 − T
# ν (cid:13)(cid:13) TV (Statistical approximation error in target space),
E 2 :=
B x (cid:13)(cid:13)(cid:13) Γ n 1 − (cid:92) ( G # ν ) n 2 (cid:13)(cid:13)(cid:13) TV = Λ ( n 1 ,n 2 ) ; given Γ n 1 = argmin τ ∈ P ( X ) ∥ τ − ˆ µ n 1 ∥ TV , B x = diam (Ω x ) with respect to the metric c , and E 3 :=
d L X (cid:0) (cid:92) ( G # ν ) n 2 , G # ˆ ν n 2 (cid:1) (Information dissipated).
The quantity Λ ( n 1 ,n 2 ) represents the cost incurred by (cid:92) ( G # ν ) n 2 for partaking in the Scheffe tournament [42] to approach ˆ µ n 1 .
Observe that, it remains an admissible amount of deviation if lim min( n 1 ,n 2 ) →∞ Λ ( n 1 ,n 2 ) ≤ t 1 (3).
The maps, ensuring lim min( n 1 ,n 2 ) →∞ Λ ( n 1 ,n 2 ) = 0 , belong to the set of ‘pure’ solutions [9].
Theoretically, the negative effect of such maps on the regeneration quality would be benign.
We elaborate on the same in Proposition (2).
Also, observe that Lemma (4) re-emphasizes the necessity of a backward translator to be an IPT.
Remark 5 (Mode collapse) .
In real situations, supports of data distributions at both ends are often non-convex.
This is an important feature that makes OT maps ( T ) discontinuous [50].
On the other hand, neural networks lack proficiency in approximating such discontinuous functions.
For multi-modal input laws, an estimated transformation approximating only the continuous branches of the target OT map results in mode collapse during translation [50].
As such, the error associated with mode collapse remains convoluted in E 2 .
Fragmenting the realized estimation loss into finer components to address mode collapse may be taken up as potential future work.
Let us now shift our focus towards the residual task a cycle-consistent I2I translator needs to execute.
4.4 Cycle consistency analysis
The cyclic loss, as given in (2), measures the expected discrepancy between the input data and its reconstructed counterpart.
However, the density estimation approach we follow allows us to reframe the objective as a divergence between distributions, given as
[Equation]
For two distributions P, Q ; ∥ P − Q ∥ 1 = 2 ∥ P − Q ∥ TV = (cid:82) (cid:12)(cid:12) ρ P − ρ Q (cid:12)(cid:12) dλ , given that dPdλ = ρ P and dQdλ = ρ Q .
This formulation provides a stronger notion of the loss.
The first result of this section discovers the relationship between translation and reconstruction.
Lemma 5.
For G ∈ F ( Y , P ( X )) and F ∈ F ( X , P ( Y )) ,
[Equation]
cyc
[Equation]
The extent to which a cycle consistent translator can be inaccurate is determined by its performance in the simultaneous generations.
This is a rather desired outcome.
However, the indication of much intrigue that this result gives is that in case the translations are ‘successful’ in both directions, cycle-consistency can be achieved.
One key feature of the input distributions that become crucial henceforth is their smoothness.
As a consequence of Corollary (1), we infer ϕ # ˆ ν n 2 → µ weakly.
A similarly constructed forward translator ψ may also ensure that ψ # ˆ µ n 1 → ν weakly.
Based on such guarantees, we make the following assumptions about the regularity of the transported laws.
Assumption 2.
The translated distributions ϕ # ν and ψ # µ possess corresponding densities given by p ϕ # ν ∈ W m x ,p ′
L (Ω x ) and p ψ # µ ∈ W m y ,q ′ L (Ω y ) ; p ′ , q ′ ∈ [1 , ∞ ) .
Before presenting upcoming theoretical results, let us introduce a tool that plays a key role henceforth.
Definition 7 (Regularly Invariant Kernels [31]) .
A measurable function K ( x, y ) :
R d × R d −→ R is said to be ‘Regular’, if for N ∈ N
(cid:82) R d sup v ∈ R d (cid:12)(cid:12) K ( v, v − u ) (cid:12)(cid:12) | u | N du < ∞ , 2.
for every v ∈ R d , | α | = 1 , ..., N − 1 we have (cid:82) R d K ( v, v + u ) du = 1 ; (cid:82) R d K ( v, v + u ) u
α du =
If such a kernel also satisfies the ‘Invariance’ property:
[Equation]
for r ≥ 1 , we call it ‘Regularly Invariant’.
We mention that the total variation metric can also be expressed as a transportation distance, the underlying cost function being c ( x, y )
= 1 x ̸
=
y .
However, as Chae et al.
[51] point out, the topologies that the TV and Wasserstein distances generate are hardly comparable.
For Sobolev densities, TV often fails to appreciate the nuances that ‘smoothness’ brings along.
A method of alleviation of such difficulties lies in regular kernels.
Minute deviations between smooth functions can be apprehended in greater detail when convoluted with such kernels.
Inevitably, regularly invariant kernels become the cornerstone of our next result.
The proof, placed in the supplement, highlights its contribution.
Theorem 3.
Given the metric c ≡ L 1 , there exists a constant M > 0 dependent on m x , such that
[Equation]
Remark 6.
This result is a multivariate generalization of Theorem 2.1 in [51]
[Equation]
Note that, a similar conclusion can also be drawn for the loss indicating the difference between the target and generated density in case of forward translation.
That is to say,
[Equation]
where M ′ is a constant depending on m y .
Likewise, any pair of translators ( G, F ) that preserve the smoothness of input densities onto generated ones satisfy Theorem (3) and (5).
The collective evidence from these two results suggest that a sufficient condition for achieving cycle-consistency is the arbitrary closeness between real and translated Sobolev-smooth densities, in both domains, under the 1-Wasserstein metric.
Moreover
[Equation]
we already know that
[Equation]
As such, establishing translation consistency under the critic L 1 c is equivalent to attaining cycle-consistency.
Now, let us focus on the sample version of the cyclic loss, given as L cyc (ˆ µ n 1 , ˆ ν n 2 , F, G ) .
The inability of translation maps to approximate optimal transports up to arbitrarily high accuracy affects cycle-consistency as well.
Noisy outputs from a backward generation process should not ideally recover, even under a ‘perfect’ forward translator.
Meanwhile, a ‘perfectly’ translated forward image will be distorted due to such imperfect backward generators.
If the effects due to the departure of translator maps from their ‘ideal’ benchmarks get multiplied, we may observe severe corruption in reconstruction quality.
Much to our relief, the next result assures that the effects of ill-posedness amplify only as a sum.
Proposition 2.
Denote B y (cid:13)(cid:13)(cid:13) Γ ′ n 2 − (cid:92) ( F # µ ) n 1 (cid:13)(cid:13)(cid:13) TV = Λ ′ ( n 1 ,n 2 ) , given that Γ
[Equation]
argmin τ ∈ P ( Y ) ∥ τ − ˆ ν n 2 ∥ TV , B y =
diam (Ω y ) with respect to the metric c ′ .
Then
[Equation]
where E ∗ 1 := 4 (cid:8)
∥ ˆ µ n 1 − µ ∥ TV + ∥ ˆ ν n 2 − ν ∥ TV (cid:9) (Cumulative statistical approximation error),
E ∗ 2 := 4 (cid:110)(cid:13)(cid:13)(cid:13) (cid:92) ( F # µ ) n 1 − F # ˆ µ n 1 (cid:13)(cid:13)(cid:13) TV + (cid:13)(cid:13)(cid:13) (cid:92) ( G # ν ) n 2 − G # ˆ ν n
[Table]
It is expected of the pair of maps that commit zero translation error (e.g. ( ϕ, ψ ), asymptotically) to belong to the ‘kernel’ of a cycle-consistent network.
In other words, the realized cyclic loss should also lie near zero.
To showcase the idea of reconstruction consistency let us concentrate on the term:
[Equation]
Since the smoothness of underlying distributions is paramount in our analysis, usage of regularly invariant kernel density estimates ( ˜ µ n 1 , ˜ ν n 2 ) instead may lead to improved approximation.
Based on the same set of observations we build ˆ p µ,n 1 ( x ) =
d
[Equation]
, x ∈ Ω x where h ≡ h ( n 1 ) .
Similarly define ˆ p ν,n 2 .
Theorem 4.
For the pair of forward-backward maps ( ψ, ϕ ), as constructed in Theorem (1)
[Equation]
The eventual nullification of the average reconstruction loss is a desirable outcome.
However, the concentration of random empirical losses around such an aggregate bears more significance.
On that note, we present the concluding result that embodies our idea of reconstruction consistency.
Corollary 2 (Regeneration consistency) .
As min ( n 1 , n 2 ) → ∞ , we observe ( ϕ ◦ ψ )
# ˜ µ n 1 → µ and ( ψ ◦ ϕ )
# ˜ ν n 2 → ν , both in total variation.
Remark 7.
While the usage of ‘smoother’ estimates produce faster convergence rates, usual empirical distributions ( ˆ µ n 1 , ˆ ν n 2 ) also lead to an outcome similar to Corollary (2), given that the VC dimensions of both Y ( P ( X )) and Y ( P ( Y )) are finite.
Conclusion and Future Work
This study establishes statistical translation and regeneration guarantees of cycle-consistent networks.
In the process, we recommend precise recipes to build translator maps to achieve such consistency.
To the best of our knowledge, it is the first endeavour of its kind in this context.
We prove that deep
ReLU-based translators, being fine approximators of Lipschitz functions, asymptotically behave like
IPTs.
We theoretically show that for Sobolev-smooth input data, deployment of the 1-Wasserstein distance and L 1 in the cyclic loss are equivalent.
This substantiates the conclusion Zhu et al.
[2] had reached for CycleGAN.
A key highlight of our analysis is the absence of any restrictions on the data dimensions.
We also discuss the ramifications of ill-posedness during translation and the impact it leaves on the regeneration.
The decomposition of the translation and cyclic errors in the process, based on independent sources of variation, is also new in this setting.
Our analytical approach may pave the way for further scrutiny of the many unexplored areas of cycle-consistent networks.
Robustness: Deep generative models, in general, are often found to be vulnerable in the presence of outliers in input data.
Required is a study on the tolerable proportion of extreme values from both domains that cycle-consistent I2I translators can handle.
Future work may also look into the robustness of such networks, given that the translation maps have a disjoint noisy component with thick tails.
Moreover, such networks are found to be prone to self-attacking.
In case the target mapping is many-to-one (e.g. photos to semantic labels), the realized translators tend to hide information as a noisy component in the translated law, imperceptible to discriminators.
Though effective defense mechanisms against self-attack (adversarial training with noise, and using guess discriminators) have been proposed [53], deterministic bounds on the permissible departure of maps from their theoretical references remain absent.
Training: Another aspect that lies beyond the scope of this article is the training process.
In our nonparametric approach, we do not recognize the set of candidate distributions to be exactly characterized by an underlying parameter space (say, Θ ).
On the other hand, training can be viewed as the process of finding out a suitable parameter value (say, ˆ θ ∈ Θ ) such that the corresponding density estimate optimizes the loss.
This is crucial since, during training, the parameters of the translator networks are responsible for shaping up this parameter space, and hence the optimum.
In this regime, the optimization trajectory can accordingly be viewed as the stochastic propagation of ˆ θ (as a function of the input sample size, and iterations or time) towards a stable value.
Hence, we feel a parametric analysis with a spirit similar to ours can serve the questions rooted in training and related optimization justice.