O PTIMAL T RANSPORT , C YCLE GAN, AND P ENALIZED LS FOR U NSUPERVISED L EARNING IN I NVERSE P ROBLEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
The penalized least squares (PLS) is a classic method to inverse problems, where a regularization term is added to stabilize the solution.
Optimal transport (OT) is another mathematical framework for computer vision tasks that provides means to transport one measure to another at minimal cost.
The cycle-consistent generative adversarial network (cycleGAN) is a recent extension of GAN to learn target distributions with less mode collapsing behavior.
Although similar in that no supervised training is required, the algorithms look different, so the mathematical relationship between these approaches is not clear.
In this article, we provide an important advance to unveil the missing link.
Speciﬁcally, we reveal that a cycleGAN architecture can be derived as a dual formulation of the OT problem, if the consistency constraint of PLS is enforced as a regularization term of the OT problem.
This suggests that cycleGAN can be considered stochastic generalization of classical PLS approaches.
Our derivation is so general that various types of cycleGAN architectures can be easily derived by merely changing the transport cost.
As proofs of concept, this paper provides a novel cycleGAN architecture for unsupervised learning in accelerated magnetic resonance imaging (MRI) and deconvolution microscopy problems, which conﬁrm the efﬁcacy and the ﬂexibility of the theory.
I NTRODUCTION
Inverse problems are ubiquitous in computer vision, biomedical imaging, scientiﬁc discovery, etc.
In inverse problems, a noisy measurement y ∈ Y from an unobserved image x ∈ X is modeled by
[Equation]
(1) where w is the measurement noise, and H : X (cid:55)→ Y is the measurement operator.
In inverse problems originating from physics, the measurement operator is usually represented by an integral equation:
[Equation]
where h ( r , r (cid:48) ; x ) is an integral kernel.
Then, the inverse problem is formulated as an estimation problem of the unknown x from the measurement y .
It is well known that inverse problems are ill-posed.
A classical strategy to mitigate the ill-posedness is the penalized least squares (PLS) approach:
[Equation]
for q ≥ 1 , where R ( x ) is a regularization (or penalty) function ( l 1 , total variation (TV), etc.) (Chaudhuri et al., 2014; Sarder & Nehorai, 2006; McNally et al., 1999).
In some inverse problems, the measurement operator H is not well deﬁned, so both the unknown operator H and the image x should be estimated.
Recently, deep learning approaches with supervised training have become the mainstream approaches for inverse problems because of their excellent and ultra-fast reconstruction performance.
For example, in low-dose x-ray computed tomography (CT) denoising problems, a convolutional neural network (CNN) is trained to learn the relationship between the noisy image y and the matched noiseless (or high-dose) label images x (Kang et al., 2017).
In the context of (3), the supervised neural network can be understood as directly learning the operation ˆ x = arg min x c ( x ; y ) .
Unfortunately, in many applications, matched label data are not available.
Therefore, unsupervised training without matched reference data has become an important research topic.
Recently, the generative adversarial network (GAN) has attracted signiﬁcant attention in the machine learning community by providing a way to generate target data distribution from random distribution (Goodfellow et al., 2014).
In particular, Arjovsky et al. (2017) proposed the so-called Wasserstein
GAN (W-GAN), which is closely related to the mathematical theory of optimal transport (OT) (Villani, 2008; Peyr´e et al., 2019).
In OT, for two given probability measures supported on the X and Y spaces, one pays a cost for transporting one measure to another.
Then, the minimization of the average transportation cost provides an unsupervised way of learning the transport map between the two measures.
Unfortunately, these GAN approaches often generate artiﬁcial features due to mode collapsing, so cycle-consistent GAN (cycleGAN) (Zhu et al., 2017), which imposes one-toone correspondence, has been extensively investigated (Kang et al., 2019; Lu et al., 2017).
Recently, cycleGAN was generalized to bridge two different domains by generating a continuous sequence of intermediate domains ﬂowing from one domain to the other (Gong et al., 2019).
Although classical PLS, OT, and cycleGAN share the commonality of unsupervised learning which does not require matched training data, there is no mathematical theory to systematically link these seemingly different approaches.
Therefore, one of the main contributions of this paper is to unveil the missing link between these methods.
In particular, we reveal that a cycleGAN architecture can be derived as a dual formulation of the OT problem when the consistency term in PLS is enforced as a regularization term for the OT problem.
Moreover, this framework is so general that various cycleGAN architectures can be easily obtained by changing the PLS cost.
Accordingly, our framework provides a principled way of deriving unsupervised neural networks for various inverse problems. As proofs of concept, we provide novel cycleGAN architectures for unsupervised learning in two physical inverse problems: accelerated MRI and deconvolution microscopy.
In these applications, we show that only one CNN generator is required, which signiﬁcantly reduces the training complexity.
The experimental results conﬁrm that the proposed unsupervised learning approaches can successfully provide accurate inversion results without any matched reference. All the proofs of the main theoretical results can be found in the Appendix.
R ELATED W ORKS 2. O PTIMAL T RANSPORT (OT)
OT compares two measures in a Lagrangian framework (Villani, 2008; Peyr ´ e et al., 2019).
Formally, we say that T : X (cid:55)→ Y transports µ ∈ P ( X ) to ν ∈ P ( Y ) , if ν
[Equation]
=
[Equation]
(cid:1) , for all ν -measurable sets B, (4) which is often simply represented by ν = T # µ, where T # is often called the push-forward operator.
Monge’s original OT problem (Villani, 2008; Peyr´e et al., 2019) is then to ﬁnd a transport map T that transports µ to ν at the minimum total transportation cost:
[Equation]
However, this is usually computationally expensive due to the nature of combinatorial assignment.
Kantorovich relaxed the assumption to consider probabilistic transport that allows mass splitting from a source toward several targets (Villani, 2008; Peyr´e et al., 2019).
Speciﬁcally, Kantorovich introduced a joint measure π ∈ P ( X × Y ) and the associated cost c ( x, y ) , x ∈ X , y ∈ Y such that the original problem can be relaxed as
[Equation]
to
[Equation]
for all measurable sets A ∈ X and B ∈ Y .
Here, the last two constraints come from the observation that the total amount of mass removed from any measurable set has to equal the marginals (Villani, 2008; Peyr´e et al., 2019).
Another important advantage of Kantorovich formulation is the dual formulation as stated in the following theorem:
Theorem 1 (Kantorovich duality theorem) .
(Villani, 2008, Theorem 5.10, p.57-p.59) Let ( X , µ ) and ( Y , ν ) be two Polish probability spaces (separable complete metric space) and let c : X × Y → R be a continuous cost function, such that | c ( x, y ) | ≤ c X ( x ) + c Y
( y )
for some c X ∈ L 1 ( µ ) and c Y ∈ L 1 ( ν ) , where L 1 ( µ ) denotes the set of 1-Lipschitz functions with the measure µ .
Then, there is duality:
[Equation]
and the above maximum is taken over the so-called Kantorovich potential ϕ whose c-transform ϕ c ( y ) =
sup x ( c ( x, y ) − ϕ
( x )) is properly deﬁned.
2. PLS WITH D EEP L EARNING P RIOR
Recently, PLS frameworks using a deep learning prior have been extensively studied (Zhang et al., 2017; Aggarwal et al., 2018)
thanks to their similarities to the classical regularization theory.
The main idea of these approaches is to utilize a pre-trained neural network to stabilize the inverse solution.
For example, in model based deep learning architecture (MoDL) (Aggarwal et al., 2018), the problem can be formulated as
[Equation]
(6) for some regularization parameter λ > 0 , where Q Θ ( x ) is a pre-trained CNN with the network parameter Θ and the input x .
This problem is usually solved in an alternating minimization framework: [Equation] Another type of inversion approach using a deep learning prior is the so-called deep image prior (DIP) (Ulyanov et al., 2018).
Rather than using an explicit prior, the deep neural network architecture itself is used as a regularization by restricting the solution space:
[Equation]
where z is a random vector.
Then, the ﬁnal solution becomes x = Q Θ ∗ ( z ) with Θ ∗ being the estimated network parameters.
Similarly, generative approaches (Van Veen et al., 2018; Bora et al., 2017; Wu et al., 2019) either estimate the random variable z from (8) by ﬁxing Θ or attempt to estimate both the random z and the network weight Θ .
M AIN C ONTRIBUTIONS
One of the basic assumptions in the aforementioned PLS formulation with a deep learning prior is that the measurement y is ﬁxed and one is interested in ﬁnding the unknown x .
In this paper, we relax the assumption to consider situations where both of them are random samples from joint probability measure π ( x, y ) .
Speciﬁcally, we propose a new PLS cost function with a novel deep learning prior as follows:
[Equation]
with p, q ≥ 1 , where the ﬁrst two terms before the semi-colon are the random vectors.
Here, the regularization parameter λ takes care of the dimensional difference of x and y , with λ = 1 for the same dimension, but for simplicity, in the rest of the paper, we assume λ = 1 .
The new PLS cost in (9) has very unique properties.
{+Suppose that the global minimum with c ( x, y ; Θ , H ) = 
+}
exists.
The corresponding global minimizer should then satisfy
[Equation] If this holds for all y , then G Θ is an inverse operator of H . In practice, due to the {+limited capacity of the neural network, the global minimum with c ( x, y ; Θ , H ) = 
+}
may never be achieved. Even in this case, thanks to the symmetric form of PLS, the cost in (9) has an important implication: the neural network G Θ is now estimated by enforcing the consistency y = H x as a regularization term.
This implies that the roles of the consistency and regularization terms in the PLS can be interchanged in our formulation.
Another important advantage is that the unknown image x can be obtained as an output of the feedforward neural network G Θ ( y ) for the given measurement y .
This makes the inversion procedure much simpler.
One may say that the PLS cost function for the DIP in (8) can produce a different variation of unsupervised learning.
However, the resulting formulation is not a feed-forward neural network.
In fact, to obtain the unknown image x , an additional optimization step is required even after the neural network is trained.
See Appendix C for more details. Note that Θ , H are the network and measurement system parameters that should be estimated.
These parameters can be found by minimizing the average transport cost for all combinations of x ∈ X and y ∈ Y with respect to the joint measure π ( x, y ) :
[Equation]
where the minimum is taken over all joint distributions whose marginal distributions with respect to
X and Y are µ and ν , respectively.
Below, we show that the average transportation cost in (11) has an interesting decomposition, which leads to the cycleGAN architecture.
Lemma 1.
If the mapping G Θ : Y (cid:55)→ X is single-valued, then the average transportation cost T (Θ , H ) in (11) can be decomposed as
[Equation]
where
[Equation]
and Γ( G Θ ) , Γ( H ) denote the graphs of G Θ , H , respectively:
[Equation]
Note the nature of the sets Γ( G Θ ) and Γ( H ) and (cid:96) cycle .
Since both Γ( G Θ ) and Γ( H ) are graphs of functions, they are low-dimensional manifolds embedded in the ambient space X ×Y .
In this sense, their relative measures become zero and (cid:96) cycle seems to be a negligible part of the whole T .
This term, however, can contribute a signiﬁcant portion of the total loss T depending on the existence of the singularity in the distribution. We defer the discussion to after the following proposition, which explains why we name the ﬁrst term (cid:96) cycle .
Proposition 1.
[Equation]
where π ∗ denotes the optimum measure and ρ and σ are measurable functions such that 0 ≤ ρ ( x ) ≤ 1 µ -a.e.
, and 0 ≤ σ ( y ) ≤ 1 ν -a.e.
The remaining term in (12), which corresponds to the GAN term, can be obtained as follows:
Proposition 2.
For p = q = 1 , the cost (cid:96) OT (cid:48)
(Θ , H ) in (14) can be computed as
[Equation]
where ϕ and ψ are 1-Lipschitz Kantorovich potential functions.
The proof of Propositions 1 and 2 can be found in Appendix B. The proof idea comes from the computation of c ( x, y ; Θ , H ) on subsets Γ( G Θ ) , Γ( H ) , and Γ( G Θ ) ∩ Γ( H ) by considering the singularity of the optimum measure π ∗ on these sets. In this regard, it is important to know how singularity of π ∗ is distributed on these manifolds. If the targets are perfectly matched as shown in
Fig.
1(a), σ ( y ) :
= π ∗ ( { x = G Θ ( y ) }| y )
= 1; ν -a.e so that π ∗ ( ·| y ) is distributed like a point mass of magnitude σ ( y ) on { x = G Θ ( y ) } .
This also leads to ρ ( x ) :
= π ∗ ( { y = H x }| x ) = 1; µ -a.e.
In this case, (cid:96) OT (cid:48)
= {+
+}
and G Θ ( y ) is a perfect inverse to compute x . On the other hand, if the joint distribution π ( x, y ) is absolutely continuous as shown in Fig. 1(b), then ρ ( x ) = 0; µ -a.e.
and σ ( y ) = 0; ν -a.e.
In this case, (cid:96) cycle = {+
+}
and the contribution of the cycle consistency term can be neglected.
The most interesting case arises when the joint measure π ( x, y ) has signiﬁcant singularities on the manifold Γ( G Θ ) ∪ Γ( H ) so that 0 < σ ( y ) =
π ∗ ( { x = G Θ ( y ) }| y ) < 1 (see Fig. 1(c)).
In fact, unsupervised learning applies to such situations where once the mappings are found, a signiﬁcant portion of the data in X and Y can be paired with the mappings, even though unpaired data remain.
This is when the cycle consistency terms plays an important role in unsupervised learning.
Then, G Θ ( y ) is a desirable approximation of x with high probability.
Now, the ﬁnal step is implementing the Kantorovich potential using CNNs with parameters Φ and Ξ , i.e. ϕ := ϕ
Φ and ψ := ψ Ξ .
By collecting all terms in the transportation cost T (Θ , H ) , the dual formulation results in a cycleGAN cost function:
[Equation]
) where
[Equation]
where (cid:96) cycle (Θ , H ) denotes the cycle-consistency loss in (15) and (cid:96) GAN (Θ , h ; Φ , Ξ) is the GAN loss given by:
[Equation]
Here, the Kantorovich 1-Lipschitz potentials ϕ := ϕ Φ and ψ := ψ Ξ correspond to the W-GAN discriminators.
Speciﬁcally, ϕ Φ tries to ﬁnd the difference between the true image x and the generated image G Θ ( y ) , whereas ψ :=
ψ Ξ attempts to ﬁnd the fake measurement data that are generated by the synthetic measurement procedure H x .
Furthermore, if ρ and σ are constant almost everywhere, they become simple scaling factors for the cycle-consistency terms:
[Equation]
Note that our formulation using (19) with (20) and (21) has many unique features compared to the standard cycleGAN (Zhu et al., 2017). • The standard cycleGAN (Zhu et al., 2017) usually requires two generators in the form of deep neural networks. On the other hand, our formulation often requires a single generator using a deep neural network, if the other generator is given by the forward model H x . This signiﬁcantly reduces the number of weights, which makes the training much more stable.
This can be viewed as the consistency term from the forward model working as a regularization term for OT.
• When the solution of the dual problem achieves the global minimum with diminishing cost, then the dual solution x = G Θ ( y ) is equivalent to the primal solution (10), validating the dual approach.
This is not the case in the conventional cycleGAN with two deep generators.
• While we only consider p = q = 1 due to the simple c -transform ϕ c ( x ) = − ϕ
( x ) for 1-Lipschitz ϕ , the use of the general PLS cost would be interesting, and it may lead to an interesting variation of the cycleGAN architecture.
This could be done using a regularized version of OT (Peyr´e et al., 2019). • The parameters ρ ( x ) and σ ( y ) in (1) and (20) originate from the singularity of the optimal joint measure. However, the estimation of ρ ( x ) and σ ( y ) is not feasible in practice, so the constant hyperparameters in (21) are usually used, and the hyper parameters should be selected by trial and error, which is the same as in the standard cycleGAN. Nevertheless, the probability interpretation may lead to a more simpliﬁed search for the hyper-parameters.
A PPLICATIONS TO I NVERSE P ROBLEMS 4. A CCELERATED MRI
In accelerated magnetic resonance imaging (MRI), the goal is to recover high-quality MR images from sparsely sampled k -space data to reduce the acquisition time.
This problem has been extensively studied using compressed sensing (Lustig et al., 2007), but recently, deep learning approaches have been the main research interest due to their excellent performance and signiﬁcantly reduced run-time complexity (Hammernik et al., 2018; Han & Ye, 2019).
A standard method for neural network training for accelerated MRI is based on supervised learning, where the MR images from fully sampled k -space data are used as references and subsampled k -space data are used as the input for the neural network.
Unfortunately, in accelerated MRI, high-resolution fully sampled k -space data are very difﬁcult to acquire due to the long scan time.
Therefore, the need for unsupervised learning without matched reference data is increasing.
In accelerated MRI, the forward measurement model can be described as
[Equation]
where F is the 2-D Fourier transform and P Ω is the projection to Ω that denotes k -space sampling indices.
To implement every step of the algorithm as image domain processing, (22) can be converted to the image domain forward model by applying the inverse Fourier transform:
[Equation]
This results in the following cost function for the PLS formulation:
[Equation]
where we set λ = 1 in (9) since the dimensions of x and y are the same. Usually, the sampling mask Ω is known so that the forward mapping for the inverse problem is deterministic.
The schematic diagram of the associated cycleGAN architecture is illustrated in Fig.
2, whose generator and discriminator architectures are shown in Fig.
6 in Appendix D. Note that we just need a [Table]
single generator and discriminator, as the mapping from the clean to aliased images is deterministic for a given sampling pattern.
As for the loss function, we use (19) with the σ = ρ = 0 .
5 for the cycle consistency term in (21).
For GAN implementation, we use the W-GAN in (20) with the Lipschitz penalty loss (Gulrajani et al., 2017) to ensure that the Kantorovich potential becomes 1-Lipschitz.
The detailed description of the training procedure is provided in Appendix D. The reconstruction results in Fig.
3 and quantitative comparison results for all test sets in Table 1 clearly show that the proposed unsupervised learning method using dedicated cycleGAN successfully recovered ﬁne details without matched references.
Moreover, the performance is even better than that of the standard cycleGAN and is comparable with that of supervised learning.
4. D ECONVOLUTION M ICROSCOPY
Deconvolution microscopy is extensively used to improve the resolution of wideﬁeld ﬂuorescent microscopy.
However, classical deconvolution approaches require the measurement or estimation of the point spread function (PSF) and are usually computationally expensive.
Recently, CNN approaches have been extensively studied as fast and high-performance alternatives.
Unfortunately, the CNN approaches usually require matched high-resolution images for supervised training.
Therefore, we are interested in developing unsupervised deep neural networks for deconvolution microscopy.
Mathematically, a blurred measurement can be described as
[Equation]
where h is the PSF.
Here, we consider the blind deconvolution problem where both the unknown PSF h and the image x should be estimated.
This results in the following cost function for the PLS formulation:
[Equation]
The corresponding cycleGAN architecture is illustrated in Fig.
In contrast to the conventional cycleGAN approaches that require two generators, the proposed cycleGAN approach needs only a single generator, and the blur image can be generated using a linear convolution layer corresponding to the PSF, which signiﬁcantly improves the robustness of network training. However, we still need two discriminators, since both linear and deep generators should be estimated.
The detailed descriptions of the network architecture and training are given in Appendix D.
Fig.
5 shows lateral views of the deconvolution results of microtubule samples by various methods.
Here, input images are degraded by blur and noise.
The supervised learning and the standard cycleGAN with two generators showed better contrast and removed blur; however, the structural continuity was not preserved.
On the other hand, in our cycleGAN approach, blur and noise were successfully removed, and the continuity of the structure was preserved.
The quantitative comparison with known ground-truth data is provided in Appendix D, which clearly conﬁrms the superiority of the proposed method,
C ONCLUSIONS
In this paper, we presented a general design principle for a cycleGAN architecture for various inverse problems.
Speciﬁcally, the proposed architecture was derived as a dual formulation of an OT problem that enforces the data consistency of PLS as a regularization term for the OT problem.
As proofs of concept, we designed novel cycleGAN architectures for accelerated MRI and deconvolution microscopy examples, providing accurate reconstruction results without any matched reference data.
Given the generality of our design principle, we believe that our method can be an important platform for unsupervised learning for inverse problems.
A A DDITIONAL T HEOREMS AND L EMMA
Theorem 2 (Optimality is inherited by restriction) .
(Villani, 2008, Theorem 4.6, p.46)
Under the same assumption of Kantorovich duality theorem, let π ∗ be an optimal transport plan and ˜ π be a nonnegative measure on X × Y
such that ˜ π ≤ π ∗ and ˜ π
[ X × Y ] > 0 .
Let π (cid:48) :
=
˜
π ˜ π [ X×Y ] , µ (cid:48) , ν (cid:48) be the maginals of π (cid:48) .
Then optimality is inherited to its restriction, i e., π (cid:48) is an optimal transference plan between µ (cid:48) and ν (cid:48) .
Theorem 3 (Restriction for the Kantorovich duality) .
(Villani, 2008, Theorem 5.19, p.75-p.76)
Under the same assumption of Kantorovich duality theorem let π ∗ be an optimal transport plan and ϕ ∗ be an optimal potential that attain the maximum and ˜
π be a measure on X × Y , such that 0 ≤ ˜ π ≤ π ∗ and Z := ˜
π
[ X × Y ] > 0 .
Let π (cid:48)
=
˜ π/Z and µ (cid:48) , ν (cid:48) be the marginals of π (cid:48) .
Then there exists ϕ (cid:48) such that ϕ (cid:48) =
ϕ ∗ on proj X ( supp ( π ∗ )) , and ϕ (cid:48) solves the dual Kantorovich problem between ( X , µ (cid:48) ) and ( Y , ν (cid:48) ) .
We are now ready to provide our duality result for the restricted measure.
Lemma 2.
Let
[Equation]
Then, we can replace (cid:96) OT (cid:48) with (cid:96) OT in the sense that both of them share the same optimal Kantorovich potential.
More speciﬁcally, if ϕ ∗ is the optimal Kantorovich potential, which satisﬁes
[Equation]
then
[Equation]
Proof.
This is a corollary of Theorem 2 and 3 by taking ˜ π
= π ∗ | X×Y\ (Γ( G Θ ) ∪ Γ( H )) .
Then, ˜ π is nonnegative measure on X × Y .
Use the same notation for Z, µ (cid:48) , ν (cid:48) in Theorem 2 and 3. When
Z = 0 , (cid:96) OT (cid:48)
= 0 , and µ (cid:48)
= 0 , ν (cid:48)
= 0 , so the statement trivially holds.
Suppose Z > 0 .
Then by Theorem 3, there exists ϕ (cid:48) such that ϕ (cid:48) =
ϕ ∗ on proj X ( supp ( π ∗ )) , and ϕ (cid:48) solves the dual Kantorovich problem between ( X , µ (cid:48) ) and ( Y , ν (cid:48) ) . [Equation] The remaining part of the proof is to show the relationship that µ (cid:48)
= 1 − ρZ µ and ν (cid:48)
= 1 − σZ ν .
From the proof of Proposition 1, the marginals of π ∗ | Γ( G Θ ) ∪ Γ( H ) are ρµ and σν . Therefore, the marginals of ˜ π = π ∗ − π ∗ | Γ( G Θ ) ∪ Γ( H ) are µ − ρµ and ν − σν so that Zµ (cid:48) = (1 − ρ ) µ and Zν (cid:48)
= (1 − σ ) ν .
[Equation]
(cid:48)
[Equation]
B P ROOF OF M AIN R ESULTS B. P ROOF OF L EMMA
Proof.
From Theorem 2, we extract information about two separated integrals.
For some optimal transportation plan π ∗ , the restrictions π ∗ | Γ( G Θ ) ∪ Γ( H ) and π ∗ | X×Y\
(Γ( G Θ ) ∪ Γ( H )) acquire optimality.
Therefore,
[Equation]
This concludes the proof.
B. P ROOF OF P ROPOSITION
Proof.
By virtue of the deﬁnitions of the cost function and the sets Γ( G Θ ) and Γ( H ) , we have
[Equation]
(cid:107) x − G Θ
( H x ) (cid:107) p on Γ( H ) , c ( x, y ; Θ , H )
=on Γ( G Θ )
∩ Γ( H ) .
Hence,
[Equation]
Furthermore, we use disintegration theorem (Simmons, 2012) to split the joint measure π ∗ as follows:
[Equation]
where π ∗ ( Y| x ) = 1 µ -a
[Equation]
e. and π ∗ ( X| y )
= 1 ν -a.e.
Then, we have (cid:90) Γ
[Equation]
H
[Equation]
(cid:107)
[Equation]
where 1 S denotes the indicator function for the set S and ρ ( x ) = (cid:82) 1 y
= H x π ∗ ( dy | x ) .
In a similar fashion, with σ ( y ) = (cid:82) 1 x = G Θ ( y ) π ∗ ( dx | y ) , we have
[Equation]
This concludes the proof.
B. P ROOF OF P ROPOSITION
Proof.
This is a just simple corollary of the original Kantorovich’s duality formulation proof and the classical results of optimal transport on the restricted measure.
We start with the ﬁrst term of l OT (Θ , H ) in (14).
Speciﬁcally, we have
[Equation]
where µ (cid:48) and ν (cid:48) are marginals of the restriction of optimal transportation plan on the restricted set
X × Y \ (Γ( G Θ ) ∪ Γ( H )) .
Now, using Lemma 2 in Appendix, we have
[Equation]
for the non-restricted marginals µ and ν , where for the last equality we use ϕ c = − ϕ for p =when ϕ is 1-Lipschitz (Villani, 2008).
Using the same technique, the second term in (14) can be computed as
[Equation]
By collecting terms, we conclude the proof.
C O THER V ARIANTS OF U NSUPERVISED L EARNING
Here, we derive the dual optimal transport formulation when the PLS cost with deep learning prior as in (8) is used as the transportation cost for the OT problem.
Speciﬁcally, the transportation cost of the OT problem with respect to the DIP cost in (8) becomes
[Equation]
where we consider y and z as random variable and we use non-squared norm for simplicity.
Then, the corresponding OT problem can be dualized as follows:
[Equation]
Although this is a nice way of pretraining deep image prior model using unmatched training data set, the ﬁnal image estimate still comes from the following optimization problem:
[Equation]
where Θ ∗ is the estimated network parameters from previous training step.
This is equivalent to deep generator model (Bora et al., 2017).
Therefore, this is not useful in designing a feed-forward neural network in an unsupervised manner.
We use dataset for fastMRI challenge (Zbontar et al., 2018) for our experiments.
This dataset is composed of MR images of knees.
We extracted 3500 MR images from fastMRI single coil validation set.
Then, 3000 slices are used for training/evaluation, and 500 slices are used for test.
These MR images are fully sampled images, so we make undersampled images by a randomly subsampling k-space lines.
The acceleration factor is four, and autocalibration signal (ACS) region contains 4% of k-space lines.
Each slice is normalized by standard deviation of the magnitude of each slice.
To handle complex values of data, we concatenate real and imaginary values along the channel dimension.
Each slice has different size, so we use only single batch.
The images are center-cropped to the size of 320 × 320 , and then the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) values are calculated
[Figure] We use U-Net generator to reconstruct fully sampled MR images from undersampled MR images as shown in Fig. 6(a).
Our generator consists of 3 × 3 convolution, Instance normalization, and leaky
ReLU operation.
Also, there are skip-connection and pooling layers.
At the last convolution layer, we do not use any operation.
Our discriminator is same as the discriminator of original CycleGAN.
We use PatchGAN (Isola et al., 2017), so the discriminator classiﬁes inputs at patch scales.
The discriminator also consists of convolution layer, instance normalization, and leaky ReLU operation as shown in Fig. 6(b).
We use Adam optimizer to train our network, with momentum parameters β 1 = 0 .
5 , β 2 = 0 .
9 , and learning rate of 0.0001.
The discriminator is updated 5 times for every generator updates.
We use batch size of 1, and trained our network during 100 epochs.
Our code was implemented by TensorFlow.
D. D ECONVOLUTION MICROSCOPY
The network architecture of the high resolution image generator G Θ from the low-resolution image is a modiﬁed 3D-Unet (C¸ic¸ek et al., 2016) as shown in Fig 7.
Our U-net structure consists of contracting and expanding paths.
The contracting path consists of the repetition of the following blocks: 3D conv- Instance Normalization (Ulyanov et al., 2016)- ReLU.
Here, the generator has symmetric conﬁguration so that both encoder and decoder have the same number of layers, i.e. κ = 7 .
Throughout the network, the convolutional kernel dimension is 3 × 3 × 3 .
There exists a pooling layer and skipped connection for every other convolution operations.
To enhance the image contrast, we add additional sigmoid layer at the end of U-Net.
On the other hand, the low-resolution image generator from high resolution input is based on a single 3D convolution layer that models a 3D blurring kernel.
The size of the 3D PSF modeling layer is chosen depending on the problem set by considering their approximate PSF sizes.
In this paper, the size of the 3D PSF layer is set to 20 × 20 × 20 .
As for the discriminators, we follow the original CycleGAN that uses multi-PatchGANs (mPGANs) (Isola et al., 2017), where each discriminator has input patches with different sizes used.
As shown in Fig 8, it consist of three independent discriminators.
Each discriminator takes patches at different sizes: original, and half, and quarter size patches. D.2. R EAL E XPERIMENT
A total 18 epi-ﬂuorescent (EPF) microscopy images of tubulin with a size of 512 × 512 × 30 were used for training, and one for validation.
As for unmatched sharp image volume, we used deblurred image generated by utilizing a commercial software AutoQuant X3 (Media Cybernetics, Rockville).
The EPF volume depth was increased to 64 using the reﬂection boundary condition.
Due to GPU memory limitations, the EPF volume was split into 64 × 64 × 64 size patches.
For data augmentation, rotation, ﬂip, translation, and scale were imposed on the input patches.
We normalized the patches and set them to [0,1].
Adam optimizer (Kingma & Ba, 2014) was also used for training.
The learning rate was initially set to 0.0001, which is decreased linearly after 40 epoch, and the total number of epoch was 200 epoch.
For the optimizer, we used only a single batch. [Table]
D.2. S IMULATION S TUDY
For simulation studies with the ground-truth data, we used synthetic microtubule network data set (Sage et al., 2017) to train and validate our model.
Speciﬁcally, from the ground-truth high resolution synthetic microtubule images, we generate blurred images by convolving them with a model PSF.
In particular, the numerical PSF was computed using the Born and Wolf model (Kirshner et al., 2011), which is given by
[Equation]
(28) where C is a normalizing constant, k = 2 π/λ is the wave number of emitted light, λ is the wavelength, NA is the numerical aperture, and n i is the refractive index of immersion layer.
For all simulation, we use NA = 1 . 4 , and n i = 1 . 5 .
Then, the simulation is performed with different λ for the PSF model (28).
The convolution was performed using DeconvolutionLab2 (Sage et al., 2017).
The convolved data was then added with the mixture of zero mean Gaussian with the standard deviation σ = 3 and Poisson noises with the parameter m = 7 .
The size of the synthetic data was 256 × 512 × 128 , and 18 samples of the synthetic data were used for training and the other one is used for validation.
Due to memory limitations, the volume was split into 64 × 64 × 64 size patches, which are used as inputs to the neural network.
Fig.
9 shows qualitative comparisons of different reconstruction methods over datasets generated by the theoretical PSFs at different wavelengths.
Here, the training was performed using the blurred data with λ = 500 nm PSF kernnel, but the inference was performed using data with wavelengthnm , 500 nm , and 600 nm , respectively.
The goal of this study is to investigate the generalization power of the proposed method with respect to different wavelengths.
The supervised method removed most blur and noise; however, it lost many ﬁne details of the tubule structure and showed false continuity over the tubule structure.
The conventional cycleGAN method showed better qualitative results than the supervised one, presenting ﬁner details and the correct continuity of the tubule structure, but still some of the structures were not fully recovered.
On the other hand, the proposed method showed the best qualitative results, recovering most of the microtubule structures.
Moreover, as shown in Table 2, the proposed method showed the best PSNR and SSIM scores.
This conﬁrmed that the proposed method generalizes well, even when the PSF model for the training data is not matched well with that of the inference phase.