S ELF - SUPERVISED REGRESSION LEARNING USING DOMAIN KNOWLEDGE : A PPLICATIONS TO IMPROVING SELF - SUPERVISED IMAGE DENOISING
Anonymous authors
Paper under double-blind review
A BSTRACT
Regression that predicts continuous quantity is a central part of applications using computational imaging and computer vision technologies.
Yet, studying and understanding self-supervised learning for regression tasks – except for a particular regression task, image denoising – have lagged behind.
This paper proposes a general self-supervised regression learning (SSRL) framework that enables learning regression neural networks with only input data (but without ground-truth target data), by using a designable operator that encapsulates domain knowledge of a speciﬁc application.
The paper underlines the importance of domain knowledge by showing that under some mild conditions, the better designable operator is used, the proposed SSRL loss becomes closer to ordinary supervised learning loss.
Numerical experiments for camera image denoising and low-dose computational tomography denoising demonstrate that proposed SSRL signiﬁcantly improves the denoising quality over several existing self-supervised denoising methods.
I NTRODUCTION
Deep regression neural network (NN)-based methods that can accurately predict real- or complexvalued output have been rapidly gaining popularity in a wide range of computational imaging and computer vision applications including image denoising (Vincent et al., 2010; Xie et al., 2012;
Zhang et al., 2017), image deblurring (Xu et al., 2014), image super-resolution (Dong et al., 2016;
Kim et al., 2016), light-ﬁeld reconstruction (Chun et al., 2020; Huang et al., 2020), object localization (Szegedy et al., 2013), end-to-end autonomous driving (Bojarski et al., 2016).
Yet, they lack a general self-supervised learning framework.
In training a regression NN f : R N → R M , the most prevalent supervised learning approach minimizes the mean square error (MSE) between what f predicts from an input x ∈ R N and a ground-truth target y ∈ R M : min f E x,y (cid:107)
[Equation]
Learning a denoising or reﬁning NN uses (1) with M = N – dubbed Noise2True – where x is a corrupted image and y is a clean (i.e., ground-truth) image.
However, it is challenging or even impossible to collect many clean images y in many practical applications, motivating research on self-supervised learning for image denoising (Ulyanov et al., 2018; Soltanayev & Chun, 2018;
Krull et al., 2019; Batson & Royer, 2019; Laine et al., 2019; Moran et al., 2020;
Quan et al., 2020; Xu et al., 2020; Hendriksen et al., 2020; Xie et al., 2020; Huang et al., 2021) – called self-supervised image denoising.
To learn a denoiser f with single noisy images, a popular self-supervised image denoising method, Noise2Self (Batson & Royer, 2019)
(see also the concurrent work (Krull et al., 2019)), and its sophisticated relaxation, Noise2Same (Xie et al., 2020), study the following MSE minimization problem:
[Equation]
These methods use some partitioning schemes in (2) to avoid that its optimal solution is just the identity mapping I .
Noise2Noise (Lehtinen et al., 2018) that learns a denoiser with pairs of two independent noisy images, is a pioneer work for self-supervised image denoising.
Motivated by
Noise2Noise, several self-supervised image denoising methods such as Noise2Inverse (Hendriksen et al., 2020), Neighbor2Neighbor (Huang et al., 2021), and (Soltanayev & Chun, 2018; Moran et al., 2020;
Xu et al., 2020) emulate pairs of two independent noisy images, by applying partitioning or adding simulated noise to single noisy measurements.
All the aforementioned methods have been developed based on some noise assumptions including pixel-wise independent noise (Krull et al., 2019; Xie et al., 2020; Huang et al., 2021) and zero-mean noise (Lehtinen et al., 2018; Batson &
Royer, 2019; Xie et al., 2020).
Yet, they lack design ﬂexibility that might relax such noise assumptions and further improve the denoising performance of NNs.
Some works estimate statistical parameters of noise, such as noise histogram (Krull et al., 2020) and parameters of Gaussian mixture noise model (Prakash et al., 2021).
This paper presents new insights on this topic.
The paper proposes a general self-supervised learning framework for regression problems, which we refer to as self-supervised regression learning (SSRL).
Proposed SSRL enables learning regression NNs with only input samples, by using a designable operator that can encapsulate domain knowledge of a speciﬁc application.
Our main results show that under some mild conditions (e.g., in image denoising, statistical noise properties in x ), the better desinable operator is used, the proposed SSRL loss becomes closer to ordinary supervised learning loss.
In addition, a designable operator with good domain knowledge can relax noise assumptions of existing self-supervised denoising methods.
Numerical experiments for camera image and low-dose computational tomography (CT) denoising with both simulated and real-world datasets – corrupted by only single noise realization – demonstrate that the proposed SSRL framework signiﬁcantly improves denoising quality compared to several existing self-supervised denoising methods.
Put together, our ﬁndings provide new insights into how using good domain knowledge can improve self-supervised denoising, underscoring the beneﬁts of understanding application-speciﬁc knowledge in SSRL.
SSRL USING DOMAIN KNOWLEDGE
The proposed SSRL loss is given by
[Equation]
where g : R N → R M is a designable operator encapsulating domain knowledge of a speciﬁc application.
We will incorporate some sophisticated setups in (3) such that f obtained by minimizing (3) cannot merely be g .
Although related theorems hold for any M (see later), we mainly focus on practical image denoising applications with pseudo-target g ( x ) (cid:54)≈ y .
2. M OTIVATION
[Figure]
section empirically shows that understanding domain knowledge is important for designing g in the proposed SSRL loss (3).
The following camera image denoising examples demonstrate that well-designed g with good domain knowledge improves the denoising performance of learned f via (3).
Suppose that camera images are corrupted by salt-and-pepper noise.
Consider two example setups for g ( · ) , median ﬁltering and BM3D denoiser (M¨akinen et al., 2020), denoted by median( · ) and BM3D ( · ) , respectively.
Figure 1 compares the denoising performance of minimum f (cid:63) with the two aforementioned g setups: f (cid:63) with median ﬁltering signiﬁcantly improved that with BM3D denoiser. This is not surprising, as median ﬁltering is widely known to be effective in reducing salt-and-pepper noise (Bovik, 2010, §3.2).
This result emphasizes the importance of understanding domain knowledge of speciﬁc applications in proposed SSRL.
2. P RELIMINARIES
We ﬁrst introduce the J -complement between two functions f and g :
Deﬁnition 1.
For a given partition J =
{ J 1 , . . .
, J B } ( | J 1 | + . . .
+ | J B | = N ) of the dimensions of input x ∈ R N , functions f : R N → R M and g : R N → R M are called J -complementary, if f ( x J c ) does not depend on g ( x J ) for all J ∈ J , where J c denotes the complement of J , and ( · ) J denotes a vector restricted to J .
That is, f and g use information from outside and inside of J to predict output and give pseudotarget, respectively.
In denoiser learning (where M = N ), Deﬁnition 1 specializes to the Jinvariance of f (Batson & Royer, 2019), by setting g
= I .
Incorporating Deﬁnition 1 into the
SSRL loss (3) is a straightforward approach to avoid that optimal f is just g in (3).
The proposed
SSRL framework assumes the followings:
Assumption 1) x J and x J c are conditionally independent given y ,
i
e., p ( x | y )
=
p ( x J | y ) p ( x J c | y ) .
Assumption 2) E
[ g ( x ) | y ]
= y .
Assumption 3) f and g are (Borel-)measurable.
In denoiser learning, Assumption 1 holds if noise in each subset J ∈ J is conditionally independent from that in J c , given y .
Assumption 1 can be satisﬁed in general regression NN learning, by adding small randomized perturbations (independent of y ) to either J or J c , similar to Moran et al.
Assumption 2 suggests a direction for designing g : suppose that x has non-zero-mean noise; one then can design g to make noise zero-mean using the domain knowledge.
Assumption 3 is satisﬁed if f and g are continuous.
This condition is mild because many regression NNs f are continuous – where their modules, convolution, matrix-vector multiplication, rectiﬁed linear unit activation, max pooling, etc. are continuous – and one can design g with measurable or continuous function.
Finally, observe that the proposed SSRL loss (3) can be rewritten by
[Equation]
We aim to either remove or control the third term, incorporating the J -complement and/or Assumptions 1-3 introduced above.
2. SSRL USING DOMAIN KNOWLEDGE WITH J - COMPLEMENT
This section studies SSRL loss (3) minimization over f that is J -complementary of g .
Our ﬁrst main result shows that under Assumptions 1–3, the SSRL loss (3) with the J -complement is the sum of the ordinary supervised learning loss and variance of g ( x ) − y , i.e., the third term in (4) vanishes.
Theorem 2.
Under Assumptions 1–3, the SSRL loss (3) with the J -complement in Deﬁnitionbecomes
[Equation]
The following equality similarly holds for any K ∈ K : E x (cid:107) f
[Equation]
x ) K
[Equation]
g
[Equation]
K
[Equation]
[Equation]
x
[Equation]
y
[Equation]
f ( x ) K − y K (cid:107)
+ (cid:107) g
( x ) K − y K (cid:107) 22 , where K is a partition of { 1 , . . .
, M } , and f ( · ) K and g ( · ) K denote f ( · ) and g ( · ) restricted to K , respectively. The optimal solution for (5) is given by [Equation]
Proof.
See Section A.1 in the appendix.
The result (6) suggests another direction for designing g : we aim to design “good” g that can make g ( x J ) close to y . Using such g , optimal solution of LHS in (6), E [ g ( x J ) | x J c ] , becomes close to its supervision counterpart, E
[ y | x J c ] . Consequently, such g reduces the second term E x,y (cid:107) g ( x ) − y (cid:107) 22 in
RHS of (5), leading (3) closer to (1). If g is ideal such that g ( x J ) = y , the SSRL loss (3) becomes the usual supervised learning loss. In designing g , domain knowledge of speciﬁc application is crucial.
Domain knowledge includes noise properties in x and pre-trained NN by existing self-supervised denoising, such as Noise2Self (Batson & Royer, 2019) and Noise2Noise (Lehtinen et al., 2018).
Speciﬁcally, the proposed SSRL loss using the J -complement is given by
[Equation]
Figure 2 illustrates (7) with complementary checkerboard masks J c and J , where f and g use almost equal amount of information, and its variant, where g uses much less information than f . The variant computes MSE only on J ∈ J ; in this setup, it is challenging for g to predict the entire image.
Relation to previous self-supervised denoising works.
In denoiser learning, the proposed SSRL loss (3) with the J -complement of f and g = I specializes to (2) with the J -invariance of f , i.e., Noise2Self.
Noise2Inverse (Hendriksen et al., 2020) and Neighbor2Neighbor (Huang et al., 2021) that emulate pairs of two independent noisy images by partitioning single measurements (e.g., CT ray measurements with independent noise and corrupted images with pixel-wise independent noise) can be viewed as Noise2Self.
Thus, SSRL loss (3) with the J -complement of f and g =
I specializes to the aforementioned Noise2Noise-motivated self-supervised denoising methods.
(Noise2Noise also can be viewed by SSRL (3) with the setup above, by constructing x with stacking two independent noisy images, where an image is corrupted by two independent noise realizations.)
2. SSRL USING DOMAIN KNOWLEDGE WITHOUT J - COMPLEMENT
This section studies the SSRL loss (3) without using the J -complement in Deﬁnition 1.
Observe by (4) that E x,y (cid:107) f
( x ) − y (cid:107) 22 + (cid:107) g ( x [Equation] ) − y [Equation]
22 = [Equation] x [Equation]
f [Equation] x ) − g ( x J ) (cid:107) 22 + 2 E x,y (cid:104) f ( x ) − y, g ( x J ) − y (cid:105) .
Inspired by Noise2Same (Xie et al., 2020), the second proposed SSRL approach is to minimize an approximation of the right hand side in this equation that does not assume that f and g are J -complementary.
Our second main result ﬁnds an upper bound for the term E x,y (cid:104) f
( x ) − y, g ( x J ) − y (cid:105) without relying on the J -complement (remind that this term vanishes if f and g are J -complementary; see Theorem 2).
Theorem 3.
Assume that Var( g ( x J ) m | y ) ≤ σ 2 , ∀ m .
Under Assumptions 1–3, the following bound holds:
[Equation]
The following bound similarly holds for any K ∈ K : E x,y (cid:104) f
( x ) K − y K , g ( x J ) K − y K (cid:105) ≤ σ (cid:112) | K |· (cid:0)
E x (cid:107) f
( x ) K − f ( x J c ) K ) (cid:107) 22 (cid:1) 1 / 2 , where K and K are deﬁned in Theorem 2.
Proof.
See Section A.2 in the appendix.
Using Theorem 3, the proposed SSRL loss that does not rely on the J -complement is given by
[Equation]
where σ is given in Theorem 3.
Here, a regression NN f can use information from the entire input x , whereas L ind ( f ) in (7) uses only partial input x J c in f .
The intuition for designing g in Section 2.3 similarly applies here.
Our aim is to design “good” g such that g ( x J ) is closer to y , consequently leading (9) close to (1).
Similar to the variant of L ind ( f ) (see Section 2.3), if the amount of information between two partitions J and J c is unbalanced in denoiser learning, one can modify (9) to compute the MSE only on J or J c in either both terms or the right term in (9).
(These variants use the second result in Theorem 3.)
We conjecture that how well designed g is, i.e., how close is pseudo-target g ( x J ) to ground-truth y , is captured by σ deﬁned in Theorem 3.
We support the conjecture with examples in Section A.3 of the appendix.
Then, this “goodness” of g balances the two terms in (9) via σ .
If g is well-designed such that g ( x J ) is close to y , i.e., σ 2 is small, then the SSRL loss (9) relies more on the ﬁrst term with “good” pseudo-target.
If g is poorly-designed such that σ 2 is large, then (9) puts more weight more on the second term that can implicitly promote the J -invariance of f .
Relation to previous self-supervised denoising work.
The proposed SSRL loss (9) becomes the
Noise2Same loss (Xie et al., 2020, Thm. 2), by replacing g ( x J ) with x and adding randomness to K =
J in the second term.
In practice, one can tune σ in (9) without knowing its exact value, similar to Noise2Same.
One might use the aforementioned conjectured behavior of (9) with expected performance of g .
2. R ELAXING NOISE ASSUMPTIONS OF EXISTING SELF - SUPERVISED DENOISING METHODS
This section explains how proposed SSRL (3) can relax noise assumptions of existing selfsupervised denoising methods.
Noise assumptions of existing self-supervised denoising methods include additive white Gaussian noise (AWGN with known variance) (Soltanayev & Chun, 2018), pixel-wise independent noise (Krull et al., 2019; Xie et al., 2020; Huang et al., 2021), and zero-mean noise (Lehtinen et al., 2018;
Quan et al., 2020) or more generally E [ x | y ]
= y (Batson & Royer, 2019;
Xie et al., 2020).
Assumption 1 of proposed SSRL relaxes the AWGN and pixel-wise independent noise assumptions, and is identical to the ﬁrst assumption of Noise2Self (Batson & Royer, 2019).
Assumption 2 of proposed SSRL can relax the second assumption of Noise2Self, E
[ x | y ]
=
y , that is also (implicitly) used in Noise2Same (Xie et al., 2020) and Noise2Noise (Lehtinen et al., 2018), by using a desinable function g .
For example, x is corrupted by additive non-zero-mean noise e that is independent of y , i.e., x = y + e , then one can design g as follows:
g ( x ) = x − E [ e ] , where E [ e ] can be estimated from calibration of imaging systems.
The next section will explain how one can design g using domain knowledge and select g if domain knowledge is unavailable in image denoising.
E XAMPLES OF HOW TO DESIGN g USING DOMAIN KNOWLEDGE , AND EMPIRICAL - LOSS APPROACH FOR SELECTING g IN IMAGE DENOISING
Understanding noise statistics or properties is the ﬁrst step towards accurate image recovery in computational imaging.
First, this section describes how to use domain knowledge for designing g in two practical imaging applications: 1) camera image denoising in mixed Poisson–Gaussian–Bernoulli noise, 2) low-dose CT denoising.
We consider both applications to be practical in the sense that they have complicated noise models or strong noise, where Assumptions 1–2 in Section 2.2 are not completely satisﬁed.
Noisy images in the ﬁrst application are corrupted by independent and identically distributed (i.i.d.)
noise with non-zero mean.
The noise in the second application is approximately zero-mean but it is likely non-i.i.d.
In designing g for each application, we will use the suggestions in Sections 2.2–2.3, investigating Assumptions 1–3.
Second, we propose a g -selection approach in image denoising that calculates some empirical measure related to (6) using only input training data.
3. D ESIGNING g USING DOMAIN KNOWLEDGE : C AMERA IMAGE DENOISING
The major noise sources in camera imaging (using charge coupled device) include object-dependent photoelectrons in image sensors, readout in camera electronics, and analog-to-digital converter and transmission errors that can be modeled by Poisson noise, AWGN, and Bernoulli (i.e., salt-andpepper) noise models (Snyder et al., 1993), (Bovik, 2010, p. 90).
We use the following practical mixed Poisson–Gaussian–Bernoulli noise model (Snyder et al., 1993; Batson & Royer, 2019):
[Equation]
where Π [0 , 255] performs 8 -bit quantization and clips pixel values outside of [0 , 255] , Bernoulli substitutes a pixel value with either 0 or 255 with probability p ( 0 and 255 are coined with equal probability), Poisson generates pixel intensity-dependent Poisson noise with gain parameter λ , and (cid:15) is
AWGN.
Figure 3 (left) shows a noisy image corrupted by the mixed noise model (10).
If an image y is corrupted only by the mixed Poisson–Gaussian noise, E [ x | y ] =
y in Assumption 2 can be satisﬁed ( E [ Poisson ( λy n ) /λ + (cid:15) | y ]
= y n , ∀ n ).
However, if Bernoulli noise is additionally considered as given in (10), the assumption E
[ x | y ]
= y will not be satisﬁed ( E [ Bernoulli p ( y n ) | y ]
= (1 − p )
y n + 127 .
5 p , ∀ n ).
The quantization-clipping operator Π [0 , 255] also makes it hard to satisfy E [ x | y ]
= y .
[Equation]
[Figure]
Following the suggestion based on Assumption 2 (see Section 2.2), we handcraft g to “approximately” satisfy E [ g ( x ) | y ]
=
y with a simple operator.
We interpret aforementioned Bernoulli noise and clipping artifact as salt-and-pepper noise.
Median ﬁltering is a computational efﬁcient method that is effective in reducing salt-and-pepper and impulse noises (Bovik, 2010, §3.2).
We design g by applying weighted median ﬁltering (Brownrigg, 1984) to a pixel with intensity either 0 orat each color channel, aiming that this g design “approximately” satisfy E [ g ( x ) | y ]
= y by suppressing the salt-and-pepper noise effects cased by Π [0 , 255] and Bernoulli p .
(This is supported by empirical results in Section S.2.)
Assumption 1 is satisﬁed because the noise in (10) is i.i.d.
and independent of y .
In Assumption 3, we conjecture that the above g design is measurable (median operator is measurable under some conditions (Rustad, 2004)).
3. D ESIGNING g USING DOMAIN KNOWLEDGE : L OW - DOSE CT DENOISING
In X-ray CT (with a monoenergetic source), the pre-log measurement data is usually modeled by the Poisson model, i.e., Poisson { ρ 0 exp( − [ Ay ] l ) } , l = 1 , . . .
, L , where ρ 0 is the number of incident photons per ray, A ∈ R L × N is a CT projection system matrix, and L is the number of measured rays.
Using the quadratic approximation to the log-likelihood of a Poisson model, the post-log measurement z ∈ R L given y can be approximated as the following Gaussian model (Sauer &
Bouman, 1993; Fessler, 2000): z | y ∼ N
( Ay, C ) , where C ∈ R L × L is a diagonal covariance matrix and its diagonal elements become more nonuniform in lower-dose CT.
This model suggests that post-log measurement can be modeled by z
=
Ay + ε , where ε ∼ N (0 , C ) .
The ﬁltered back-projection (FBP) method (Kak & Slaney, 1988, §3) performs computationally efﬁcient CT reconstruction and has been widely used in commercial CT scanners (Pan et al., 2009).
In low-dose CT, however, reconstructed image x = Fz suffers from strong noise and streak artifacts, where
F ∈ R N × L denotes a linear FBP operator, motivating research on learning denoising NNs.
Figure(right) shows a noisy FBP image in low-dose CT.
Using the statistical results above, we model that a reconstructed image by F is corrupted by an arbitrary additive noise e :
[Equation]
Low-dose CT uses all projection rays similar to standard-dose CT (but with substantially reduced dose) where F approximately inverts A , i.e., FA ≈ I , so we conclude that under (11), E [ e ] ≈ 0 and E [ x | y ]
≈ y ( e and x become approximately independent).
The above domain knowledge in low-dose CT indicates that handcrafting g to have zero-mean e to satisfy Assumption 2 can be redundant.
Following the suggestion motivated by Theorem 2 (see
Section 2.3), we set g as a pre-trained denoiser by the existing self-supervised denoising methods (Batson & Royer, 2019; Hendriksen et al., 2020; Xie et al., 2020).
Since such pre-trained g will have some denoising capability and give better reference than g
= I to (7) and (9) (see empirical results in Section S.2), we expect that proposed SSRL losses (7) and (9) improve the denoising quality over the aforementioned existing self-supervised denoising methods.
Assumption 1 is unlikely satisﬁed because in FBP images, neighboring noise components are likely to be correlated, i.e., Var( e ) ≈ FCF (cid:62) using FA ≈ I and noise model (11).
Assumption 3 is satisﬁed as we use the conventional denoisiong NN, DnCNN (Zhang et al., 2017) and (modiﬁed)
U-Net (Ronneberger et al., 2015), that are a continuous function.
3. E MPIRICAL - LOSS APPROACH FOR SELECTING g IF DOMAIN KNOWLEDGE UNAVAILABLE
If accurate domain knowledge of a speciﬁc application is unavailable, it would be challenging to explicitly design g .
In such cases in denoising, our general suggestion is to measure an existing selfsupervised denoising loss, an upper bound of (cid:107) g ( x J ) − y (cid:107) 22 or its variant that measure the quality of g , only with input training data.
The lower quantity implies that g is better and implicitly encapsulates better domain knowledge.
In camera image denoising with the real-world dataset (Abdelhamed et al., 2018), the empirical measure of the Neighbor2Neighbor loss (Huang et al., 2021) – E (cid:107) g
( x J ) − x J c (cid:107) 22 – with setting g as I and median ﬁltering are 0 . 0052 and 0 . 0048 , respectively.
In low-dose CT denoising with the real-world dataset (Moen et al., 2021), the empirical measure of the Noise2Self loss (Batson & Royer, 2019) – E (cid:107) g
( x J ) J c − x J c (cid:107) 22 – with setting g as I and pre-trained DnCNN by Noise2Self are 22044 .
5 and 17062 .
0 (in HU 2 where HU stands for modiﬁed Hounsﬁeld unit), respectively.
We expect better SSRL performance with the selected g designs over I .
E XPERIMENTAL RESULTS AND DISCUSSION
We evaluated proposed SSRL in two practical imaging applications in Section 3 with both simulated and real-world datasets.
For these applications, we mainly focuses on comparisons with selfsupervised denoising methods using single noisy input samples, particularly when statistical noise parameters are unavailable. We compared the performances of the following methods: Noise2Self (Batson & Royer, 2019), Noise2Noise-motivated methods that emulate pairs of two independent noisy images – Neighbor2Neighbor (Huang et al., 2021) or Noise2Inverse (Hendriksen et al., 2020)
– Noise2Same (Xie et al., 2020), and corresponding SSRL to each aforementioned method.
We also included Noise2True results as baseline.
For
f
or g in all the methods, we used the conventional denoising NN architecture,
DnCNN (Zhang et al., 2017) or modiﬁed U-Net used in Noise2Self.
We include experiment setup, and image and numerical results for/from real-world datasets in Sec.
A.4.
E XPERIMENTAL SETUP FOR SIMULATED DATASETS
Camera image denoising in mixed noise.
We evaluated the proposed SSRL framework with three RGB camera image datasets, ImageNet ILSVRC 2012 Val (Russakovsky et al., 2015), BSD 300 (Martin et al., 2001), Set 5 (Bevilacqua et al., 2012).
For training, we used the ImageNet ILSVRC 2012 Val dataset with 20 , 000 images; for tests, we used the BSD 300 and Set 5 datasets consisting of 300 and 5 images, respectively.
We simulated noisy images with the following imaging parameters introduced in (10): λ = 30 , σ (cid:15)
= 60 , and p = 0 .
We evaluated the denoising quality by the most conventional error metric in camera image denoising, PSNR and structural similarity index measure (SSIM).
In Neighbor2Neighbor setup, we emulated two independent noisy images from single noisy images by random neighbor sub-sampling with 2 × 2 -window (Huang et al., 2021).
Low-dose CT denoising.
We evaluated the proposed SSRL framework with The 2016 Low Dose
CT Grand Challenge data (McCollough, 2016).
We selected 200 regular-dose chest images of size
N =
× 512 and the 3 mm slice thickness from four patients.
For training, we used 170 ( 85 %) chest images from three patients; for tests, we used 30 ( 15 %) chest images from the other patient.
We simulated low-dose sinograms using the Poisson model with the selected regular-dose chest datasets.
In particular, we simulated sinograms of size L = 736 × 1152 (‘detectors’ × ‘projection views’),
Reference
[Figure]
SSIM
[Figure]
SSIM
[Figure]
with fan-beam geometry corresponding to a no-scatter monoenergetic source
×
We used FBP (Kak & Slaney, 1988, §3) to reconstruct images with resolution 0 .
69 mm × 0 .
69 mm.
We evaluated the denoising quality by the most conventional error metric in CT application, RMSE in HU.
In Noise2Inverse setup, we emulated two independent noisy images by partitioning single sinograms with odd and even views and applying FBP to two partitioned independent sinograms.
4. C OMPARISONS BETWEEN DIFFERENT SELF - SUPERVISED DENOISING METHODS
Compare each existing self-supervised denoising method to its corresponding SSRL setup in Figures 4–5, and Figures S.1–S.4 and Tables S.1–S.3; see three comparison sets, each grouped by red box.
For both applications, proposed SSRL achieves signiﬁcantly better image denoising quality, i.e., closer to the Noise2True quality, compared to the existing methods, Noise2Self, Neighbor2Neighbor, Noise2Inverse, and Noise2Same, regardless of the regression NN architecture.
We show DnCNN prediction uncertainty of all the methods in Figure S.5.
Camera image denoising in mixed noise (simulated data).
Figures 4 and S.3 show that in all the three comparison sets, SSRL gives closer image quality, particularly color saturation, to Noise2True than existing methods, Noise2Self, Neighbor2Neighbor, and Noise2Same.
Setting g as weighted median ﬁltering avoids bias in SSRL loss caused by salt-and-pepper noise.
Yet, compared to Noise2True, denoised images obtained by proposed SSRL lack saturation and detail preservation.
For saturation and detail preservation comparisons, see Figures 4, S.1 and S.3.
Comparing the three comparison sets in Figures 4 and S.3, and Tables S.1–S.2 shows that all the
Noise2Self, Neighbor2Neighbor, and Noise2Self setups have comparable results in terms of PSNR values.
The potential reason is that in this application, all the three setups similarly satisfy Assumptions 1–3 (see Section 2.2); in particular, Assumption 1 is well-satisﬁed by pixel-wise i.i.d.
noise.
In this application, the zero-mean noise assumption of the existing self-supervised denoising methods is violated, whereas its counterpart in proposed SSRL, Assumption 2, is “approximately” satisﬁed by g in Section 3.1.
This suggests the importance of satisfying Assumption 2 with good g .
Low-dose CT denoising (simulated data).
In all the three comparison sets, SSRL better recovers low-contrast regions (e.g., soft tissues) and small details, and signiﬁcantly reduces noise and artifacts throughout the image, over existing methods, Noise2Self, Noise2Inverse, and Noise2Same.
See
Reference
Noise2True
Noise2Self
Proposed SSRL in Noise2Self setup
RMSE = 16 .
3 HU
RMSE = 16 .
3 HU
RMSE = 16 .
3 HU
RMSE = 36 .
2 HU
RMSE = 36 .
2 HU
RMSE = 36 .
2 HU
RMSE = 25 .
HU
RMSE = 25 .
HU
RMSE = 25 .
HU
Noise2Inverse
Proposed SSRL in Noise2Inverse setup
Noise2Same
Proposed SSRL in Noise2Same setup
RMSE = 22 . 9 HU
RMSE = 22 . 9 HU RMSE = 22 . 9 HU
RMSE = 21 .
9 HU
RMSE = 21 .
9 HU
RMSE = 21 .
9 HU
RMSE = 28 .
4 HU
RMSE = 28 .
4 HU
RMSE = 28 .
4 HU
RMSE = 26 .
HU
RMSE = 26 .
HU
RMSE = 26 .
HU zoom-ins and circled small details in Figures 5 and S.4, and error images in Figure S.2, particularly in ‘Noise2Self vs. Propose SSRL in Noise2Self setup’ and ‘Noise2Same vs. Proposed SSRL in
Noise2Same setup.’
The results might imply that simply setting g as pre-trained NN by existing selfsupervised denoising methods works like a charm in SSRL.
Proposed SSRL in the Noise2Inverse setup can provide images with image quality that is comparable to conventional FBP at 10 times higher dose (when ρ 0 = 5 × 10 5 , RMSE = 20 .
5 HU on average; see DnCNN results in Figure 5).
Next, comparing Noise2Self and Noise2Same result sets to that of Noise2Inverse in Figures 5, S. and S.4, and Table S.3 shows that Noise2Inverse setup signiﬁcantly improves the denoising quality, compared to Noise2Self and Noise2Same setups.
We conjecture that violation of Assumption 1 – that is satisﬁed in the Noise2Inverse setup but unlikely to be satisﬁed in the Noise2Self and
Noise2Same setups in this application – degrades the performance.
Camera image denoising and low-dose CT denoising with real-world datasets.
Denoised image results in Figures A.2–A.3 and S.9–S.10 from the two real-world datasets (Abdelhamed et al., 2018; Moen et al., 2021) demonstrate that SSRL improves existing self-supervised denoising methods, particularly Neighbor2Neighbor, Noise2Self, and Noise2Same, without having their exact noise properties.
The results well correspond to our expectation in Section 3.3.
C ONCLUSION
It is important to develop SSRL that enables comparable prediction performances to supervised learning, because it is extremely challenging to collect many ground-truth target samples in many practical computational imaging and computer vision applications.
The proposed SSRL framework is applicable to a wide range of regression problems, while bridging the gap between SSRL and supervised regression learning via domain knowledge of applications.
To achieve closer prediction performance to supervised learning, SSRL uses domain knowledge to design a better pre-processing operator g such that g ( x J ) becomes closer to y .
For camera image denoising and low-dose CT denoising with both simulated and real-world datasets, SSRL achieves more accurate prediction compared to the existing self-supervised denoising methods (Batson & Royer, 2019; Huang et al., 2021;
Hendriksen et al., 2020; Xie et al., 2020).
Our future work is extending proposed SSRL to other machine learning problems such as teacher student models (see Section S.4), meta-learning, and online learning.
On the application side, our future work is applying proposed SSRL to regression problems that map data from one domain to another different.
R EPRODUCIBILITY
Section 2.2 speciﬁes all the theoretical assumptions.
Section A.1 and Section A.2 in the appendix include the complete proofs of Theorem 2 and Theorem 3, respectively.
Section S.1 in the supplement includes the complete implementation details including the hyperparameter selection strategies and the chosen hyperparameters.
We included an anonymized zip ﬁle that includes test codes, test data, trained models, and instructions and codes that provide complete description of the data processing steps, as supplementary materials.
We will make our codes (for data construction, training and test) and trained models publicly available on GitHub if the paper is accepted.
A PPENDIX A. P ROOFS FOR T HEOREM
Observe ﬁrst that combining Assumptions 1 & 3 and the J -complement implies that f ( x ) m and g ( x ) m are conditionally independent, given y , i.e.,
[Equation]
Using this result with Assumption 2 and reminding that the J -complement implies E x (cid:107) f ( x ) − g ( x ) (cid:107)
= E x
[Equation]
f ( x J c ) − g ( x J ) (cid:107) 22 , we obtain the following result from (4):
[Equation]
where the equality uses
[Equation]
in which the second equality uses the ﬁrst result above and the third equality holds by Assumption 2.
This completes the proofs.
A. P ROOFS FOR T HEOREM
We ﬁrst obtain the following bound:
[Equation] [Equation]
=
[Equation]
E
[Equation]
E y
[Equation]
Cov
[Equation]
x
[Equation] [Equation]
(cid:88) m
[Equation]
≤
[Equation]
m
[Equation]
where the second equality holds by Assumption 2, the third equality uses Cov( X, Y | Z ) =
E [ XY | Z ] − E
[ X | Z ] E [ Y | Z ] where X , Y , and Z are random variables or vectors, the ﬁfth equality holds because f ( x J c ) m does not correlate with g ( x J ) m , ∀ m (due to Assumptions 1 and 3), so subtracting f ( x J c ) m from f ( x ) m does not change the covariance.
Now, the ﬁrst inequality uses the Pearson correlation coefﬁcient bound, the second inequality uses the Jensen’s inequality
E √ X ≤ √ E X , the third inequality uses the Jensen’s inequality (cid:80) m √ a m ≤ (cid:112) M (cid:48) (cid:80) m a m for any a ∈ R M (cid:48) , and the last inequality holds by the conditional variance bound speciﬁed in Theorem 3.
We bound and rewrite the ﬁnal result above and this completes the proof:
[Equation]
where the equality uses the ﬁltration property of conditional expectation.
A. E XAMPLES THAT SUPPORT CONJECTURED BEHAVIOR OF (9) WITH σ
The ﬁrst example in general regression models the pseudo-target as follows: g ( x J ) =
y + e 1 , where e 1 ∈ R M is some arbitrarily additive noise independent of y .
This gives Var( g ( x J ) m | y ) =
Var( y m + ( e 1 ) m | y )
=
Var( e 1 ) m ≤ σ 2 , m = 1 , . .
, M .
Under this model, how close is g ( x J ) to y is captured by σ .
The second example in image denoising assumes that x is corrupted by AWGN e 2 ∈ R N that is independent of y .
Setting g as a linear mapping G ∈ R J × N gives Var( g ( x J ) n | y )
=
Var(( Gy J ) n + ( G ( e 2 ) J ) n | y )
=
Var(( G ( e 2 ) J ) n ) ≤ σ 2 , ∀ n = 1 , . .
. , N .
Under this model, how close is g ( x J ) to y is captured by σ .
A. E XPERIMENTAL SETUP AND RESULTS FOR / FROM REAL - WORLD DATASETS A.4. E XPERIMENTAL SETUP FOR REAL - WORLD DATASETS
We also evaluated the proposed SSRL framework with real-world camera image and low-dose CT datasets, where we do not have their complete noise properties/statistics.
We chose the publicly available SIDD sRGB Data (Abdelhamed et al., 2018) and Low Dose CT Image and Projection
Data (Moen et al., 2021), where both the datasets include high-quality images or standard-dose FBP images so that one can run Noise2True experiments and obtain quantitative results.
We used the
DnCNN architecture for all experiments.
For each experiment, we used the same implementation setup (such as hyperparameters and masking scheme) as that in the corresponding simulated data experiment.
In particular, g is median ﬁlter and pre-trained denoiser via existing self-supervised denoising methods in camera image denoising and low-dose CT denoising experiments, respectively. [Figure]
Camera image denoising with SIDD sRGB Data.
We used the SIDD sRGB training and validation datasets for training and tests, respectively.
We chose the representative comparison setup, Neighbor2Neighbor, from the camera image denoising experiments using simulated data in Section 4.1.
Low-dose CT denoising with Low Dose CT Image and Projection Data.
We followed the data construction setup in Section 4.1 that was used in simulated data experiments; we remark that chest CT scans in the Low Dose CT
Image and Projection Data use the 1 mm slice thickness.
We cannot run Noise2Inverse experiments because the Low Dose CT Image and Projection Data does not provide two independent half-view FBP images.
We thus chose the other comparison setups, Noise2Self and Noise2Same.
A.4. M AIN EXPERIMENTAL RESULTS FROM REAL - WORLD DATASETS
This section includes main experimental results such as denoised images and calculated performance measure, from the real-world datasets.
Section S.3 in the supplement includes their supplementary results.
[Figure]
[Figure]
RMSE = 59 .
2 HU
RMSE = 59 .
2 HU
[Figure]
RMSE = 106 . 8 HU RMSE = 106 . 8 HU
[Figure]
RMSE = 71 .
2 HU
RMSE = 71 .
2 HU
[Figure]
RMSE = 90 .
5 HU
RMSE = 90 .
5 HU
[Figure]
RMSE = 71 .
HU
RMSE = 71 .
HU
S ELF - SUPERVISED REGRESSION LEARNING USING DOMAIN KNOWLEDGE : A PPLICATIONS TO IMPROVING SELF - SUPERVISED IMAGE DENOISING (S UPPLEMENT )
Anonymous authors
Paper under double-blind review
S. I MPLEMENTATION DETAILS , DATA AND CODE LICENSES , AND LIBRARY VERSIONS
This section describes implementation details, lists hyperparameters, and speciﬁes license of datasets and codes used in this study.
S.1. D ATA AND CODE LICENCES , AND LIBRARY VERSIONS
The ImageNet ILSVRC 2012 Val and BSD 300 datasets have the Custom license (research, non-commercial), the Set 5 data has the Unknown license, and the SIDD sRGB Data (Abdelhamed et al., 2018) has the MIT license.
We obtained The 2016 Low Dose CT Grand Challenge data (McCollough, 2016) from https://aapm.app.box.com/s/ eaw4jddb53keg1bptavvvd1sf4x3pe9h/file/856956352254 , and Low Dose CT Image and Projection Data (Moen et al., 2021) from https://doi.org/10.7937/9npb-2637 .
The 2016 Low Dose CT Grand Challenge data and Low Dose CT Image and Projection Data have the Custom license and the patient information is fully redacted.
We implemented all the methods speciﬁed in Section 4 by modifying the Noise2Self code (Batson & Royer, 2019)
(GitHub repository: https://github.com/czbiohub/noise2self with version Dec. 17, 2019) that is licensed under the MIT license.
For all training and testing experiments, we used Pytorch 1.0.0 or 1.7.0 (Paszke et al., 2019) with the BSD-style license.
For simulating low-dose FBP images, we used the Michigan image reconstruction toolbox (MIRT)
(Fessler, 2016) of which license information is declared on its release page.
For sinogram generation and
FBP reconstruction, we used the “ Gtomo2 dscmex.m ” routine (updated on Dec. 10, 2006) and the “ fbp2.m ” routine (updated on Dec. 21, 2005), respectively.
S.1. C OMMON IMPLEMENTATION DETAILS IN BOTH APPLICATIONS
For all the existing self-supervised denoising methods speciﬁed in Section 4 and Noise2True, we ﬁnely tuned their hyperparameters, including the initial learning rate, learning rate decay parameters, minibatch size, number of DnCNN layers, and balancing parameter σ (see, e.g., (9)), to achieve the best numerical results.
We simply applied the chosen hyperparameter sets to corresponding
SSRL setups.
We applied the chosen learning rate decay parameters, minibatch size, and number of
DnCNN layers in Noise2True experiments to all the self-supervised denoising methods.
For the existing self-supervised denoising methods, Noise2Self (Batson & Royer, 2019) and
Noise2Same (Xie et al., 2020), we used their default masking setups.
The Noise2Self default setup uses the deterministic masking scheme for each J that equi-spacedly samples 6 .
25% of the number of pixels in each training image (i.e., a single pixel is selected in each 4 × 4 window).
The
Noise2Same default setup uses the saturated sampling scheme (Xie et al., 2020; Krull et al., 2019) for each J that randomly samples ≈ 0 .
5% of the number of pixels in each training image (i.e., a single pixel is sampled in each 14 × 14 window).
In training denoising NNs, both methods interpolate missing pixels in x J c by applying weighted average to their 8 neighboring pixels, and use interpolated x J c as input to denoisers.
For the existing Noise2Noise-motivated methods that emulate pairs of two independent noisy images, Neighbor2Neighbor (Huang et al., 2021) and Noise2Inverse (Hendriksen et al., 2020), we calculated their loss with non-masked images as proposed.
We tested all trained regression NNs to non-masked images – rather than masked images with J c – as this setup gave higher denoising accuracy than prediction with masked images (Batson & Royer, 2019; Xie et al., 2020).
S.1. I MPLEMENTATION DETAILS FOR EXPERIMENTS WITH SIMULATED CAMERA IMAGE DENOISING DATA
The common hyperparameters for all learning methods were deﬁned as follows.
(In this application, these gave good image denoising performance across all existing self-supervised denoising methods and Noise2True, since we rescaled or normalized training images; see details below.)
We used the default 17 -layer DnCNN
(Zhang et al., 2017) and the modiﬁed U-Net used in Noise2Self (Batson & Royer, 2019), and trained all DnCNNs and U-Nets with the mini-batch version of Adam (Kingma & Ba, 2015).
We selected the initial learning rate, the batch size, and the number of epochs as 8 × 10 − 4 , 8 , and 190 , respectively, and decayed the learning rates by a factor of 0 .
5 every 50 , 000 iterations.
(For Neighbor2Neighbor (Huang et al., 2021), we set the batch size as 32 , as it reduces training image size with 2 × 2 sub-sampling window.)
We used the data augmentations in
Noise2Same (Xie et al., 2020), i.e., random crops with size 256 × 256 , rotation and ﬂipping.
Except for Noise2Same, we rescaled all images to [0 , 1] , following (Batson & Royer, 2019; Huang et al., 2021).
In Noise2Same experiments (including SSRL-Noise2Same), we normalized each image by subtracting its mean and dividing by its standard deviation at each channel, following (Xie et al., 2020).
For proposed SSRL in the Noise2Self and Noise2Same setups (referred to as SSRL-Noise2Self and
SSRL-Noise2Same, respectively)
, we used the deterministic masking scheme in Figure 2(bottom) for each J with ≈ 11 .
and ≈ 1 . 2% sampling ratio, respectively – i.e., a single pixel is selected in each 3 × 3 and 9 × 9 window, respectively.
We observed in this application that using sufﬁcient amount of information for a linear interpolation in f is useful for giving good prediction.
For weighted median ﬁltering (Brownrigg, 1984) g in all SSRL setups, we used the following weights: [1 , 2 , 1; 2 , 9 , 2; 1 , 2 , 1] , where a box denotes the central weight.
The dilation rates of weighted median ﬁltering for SSRL in the Noise2Self, Noise2Same, and Neighbor2Neighbor setups are 3 , 9 , and 1 , respectively, corresponding to the distances between pixels in J .
For SSRL-Noise2Self, we computed L ind in (7) only on J (see Figure 2(bottom)).
For SSRL-Noise2Same, we used the same balancing parameter σ as Noise2Same used, i.e., σ = 1 , and computed L (both terms) in (9) only on J .
The DnCNN and U-Net training time for each experiment was less than 72 hours with an NVIDIA
TITAN V GPU.
S.1. I MPLEMENTATION DETAILS FOR EXPERIMENTS WITH SIMULATED LOW - DOSE CT DENOISING DATA
We used fan-beam geometry for sinogram simulation, where width of each detector column is 1 .
2858 mm, source to detector distance is 1085 .
6 mm, and source to rotation center distance is 595 mm.
For the FBP method, we used a ramp ﬁlter because in general, it better preserves the sharpness of edges on reconstructed images than Hanning ﬁlter
(but overall noise increases).
The common hyperparameters for all learning methods were deﬁned as follows.
We used 8 -layer
DnCNN (Zhang et al., 2017) with its default setup and the modiﬁed U-Net used in Noise2Self (Batson & Royer, 2019), and trained all DnCNNs and U-Nets with the mini-batch version of Adam (Kingma & Ba, 2015).
We selected the batch size and the number of epochs as 2 and 1 , 000 , respectively, and decayed the learning rates by a factor of 0 .
95 every 10 epochs.
In training DnCNNs and U-Nets, we selected the initial learning rates as 0 .
1 and 5 × 10 − 5 , respectively, unless stated otherwise.
For proposed SSRL-Noise2Self and SSRL-Noise2Same, we used complementary checkboard masks
J and J c in Figure 2(top).
We observed in this application that if g is set to use small amount of information, i.e., | J | (cid:28) | J c | , then pre-trained g makes poor prediction.
For SSRL-Noise2Self, we set g as pre-trained NN by Noise2Self with complementary checkerboard masks (see its inference results with 8 -layer DnCNN and U-Net in Figures S.6(a) and S.4, respectively).
In training DnCNNs, we used the same initial learning rate as that used in pre-training g , 0 . 01 .
We computed L ind as given in (7) (see Figure 2(top)).
For SSRL in the Noise2Inverse setup, we set f and g as f/ 2 and I − g/ 2 , respectively, where g is pre-trained NN by Noise2Inverse.
In training DnCNNs, we used the same initial learning rate as that used in pre-training g , 0 . 001 .
In inference, we averaged the predictions from f and g , as this corresponds to training setup above.
In all Noise2Inverse inferences (including SSRLNoise2Inverse), we input full-view FBP images since this is consistent with other experiments and gave better denoising performance than denoising half-view FBP images.
For SSRL-Noise2Same, we set g as pre-trained NN by Noise2Same with complementary checkerboard masks (see its test results with DnCNN and U-Net in Figures S.6(b) and S.4, respectively), and computed L (both terms) in (9) only on J c .
We chose the balancing parameter σ as 15 , setting the ratio of the ﬁrst term to the squared second term 2 σ √ M (cid:107) f
( x ) J c − f ( x J c ) J c (cid:107) 22 in (9) as 10 .
For Noise2Same with either the default setup and complementary checkerboard masks, we chose the balancing parameter σ as 500 .
For self-supervised denoising methods with complementary checkerboard masks, we interpolated missing pixels in both x J c and x J , by averaging their 4 neighboring pixels.
The DnCNN and U-Net training time for existing self-supervised denoising methods and proposed
SSRL methods was less than 10 hours and 12 hours, respectively, with an NVIDIA TITAN
Xp GPU.
It took total less than 22 hours to train both f and g .
S. S UPPLEMENTARY EXPERIMENTAL RESULTS FROM SIMULATED DATASETS
This section mainly includes supplementary materials to Section 4.
Figures S.1 and S.2 show error maps of denoised images via DnCNNs from different learning methods in camera image denoising in mixed noise and low-dose CT, respectively. These show for both applications that in all the three comparison setups, SSRL signiﬁcantly reduces errors and artifacts across the entire image, over existing self-supervised denoising methods. Red boxes compare existing self-supervised denoising method to proposed SSRL in the corresponding setup. Figures S.3 and S.4 show denoised images via U-Nets from different learning methods in camera image denoising in mixed noise and low-dose CT, respectively.
These demonstrate for both applications that in all the three comparison setups, SSRL signiﬁcantly improves existing self-supervised denoising methods regardless of the regression neural network architecture.
Tables S.1–S.3 report quantitative image denoising results with DnCNN and U-Net with BSDand Set 5 data in camera image denoising in mixed noise, and with chest slices of The 2016 Low Dose CT Grand Challenge data in low-dose CT. Red boxes in Tables S.1–S.3 compare an existing self-supervised denoising method to proposed SSRL in the corresponding setup. Figure S.5 compares prediction uncertainty of trained DnCNN denoisers via different learning methods in both applications.
The error bar graphs in Figure S.5 show that for both applications, in all the three comparison sets, proposed SSRL gives similar or lower prediction uncertainty over the
[Table]
existing self-supervised denoising methods, Noise2Self, Neighbor2Neighbor, Noise2Inverse, and
Noise2Same. Figure S.6 shows denoised images from Noise2Self and Noise2Same with DnCNN and complementary checkerboard masks J and J c , and reports the corresponding quantitative test results, in low-dose CT denoising. Comparing the results in Figure S.6 to those of Noise2Self and Noise2Same using the default masking setups in Figures 5 and S.2 shows that checkerboard masking improves denoising quality over default masking in Noise2Self, and achieves comparable denoising performance to default masking in Noise2Same. (See their default masking setups in Section S.1.2.)
SSRL-Noise2Self with checkerboard masking achieves signiﬁcantly better denoising quality compared to Noise2Self with checkerboard masking.
We used pre-trained DnCNN from these two setups as g in the corresponding SSRL setup. Figure S.7 empirically supports our claim that g design “approximately” satisfy E [ g ( x ) | y ] = y in natrual image denoising in Section 3.1. We calculated empirical E [ x − y | y ] and E [ g ( x ) − y | y ] with simulated noisy BSD 300 data, where g is median ﬁltering.
The empirical measures for avg( | E [ x − y | y ] | ) and avg( | E [ g ( x ) − y | y ] | ) are 0 . 0201 and 0 . 0098 , respectively, supporting the
[Table]
[Figure]
PSNR
[Figure]
dB
SSIM
[Figure]
PSNR =
[Figure]
dB SSIM = 0 . PSNR =
[Figure]
1 dB
SSIM = 0 .
PSNR
[Figure] [Figure]
dB
SSIM = 0 .
[Table]
[Figure]
claim that E [ g ( x ) | y ]
= y is approximately satisﬁed with median ﬁltering g . Here, avg denotes the averaging operator. Figure S.8 empirically supports our claim that noise of FBP-reconstructed images in low-dose CT, i
[Figure]
e., e in (11), has approximately zero-mean. The position of the patient table base is similar across Reference
[Figure]
SSIM
[Figure]
PSNR = 26 . 0 dB
SSIM = 0 .
[Figure]
SSIM
[Figure]
PSNR = 20 . 9 dB SSIM = 0 .
[Figure]
PSNR
[Figure]
dB
SSIM
[Figure]
[Figure]
PSNR =
[Figure]
dB
SSIM =
[Figure]
703 PSNR = 20 . 8 dB SSIM = 0 . PSNR =
[Figure]
dB SSIM = 0 . 756 PSNR = 22 . 4 dB
SSIM = 0 .
PSNR = 19 . 5 dB
SSIM =
[Figure]
Reference
[Table]
RMSE = 18 .
5 HU
RMSE = 18 .
5 HU
[Table]
RMSE =
32 . 3 HU
RMSE =
32 . 3 HU
[Table]
RMSE = 26 .
7 HU
RMSE = 26 .
7 HU
[Table]
RMSE = 24 .
HU
RMSE = 24 .
HU
RMSE =
[Table]
5 HU
RMSE = 23 .
5 HU
RMSE = 23 .
5 HU
RMSE
[Table]
RMSE = 31 .
1 HU
RMSE = 31 .
1 HU
RMSE
[Table]
RMSE = 28 .
2 HU
RMSE = 28 .
2 HU
[Table]
[Figure]
RMSE = 30 .
9 HU
RMSE = 30 . 9 HU
[Figure]
RMSE = 28 .
7 HU
RMSE = 28 .
7 HU
FBP images, so it gave higher errors in the calculated sample mean; see the bottom of the image in
Figure
The following empirical results support the claim in Section 3.2 that pre-trained g will give better reference than g
= I in low-dose CT denoising: the empirical measure of the Noise2Self loss (Batson & Royer, 2019) – E (cid:107) g
( x J ) J c − x J c (cid:107) 22 – with simulated noisy CT test samples by setting g as I and pre-trained DnCNN by Noise2Self are 2455 .
7 and 1839 .
1 (in HU 2 ), respectively.
S. S UPPLEMENTARY EXPERIMENTAL RESULTS FROM REAL - WORLD DATASETS
Figures S.9 and S.10 show error maps of denoised images via DnCNNs from different learning methods in camera image denoising and low-dose CT with real-world datasets, respecitively.
These show for both applications that in Neighbor2Neighbor, Noise2Self, or Noise2Same comparison setups, SSRL signiﬁcantly improve the entire image without having their exact noise properties.
S. P RELIMINARY RESULTS WITH THE TEACHER - STUDENT LEARNING PERSPECTIVE
The proposed SSRL framework is applicable to teacher-student learning (Wang & Yoon, 2021; Hinton et al., 2015; Buciluˇa et al., 2006) that aims to learn a smaller student network from bigger teacher network(s).
We ran preliminary experiments in self-supervised low-dose CT denoising (using simulated data).
The teacher model g is the pre-trained 8 -layer DnCNN by Noise2Self (with checkerboard masking), and we set the student model f as { 8 , 7 , 6 , 5 , 4 , 3 } -layer DnCNN.
Applying SSRLNoise2Self, we obtained the following numerical results: the RMSE (HU) values of student models [Figure] [Figure] with { 8 , 7 , 6 , 5 , 4 , 3 } -layer DnCNNs are { 25 . 0 , 25 . 3 , 25 . 5 , 25 . 4 , 25 . 2 , 27 . 5 } , respectively.
The student DnCNNs that have the equal or lower complexity compared to its teacher network, signiﬁcantly improves its teacher model of which RMSE value is 30.9 (in HU).
The results might imply that student models learned from SSRL can outperform their teacher model, if they retain sufﬁciently high network complexity as compared to their teacher’s
(e.g., 3 -layer DnCNN).
In addition, we have additional SSRL experiment in low-dose CT denoising with the “iterative” teacher-student perspective.
The teacher model g is pre-trained 5 -layer DnCNN from the non-iterative teacher-student SSRL method above, and we set the student model f as 3 -layer DnCNN.
We obtained 27 .
3 RMSE (in
HU), implying only marginal improvement over the 3 -layer DnCNN obtained by the non-iterative teacher-student SSRL method.
We conjecture that iterative teacher-student SSRL needs more sophisticated g -setups, such as an ensemble of teacher models (Hinton et al., 2015).