Online Deep Equilibrium Learning for Regularization by Denoising
Abstract
Plug-and-Play Priors (PnP) and Regularization by Denoising (RED) are widelyused frameworks for solving imaging inverse problems by computing fixed-pointsof operators combining physical measurement models and learned image priors.
While traditional PnP/RED formulations have focused on priors specified usingimage denoisers, there is a growing interest in learning PnP/RED priors that areend-to-end optimal.
The recent Deep Equilibrium Models (DEQ) framework hasenabled memory-efficient end-to-end learning of PnP/RED priors by implicitlydifferentiating through the fixed-point equations without storing intermediate activation values.
However, the dependence of the computational/memory complexityof the measurement models in PnP/RED on the total number of measurementsleaves DEQ impractical for many imaging applications.
We propose ODER as anew strategy for improving the efficiency of DEQ through stochastic approximations of the measurement models.
We theoretically analyze ODER giving insightsinto its convergence and ability to approximate the traditional DEQ approach.
Ournumerical results suggest the potential improvements in training/testing complexitydue to ODER on three distinct imaging applications.
Introduction
There has been considerable recent interest in using deep learning (DL) in the context of imaginginverse problems (see recent reviews [1–3]).
Instead of explicitly defining a regularizer, the traditional DL approach is based on training a convolutional neural network (CNN) architecture, such as UNet [4], to invert the measurement operator by exploiting the natural redundancies in the imagingdata [5–9].
Plug-and-Play Priors (PnP) [10] and Regularization by Denoising (RED) [11] are twowell-known alternative approaches to the traditional DL that enable the integration of pre-trained CNNdenoisers, such as DnCNN [12] or DRUNet [13], as image priors within iterative algorithms.
Whenequipped with advanced CNN denoisers, PnP/RED provide excellent performance by exploiting boththe implicit prior, characterized by a denoiser, and the measurement model [14–22].
Deep Unfolding(DU) is a related approach that interprets the iterations of an image recovery algorithm as layers of aneural network and trains it end-to-end in a supervised fashion.
Unlike in PnP/RED, the CNN in DUis trained jointly with the measurement model, leading to an image prior optimized for a given inverseproblem [23–29].
DU architectures, however, are usually limited to a small number of unfoldediterations due to the high computational and memory complexity of training.
Recent work on Neural ODEs [30–32] and Deep Equilibrium Models (DEQ) [33–38] has shownthe potential benefits of implicit neural networks in a number of DL tasks.
For example, DEQ was recently used to train CNN priors within PnP/RED iterations by differentiating through the fixed points of the corresponding iterations [35].
Training PnP/RED using DEQ is equivalent to trainingan infinite depth feedforward network integrating a physical measurement model and CNN prior.
However, the training of such networks can still be a significant computational and memory challengein applications that require processing of a large number of sensor measurements.
Specifically, thedata-consistency layers in [35] are based on batch processing, which means that the entire set ofmeasurements is processed at each layer.
While this type of batch data processing is known to besuboptimal in traditional large-scale optimization [39–42], the issue has never been considered in thecontext of training of implicit networks such as those specified via PnP/RED iterations.
This paper addresses this issue by proposing Online Deep Equilibrium RED (ODER) as the first
DEQ framework for inverse problems that adopts stochastic processing of measurements withinan implicit neural network.
We argue that the proposed online approach can improve training andtesting efficiency compared to its batch counterpart in a number of applications where the numberof measurements is large.
ODER can be implemented using the fixed-point iterations of RED byintroducing stochastic approximations to the corresponding forward and backward DEQ passes.
The CNN prior within ODER is trained end-to-end to remove artifacts due to the imaging systemand stochastic processing.
We present theoretical insights into the convergence of forward andbackward passes of ODER, and show its ability to approximate learning using the traditional batch DEQ approach.
We show the practical relevance of ODER by solving inverse problems in intensitydiffraction tomography (IDT) [21,43], sparse-view computed tomography (CT) [44] and acceleratedparallel magnetic resonance imaging (MRI) [45,46].
Our numerical results show the ability of ODERto match the imaging quality of the batch DEQ learning at a fraction of complexity.
Our work thusaddresses an important gap in the current literature on PnP/RED, DU, and DEQ by providing anefficient framework applicable to a wide variety of imaging inverse problems.
All proofs and some technical details that have been omitted for space appear in the appendix, whichalso provides more background and simulations.
Background
Inverse problems.
Many imaging problems—such as IDT, CT, and MRI—can be formulated as aninverse problem involving the recovery of an image x ∗ ∈ R n from noisy measurements y
=
Ax ∗ + e ,where A ∈ R m × n is the measurement operator and e ∈ R m is the noise.
A common approach toestimate x ∗ is to solve an optimization problem
[Equation]
where g is a data-fidelity term that quantifies consistency with the observed data y and h is aregularizer
that encodes prior knowledge on x .
A widely-used data-fidelity term and regularizer ininverse problems are g ( x ) = 12 ∥
y − Ax ∥ 22 and the total variation (TV) function h ( x )
=
τ ∥ Dx ∥ 1 ,where D is the gradient operator and τ > 0 is the regularization parameter [47–49].
PnP, RED, and DU.
PnP [10,15] and RED [11] are two related classes of iterative algorithms thatuse additive white Gaussian noise (AWGN) denoisers, such as BM3D [50] or DnCNN [12], as priorsfor inverse problems (see the recent review [51]).
Since for general denoisers PnP/RED do not solvean optimization problem [18], it is common to interpret PnP/RED as fixed-point iterations of somehigh-dimensional operators.
For example, given a denoiser D θ : R n → R n parameterized by a CNNwith weights θ , the steepest descent variant of RED (SD-RED) [11] can be written as
[Equation]
) with G
[Equation]
x
[Equation]
x
[Equation]
where g is the data-fidelity term, and γ, τ > 0 are the step size and the regularization parameters,respectively.
SD-RED thus seeks to compute a fixed-point x ∈ R n of the operator
Tx ∈ Fix ( T θ ) :=
[Equation]
The solutions of (3) balance the requirements to be both data-consistent (via ∇ g ) and noise-free(via ( I − D θ ) ), which can be intuitively interpreted as finding an equilibrium between the physical measurement model and learned prior model.
Remarkably, this heuristic of using denoisers notnecessarily associated with any h within an iterative algorithm exhibited great empirical success [20,22,52–59] and spurred a great deal of theoretical work on PnP/RED [14,17–19,60–66].
It is worthmentioning that there has been considerable effort in reducing the test-time computational/memorycomplexity of PnP/RED by designing online and stochastic PnP/RED algorithms [21,61,65,67].
DU (also known as algorithm unrolling ) is a DL paradigm that has gained popularity due to its abilityto systematically connect iterative algorithms and deep neural network architectures (see reviewsin [3,68]).
Many PnP/RED algorithms have been turned into DU architectures by parameterizingthe operator D θ as a CNN with weights θ , truncating the PnP/RED algorithm to a fixed number ofiterations, and training the corresponding architecture end-to-end in a supervised fashion.
Recentwork has explored strategies for reducing the memory and computational complexity of training DUarchitectures [69,70], However, a key bottleneck in DU training is the necessity to store the intermediate activation values required for computing the backpropagation updates, which fundamentallylimits the number of unfolding layers one can practically use in large-scale applications.
DEQ.
DEQ [33] is a recent method for training infinite-depth, weight-tied feedforward networks byanalytically backpropagating through the fixed points using implicit differentiation.
The DEQ outputis specified implicitly as a fixed point of an operator T θ parameterized by weights θx = T θ ( x ) .
The DEQ forward pass estimates x in (4) by either running a fixed-point iteration or using anoptimization algorithm.
The DEQ backward pass produces gradients with respect to θ by implicitlydifferentiating through the fixed points without the knowledge of how they are estimated
[Equation]
where ℓ is the loss function, x ∗ is the training label, and I is the identity mapping.
The vector productwith the inverse-Jacobian in (5) can be approximated by solving the following fixed-point equation
[Equation]
Recent work has also explored Jacobian-free DEQ by replacing the inverse-Jacobian with an identitymapping I , leading to a faster training [37].
The comparison of equations (2), (3), and (4) highlights an elegant connection between PnP/RED and
DEQ.
This connection was explored in the recent work [35] by using DEQ for learning the weightsof the CNN prior D θ end-to-end within PnP/RED iterations.
Within the framework of [35], PnP/REDis used for the forward pass and a backward pass is obtained by using (6) on the PnP/RED operators.
Specifically, the CNN prior in SD-RED can be trained by running the backward pass using T θ in (2)
[Equation]
This work makes several new contributions to the existing literature on PnP/RED, DU, and DEQ.
The focus is to explore the impact of approximating T θ in (4) with a “simpler” operator (cid:98) T θ .
Following [35], we focus on inverse problems by using PnP/RED operators of form (2) that integratethe physical measurement models and learned CNN priors.
We give algorithmic, theoretical, andnumerical evidence that one can achieve significant memory/computational gains, while preservingthe performance of the original DEQ approach in [35].
It is worth noting that the results here have thepotential to generalize to many other implicit neural networks beyond those specified via PnP/RED.
Online Deep Equilibrium Method
We consider inverse problems where the data-fidelity term g can be expressed as
[Equation]
where each g i depends only on the subset
y i ∈
R m i of the full measurements y ∈ R m as
[Equation]
with m = m 1 + m 2 + · · · + m b .
We are primarily interested in scenarios where the memory/computational complexity of the gradient∇ g is proportional to b .
Thus, when b → ∞ , the memory and computational complexity of traditional
DEQ to train the CNN prior within the batch PnP/RED algorithms becomes impractical.
To decouple the computational/memory complexity of DEQ from b , we adopt online processing ofmeasurements, where g is approximated using a minibatch of w ≪ b measurements
[Equation]
where { i 1 , . .
, i w } are i.i.d random variables selected uniformly from the set { 1 , . . .
, b } .
Notethat (9) directly implies the unbiasedness of the online gradient E [ ∇ (cid:98) g
( x )]
= ∇ g ( x ) and Hessian 124 E
[ H (cid:98) g ( x )]
=
H g ( x ) with the expectations taken over the random indices { i 1 , . .
, i w } .
3.1 Forward Pass
The forward-pass of ODER is performed as follows
[Equation]
where R θ = I − D
θ is the residual of the CNN prior D θ .
The residual R θ takes artifact-corruptedimages at the input and produces the corresponding artifacts at the output.
Note how the ODERforward-pass is independent of b since it uses a minibatch approximation ∇ (cid:98) g in (9).
It is worth mentioning that when considered separately from DEQ, the forward pass of ODERcorresponds to the existing online RED algorithm [21, 71].
The contribution of this work is thusnot the forward pass, but the integration of the online forward pass into the DEQ framework via theonline backward pass, resulting in a more scalable and flexible DEQ framework for inverse problems.
3.2 Backward Pass
The backward pass of ODER for the MSE loss in (5) is performed as follows
[Equation]
starting from b 0 = 0 , where x K is the final iterate of the forward pass (10) at iteration K ≥ 1 .
In both traditional and online backward passes, conventional auto-differentiation tools enable thecomputation of the Jacobian-vector products in (7) and (11).
However, the key difference is that thecomputational complexity of ODER does not depend on the total number of measurements b .
3.3 ODER Learning
ODER seeks to minimize the MSE loss over p ≥ 1 training samples [Equation] [Equation]
using approximate gradients ∇ ˆ ℓ j ( θ ) computed via the online forward and backward passes that areindependent of b .
Here, θ denotes the weights of the CNN prior, x ∗ j is the j th training label, andx j ( θ ) is the fixed-point of the full-batch SD-RED algorithm (3).
The backward pass of ODER canbe integrated within any gradient-based optimizer, such as the stochastic gradient descent (SGD) .
At training iteration t ≥ 1 we generate two sets of independent random variables.
First, the indexj t is selected uniformly at random from { 1 , . . .
, p } , then online forward and backward passes arecomputed using the measurement models in (9).
We can thus express the SGD update rule as follows
[Equation] where β > 0 is the SGD learning rate, x Kj t and b Kj t are the final iterates of the ODER forward andbackward passes at the training index j t after K ≥ 1 iterations.  Theoretical Analys Our theoretical analysis relies on a set of explicit assumptions serving as sufficient conditions.
Theproofs of all the theorems will be provided in the supplement.
Assumption 1.
Each g i is twice continuously differentiable and convex.
There exists λ >
such thateach gradient ∇ g i and Hessian H g
i are λ -Lipschitz continuous.
The fact that g is twice continuously differentiable is needed for the backward pass.
The assumptionthat all the Lipschitz constants are the same is only needed to streamline mathematical exposition.
Assumption 2.
D θ ( x ) is continuously differentiable with respect to θ and x .
There exists α > 0 suchthat D θ ( x ) , ∇ x D θ ( x ) , and ∇ θ D θ ( x ) are α -Lipschitz continuous with respect to θ and x .
Finally,we also assume that D θ is a contraction, which means that there exists κ < 1 such that
[Equation]
Since D θ is a CNN, its differentiability is a standard assumption.
The contractive D θ and convex g ,ensure that T θ is a contraction, enabling provable convergence of the forward and backward passes.
The design of contractive T θ is a common PnP/RED strategy to ensure convergence [19,35,65].
Assumption 3.
There exists R > 0 such that for all x ∈ Fix ( T ) and b ∈ Fix ( F ) , we have∥ x
k − x ∥ 2 ≤ R and ∥
b k − b ∥ 2 ≤ R for all k ∈ { 1 , . . .
, K } .
The existence of the bound R is reasonable, as many images have bounded pixel values.
Similarly,the bound on b k is also reasonable for ensuring bounded DEQ gradients.
Assumption 4.
There exists ν > 0 such that for all x ∈ R n , we have
[Equation]
and E (cid:2) ∥ H [Equation] ( x
[Equation]
where the expectations are taken over { i 1 , . .
, i w } .
The variance bounds are standard in stochastic algorithms.
The variance bounds on the gradient and
Hessian approximations are thus reasonable in this context.
The decrease of the bounds for highervalues of w is natural since (cid:98) g is an unbiased estimator of g obtained averaging w independent terms.
Theorem 1.
Run the forward pass of ODER for k ≥ 1 iterations under Assumptions 1-4 using thestep size 0 < γ < 1 / ( λ + τ ) .
Then, the sequence of forward pass iterates satisfies
[Equation]
for some constant 0 < η < 1 where x ∈ Fix ( T ) .
Theorem 1 is a variation on the convergence results for online RED/PnP [21,61,65,67], showing thatthe forward pass converges to x ∈ Fix ( T ) up to an error term that can be controlled via γ and w .
Theorem 2.
Run the backward pass of ODER for k ≥ 1 iterations under Assumptions 1-4 fromb 0 =
using the step-size 0 < γ < 1 / ( λ + τ ) .
Then, the sequence of backward pass iterates satisfies
[Equation]
where 0 < η < 1 , B 1 > 0 and B 2 > 0 are constants independent of k and w , and b ∈ Fix ( F ) .
Theorem 2 shows that the online backward pass in expectation converges to b up to an error term thatcan be controlled via w .
The complete expressions for constants B 1 and B 2 are in the proof.
Assumption 5.
Function ℓ has a global minimizer θ ∗ and has a L -Lipschitz continuous gradient,which means that for all θ , ϕ , we have ∥∇ ℓ ( θ ) − ∇ ℓ ( ϕ ) ∥ 2 ≤ L ∥ θ − ϕ ∥ 2 .
Assumption 6.
The loss function in (12) and indices { j t } in (13) are such that
[Equation]
where the expectations are taken with respect to the random index uniformly as j t ∈ { 1 , . .
. , p } .
The existence of a minimizer and the Lipschitz continuity of the loss gradient are standard assumptionsin the literature [42,72,73].
Note that we do not assume that the training loss ℓ is convex.
Assumption 6is the standard assumption used in the analysis of SGD.
Theorem 3.
Train ODER using SGD for T ≥ 1 iterations under Assumptions 1-6 using the step-sizeparameters 0 < β ≤ 1 /L and the minibatch size w ≥ 1 .
Select a large enough number of forwardand backward pass iterations K ≥ 1 to satisfy 0 < η K ≤ 1 / √ w .
Then, we have that
[Equation]
where C 1 > 0 and C 2 > 0 are constants independent of T and w .
The complete expressions for constants C 1 and C 2 are in the proof.
Theorem 3 allows us to understandthe ability of the iterates generated using (13) to approximate the stationary points of the desiredloss (12).
The error terms in the bound depend on the training step-size β and the minibatch size w ,both of which can be controlled during training.
In summary, we presented several theoretical resultsthat give insights into ODER by stating explicit sufficient conditions and approximation bounds.
Numerical Evaluation
We numerically validate ODER in the context of three computational imaging modalities: IDT,sparse-view CT, and parallel MRI.
Our goal is to both (a) empirically evaluate the performance of
ODER and (b) highlight its effectiveness for processing a large number of measurements.
We adoptℓ 2 -norm loss g ( x ) =
12 ∥
y − Ax ∥ 22 as the data-fidelity term for all three imaging modalities.
ODER is compatible with any CNN architecture used to implement D θ .
We use a tiny U-Net architecture [70] for ODER and the traditional RED (DEQ) [35].
We have added spectral normalization [74]to all the layers of CNN for stability (see the supplement for the numerical evaluation of the contractiveness of T θ on all three modalities).
Similar to [35], the CNN prior of ODER and RED (DEQ) areinitialized using pre-trained denoisers.
During the training of both ODER and RED (DEQ), we usethe Nesterov acceleration [72] for the forward pass and Anderson acceleration [75] for the backwardpass.
We also adopt the stopping criterion from [35,76] by setting residual tolerance to 10 − 3 for bothforward and backward iterations (see supplement for additional details).
For reference we include several other well-known baseline methods, including TV [47], U-Net [4]and ISTA-Net + [23].
We also include the unfolded RED (Unfold) [70] and the traditional RED(Denoising)
[11] to illustrate the improvements due to DEQ.
TV is an iterative method that doesnot require training, while other methods are all DL-based with publicly available implementations.
We use the U-Net architecture in [4] as the AWGN denoiser for RED, while we use the same tiny
U-Net for RED (Unfold) as in RED (DEQ).
For each imaging modality, we trained the denoiser in
RED (Denoising) for AWGN removal at five noise levels corresponding to σ ∈ {2, 7, 5, 10, 15}.
Foreach experiment, we select the denoiser achieving the highest SNR.
In all the experiments, we train
ODER and RED (DEQ) using the same training strategy and parameter initialization settings.
We usefminbound in the scipy.optimize toolbox to identify the optimal regularization parameters for
TV, RED (Denoising), ODER and RED (DEQ) at the inference time.
5.1 Image Reconstruction in IDT
IDT [43] is a data intensive computational imaging modality that seeks to recover the spatialdistribution of the complex-valued permittivity contrast of an object given a set of its intensity-onlymeasurements.
Specifically, A consists of a set of b complex measurement operators [ A 1 , . . .
, A b ] T ,where each A i is a convolution corresponding to the i th measurement y i .
In the simulation, werandomly extracted and cropped 400 slices of 416 × 416 images for training, 28 images for validationand 56 images for testing from Brecahad database [77].
Following the setup in [21,43], we generatedb =
500 intensity measurements under AWGN corresponding to { 15 , 20 , 25 } dB of input SNR.
ODER and RED (DEQ) were trained at the noise level corresponding to 20 dB input SNR.
In our
[Table]
.966 e
[Table]
T i m e ( h r )
Training loss against time
Training time against epoch
Time (hr)
[Table]
dB
[Table]
N R ( d
B )
Training SNR against time
Time (hr)
Epoch
S N
[Table]
( d
B )
Epoch Epoch Times (
[Table]
L o ss 1.966 e !"
[Table]
/+kiXEh12n2PPhPVtYosYpQTR8c4COkCD1QsYu9Pd/bO9md18v34wen6xcIPGv1hIJad4yvolh4lErdMMmvkLgOlj8hfd2torPAn1S5c44mMj5kGQLNYH4bNVLJcDxuH+zhK8Le3ymf5VSb5IoApiU5Fr/TOYWQVdrFkH6JWzj/ObWZPHHr+XBV3d2Jh/ufPDFna1PPhh+GLsxenv019Ht0WT00eiT0cPR8eh0lI7+OfrX6N+j/2x+v/nD5k+bP/eiL7806GyOgs/m//4P6NZ/IQ==</latexit> R e l a t i v e
R e s i du a l k F ( b )  b kk b k
R e l a t i v e r e
s i d u a l
[Figure] [Figure]
[Table]
comparisons, we also included the recent SGD-Net [70] method that corresponds to RED (Unfold),but uses stochastic data-consistency layers similar to ODER.
SGD-Net allows for more unfoldediteration blocks by improving the usage of limited GPU memory.
Both ODER and RED (DEQ) weretrained using SGD, while all other methods were trained using Adam [78].
Fig.
1 compares the average loss and SNR achieved by RED (DEQ) with ( b = 500 ) and ODER withw ∈ { 100 , 250 } during training.
It took 67.49 hours to train RED (DEQ) for 180 epochs.
It took24.76 and 39.23 hours to train ODER with ( w = 100 ) and ( w = 250 ), respectively, for the samenumber epochs.
Table 1 provides the final SNR achieved by ODER and several baseline methods onthe test data.
The runtime in the table corresponds to the average inference time that excludes themodel loading.
ODER with ( w = 100 ) is around 3 × faster than RED (DEQ) on both GPU and CPU.
Fig.
2 ( left ) highlights the faster convergence of ODER compared to RED (DEQ) to the similar SNR.
ODER is memory efficient due to its online processing of measurements.
The memory considerationsin IDT include the size of all the variables related to the desired image x , the measured data y i , andthe variables related to the measurement operator { A i } .
ODER addresses the problem of storing andprocessing the measurements and the measurement operators on the GPU during end-to-end training.
Table 1 shows the total memory (GB) used by ODER and RED (DEQ) for reconstructing a 416 × 416pixel permittivity image.
While RED (DEQ) requires 3 .
56 GB of GPU memory in every iteration,
ODER with w = 100 requires only 0 .
71 GB, which is about 20% of the full volume.
5.2 Image Reconstruction in Sparse-View CT
We consider simulated data obtained from the clinically realistic CT images provided by Mayo
Clinic for the low dose CT grand challenge [79].
Specifically, 2070 2D slices of size 512 ×
512corresponding to 7 patients were used to train the models.
The test images correspond to 55 slicesrandomly selected from another patient.
We implement the measurement operator A and its adjoint
SNR VS Time 41.94dB 38 .
40 d B
Time (s)
S N R ( d B )
Testing SNR (dB) on IDT 35.95 dB
Time (s) 34.83 dB
Testing SNR (dB) on Sparse CT 27.41 dB 26.85 dB
Time (s)
Testing SNR (dB) on Parallel MRI
[Table]
A T with PyTorch implementation of Radon and IRadon 1 transform.
We assume that the CT machineis designed to project from nominal angles with b ∈ { 90 , 120 , 180 } projection views that are evenlydistributed on a half circle and 724 detector pixels.
We add Gaussian noise to the sinograms tomake the input SNR equal to 50 dB. We empirically found that using Adam [78] is around 2 × fasterthan applying SGD when training both ODER and RED (DEQ).
We thus trained all learning-basedmethods using Adam.
Table 2 reports the average SNR and SSIM results for ODER with ( w/b ) of{ 30 / 90 , 40 / 120 , 50 / 180 } and all baselines.
Fig.
2 (middle) reports the convergence speed of ODERwith ( w = 50 ) for sparse-view CT with full batch ( b = 180 ) views.
The visual comparisons are in
Fig.
3 (bottom).
Note how ODER matches the performance of RED (DEQ) and outperforms RED(Denoising) and RED (Unfold) across different projection views.
5.3 Image Reconstruction in Accelerated Parallel MRI
We simulated a multi-coil CS-MRI setup using radial Fourier sampling [80,81].
The measurementoperator A thus consists of a set of b complex measurement operators depending on a set of receivercoils { S i } [82].
For each coil, we have A i
= P F S i , where P is the diagonal sampling matrix, Fis the Fourier transform, and S i is the diagonal matrix of sensitivity maps.
ODER is evaluated ontwo brain MRI datasets.
The first dataset [23] provides 800 slices of 256 × 256 images for trainingand 50 slices for testing.
The second dataset [83] contains a randomly selected 400 volumes of320 × 320 × 10 images for training, and 32 volumes for testing.
We synthesized the total number of( b = 96 ) 2D/3D coil sensitivity maps using the SigPy [84] for each dataset, respectively.
Since allthe CNNs in our numerical study are 2D, we apply them slice-by-slice when forming 3D volumes(all slices are passed in parallel using batch processing).
We trained all learning-based methods using
Adam.
Fig.
2 (right) reports the convergence speed of ODER for CS-MRI at 20% sampling.
Table 3reports the average SNR and SSIM values for ODER with ( w = 48 ) and all baseline methods.
Thevisual comparison can be found in Fig.
3 (top) at 10% sampling.
Visual 29.
Ground truth
TV
RED (Denoising)
ODER (48/96)
RED ( DEQ ) RED ( Unfold ) 20.
Ground truth
TV
RED (Denoising)
ODER (30/90)
RED ( DEQ ) RED ( Unfold )
Ground truth
Back propagation
SGD-Net (100/500)
ODER (100/500)
RED ( DEQ ) Ground truth
TV
RED (Denoising)
ODER (250/500)
RED ( DEQ ) 0 0.
Method
[Table]
Time
[Table]
Conclusion and Future Work
This work proposes ODER as a new online DEQ learning method for RED, analyzes its theoreticalproperties in terms of convergence and accuracy [Table] and applies it to three widely-used imaging inverseproblems.
ODER extends the recent DEQ approach in [35] by introducing randomized processing ofmeasurements.
Our extensive theoretical and numerical results corroborate the potential of ODERto reduce the computational/memory complexity of training and testing, while achieving similarimaging quality as RED (DEQ).
The future work can explore to further improve our analysis anddesign distributed variants of ODER to enhance its performance on parallel computing architectures.
Broader impact
This work is expected to impact the area of imaging inverse problems with potential applicationsto computational microscopy, medical imaging [Table] and image restoration.
There is a growing needin imaging to deal with noisy and incomplete measurements by integrating multiple informationsources, including physical sensor models and learned priors characterizing the properties of thedesired image.
The ability to accurately solve inverse problems might lead to new imaging tools fordiagnosing health conditions, understanding biological processes, or inferring properties of complexmaterials.
Learning based methods, including PnP/RED and ODER, have the potential to enable newtechnological capabilities; yet, they also come with a downside of being more complex, requiringhigh-levels of technical sophistication from potential users.
While we aim to use our method to enablepositive contributions to humanity, one can also imagine nonethical usage of imaging technology.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] The main claims and contributions in this papers are
the new DEQ framework for inverse problems that adopts stochastic processing of
measurements within an implicit neural network, where we present both theoreticaland numerical analysis.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sections A-C in
the Supplement.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the Supplemental material or as a URL)?
[Yes] In theSupplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] In the Supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Partially no, since we report the variance over different
test images in Fig.
2.(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In the Supplement.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite those existing
assets in this work, including, baseline methods, training, and testing datasets.
(b) Did you mention the license of the assets?
[N/A] We only used publicly available
datasets.
(c) Did you include any new assets either in the Supplemental material or as a URL?
[No]In Supplement we only provide additional technical details and numerical results.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The datasets used in this work is public available and being
used in many existing works.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The bio-medical images used in this work are
publicly available and widely-used.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]