Improved Imaging by Invex Regularizers with Global Optima Guarantees
Abstract
Image reconstruction enhanced by regularizers, e.g., to enforce sparsity, low rank or smoothness priors on images, has many successful applications in vision tasks such as computer photography, biomedical and spectral imaging.
It has been well accepted that non-convex regularizers normally perform better than convex ones in terms of the reconstruction quality.
But their convergence analysis is only established to a critical point, rather than the global optima.
To mitigate the loss of guarantees for global optima, we propose to apply the concept of invexity and provide the first list of proved invex regularizers for improving image reconstruction.
Moreover, we establish convergence guarantees to global optima for various advanced image reconstruction techniques after being improved by such invex regularization.
To the best of our knowledge, this is the first practical work applying invex regularization to improve imaging with global optima guarantees.
To demonstrate the effectiveness of invex regularization, numerical experiments are conducted for various imaging tasks using benchmark datasets.
Introduction
Image reconstruction (restoration) enhanced by regularizers has a wide application in vision tasks such as computed tomography [1, 2], optical imaging [3, 4], magnetic resonance imaging [5, 6], computer photography [7, 8], biomedical and spectral imaging [9, 10].
In general, an image reconstruction task can be formulated as the solution of the following optimization problem:
[Equation]
Here f ( x ) models a data fidelity term, which usually corresponds to an error loss for image reconstruction, and is assumed to be differentiable.
The other function g ( x ) acts as a regularizer which can be non-smooth.
It imposes image priors such as sparsity, low rank or smoothness [11].
The use of an appropriate regularizer plays an important role in obtaining robust reconstruction results.
Convex regularization has been popular in the last decade [11, 12, 13, 14, 15], because it can result in guaranteed global optima.
The most well-known examples include the ℓ 1 -norm and nuclear norm, which are the continuous and convex surrogates of the ℓ 0 -pseudo norm and rank, respectively [16].
Although convex regularizers have demonstrated their success in signal/image processing, biomedical informatics and computer vision applications [13, 17, 18, 19], they are suboptimal in many cases, as they promote sparsity and low rank only under very limited conditions (more measurements from the scene are needed [20, 21]).
To address such limitations, non-convex regularizers have been proposed.
[Table]
For instance, several interpolations between the [-(cid:96) 
-]ℓ 0 -pseudonorm and the ℓ 1 -norm have been explored including the ℓ p -quasinorms
(where 0 < p < 1 ) [22], Capped- ℓ 1 penalty [23], Log-Sum Penalty [20], Minimax Concave Penalty [24], Geman Penalty [25].
However, these non-convex regularizers unfortunately come with the price of losing global optima guarantees.
Image reconstruction methods based on Eq.
(1) include model-based approaches that directly solve
Eq.
(1) using well-established optimization techniques, e.g., proximal operators and gradient descent rules [26, 27, 28], learning-based approaches that train an inference neural network [29, 30], as well as hybrid approaches that draw links between iterative signal processing algorithms and the layer-wise neural network architectures [31, 32].
Many of these exploit non-convex assumptions over f ( x ) and/or g ( x ) , for which we present a summary of some commonly used or successful ones in Table 1.
The table includes algorithms like the iterative reweighted least squares (IRLS) [33, 34], where the regularizer is a composition between the one-dimensional ℓ p -quasinorm and the trace of a matrix.
In [35, 36], the objective function F ( x ) is assumed to form a semi-algebraic or tame optimization problem solved by gradient descent algorithms.
In [37], the regularizer g ( x ) is assumed to be the subtraction of two convex functions, and the general iterative shrinkage and thresholding (GIST) algorithm is proposed to optimize F ( x ) .
Lastly, [38] assumes non-convex f ( x ) but convex g ( x ) and proposes the inertial proximal (iPiano) algorithm for optimization.
For algorithms with the convexity assumptions removed, e.g., those in Table 1, their convergence analysis unfortunately can only be established for a critical point.
Ideally, we always prefer algorithms that can find the optimal solution for the target problem.
One way to mitigate the loss of guarantees for global optima is by revisiting the concept of invexity which was first introduced by Hanson [39],
Craven and Glover [40] in the 1980s.
What makes this class of functions special is that, for any point where the derivative of a function vanishes (stationary point), it is a global minimizer of the function.
Convexity is a special case of invexity.
Since 1990s, a lot of mathematical implications for invex functions have been developed, but with the lack of practical applications [41].
Examples of the few successful works implementing the invexity theory include [42, 43, 44].
To the best of our knowledge, there is no existing work on the application of invex regularization for imaging.
In this paper, we focus on image reconstruction problems formulated in the form of Eq.
(1), where the data fidelity term f ( x ) is based on the ℓ 2 -norm and an invex regularizer g ( x ) is used.
Most invex theory research lacks clarity on how to benefit practical applications, and this does not encourage the practitioners to exploit the invex property [41].
We aim at filling this gap by providing for the first time concrete and useful invex optimization formulations for imaging applications.
Specifically, we make the following contribution:
• Provide the first list of regularizers with proved invexity that fits optimization problems for imaging applications.
• Establish convergence guarantees to global optima for three types of advanced image reconstruction techniques enhanced by invex reguarlizers.
• Empirically demonstrate the effectiveness of invex regularization for various imaging tasks.
Preliminaries
Throughout this paper, we use boldface lowercase and uppercase letters for vectors and matrices, respectively.
The i -th entry of a vector w , is w [ i ] .
For vectors, ∥ w ∥ p is the ℓ p -norm.
An open ball is defined as B ( x ; r ) = { y ∈ R n :
∥ y − x ∥ 2 < r } .
The operation conv ( A ) represents the convex hull of the set A , and the operation sign ( w ) returns the sign of w .
We use σ i ( W ) to denote the i -th singular value of W assumed in descending order.
We present several concepts needed for the development of this paper starting with the definition of a locally Lipschitz continuous function.
Definition 1 ( Locally Lipschitz Continuity ) .
A function f : R n → R is locally Lipschitz continuous at a point x ∈ R n if there exist scalars K > 0 and ϵ > 0 such that
[Equation]
for all y , z ∈ B ( x , ϵ ) .
Since the ordinary directional derivative being the most important tool in optimization does not necessarily exist for locally Lipschitz continuous functions, it is required to introduce the concept of subdifferential [45] which is calculated in practice as follows.
Theorem 1 ( Subdifferential ) .
[45, Theorem 3.9] Let f : R n → R be a locally Lipschitz continuous function at x ∈ R n , and define Ω f = { x ∈ R n | f is not differentiable at the point x } .
Then the subdifferential of f is given by
[Equation]
i → x and ∇ f
[Equation]
x
[Equation]
The notion of subdifferential is given for locally Lipschitz continuous functions because it is always nonempty [45, Theorem 3.3].
Based on these, the concept of invex function is presented as follows.
Definition 2 ( Invexity ) .
Let f : R n → R be locally Lipschitz; then f is invex if there exists a function η : R n × R n → R n such that
[Equation]
R [Equation] ∀ ζ ∈ ∂f ( y ) . It is well known that a convex function simply satisfies this definition for η ( x , y ) = x − y .
The following classical theorem [46, Theorem 4.33] makes connection between an invex function and its well-known optimum property that supports the motivation of designing invex regularizers.
Theorem 2 ( Invex Optimality ) .
[46, Theorem 4.33]) Let f : R n → R be locally Lipschitz.
Then the following statements are equivalent.
f is invex.
Every point y ∈ R n that satisfies 0 ∈ ∂f ( y ) is a global minimizer of f . 3.
Definition 2 is satisfied for η : R n × R n → R n given by
[Equation]
where ζ ∗ y is an element in ∂f ( y ) of minimum norm.
Invex Functions
We start this section by firstly presenting five examples of invex functions that are useful for imaging applications.
Four of these have been labelled as non-convex in existing works [47, 48].
This is the first time that they are formally proved to be invex functions.
We prove their invexity by showing they satisfy Statement 2 of Theorem 2 (see proof in Appendix A of supplementary material).
Lemma 1 ( Invex Functions ) .
All of the following functions are invex:
[Equation]
[Equation]
g ( x ) = n (cid:88) i =
log(1 + | x
[ i ] | ) − | x
[ i ] | 2 + 2 | x [ i ] | .
We provide further insights of these functions in Section A.1 of Supplemental material.
Tablesummarizes their applications.
Specifically, Eq.
(6) is known as quasinorm, and has attracted a lot of attention because it has resulted in theoretical improvements for matrix completion and compressive sensing [22, 49].
The analysis on the quasinorms is valid with and without the constant ϵ .
We prefer to add ϵ in order to formally satisfy the Lipschitz continuity in Definition 1.
Eqs.
(7) and (8) enhance the convex ℓ 1 -norm regularizer, and they have significantly improved image denoising [47].
Eq.
(9) has been used as the loss function to improve support vector classification [50].
[Table]
We propose the last function in Eq.
(10) by the subtraction between Eq.
(7) and Eq.
This design is motivated by the optimization framework in [37] where the regularization term is assumed to be the subtraction of two convex functions (see GIST in Table 1).
This has been found to be highly successful in imaging applications (see the survey [48]).
But until now there is no evidence that this subtraction produces another convex function (if exists) potentially useful in imaging applications.
Therefore, we propose this example to show that at least this is possible in the invex case.
Additionally, we present another way of constructing an invex function in the following lemma.
It establishes that an invex function f : R m → R composed with an affine mapping Hx − b for
H ∈ R m × n , x ∈ R n and b ∈ R m , is also invex if H is full row-rank.
This condition on H is a mild assumption, because we show in Section 4 imaging application examples that satisfy this criterium.
Lemma 2 ( Affine Invex Construction ) .
Let f : R m → R be a continuously differentiable invex function, H ∈ R m × n have full row rank, and b ∈ R m be a vector.
Then the function h ( x ) =
f
( Hx − b ) is invex.
Similar to Lemma 1, it is proved by showing that the composed function satisfies Statement 2 of
Theorem 2 (see Appendix B of supplementary material).
Eq.
(9) is an example of such an invex construction that satisfies the continuously differentiable assumption in Lemma 2, easily verified its proof in Appendix B. A practical implication of Lemma 2 for imaging applications appears when we want to solve linear system of equations (e.g. [50]).
We demonstrate an application of this kind of invex construction in Section 4.2.2 to improve a widely used image restoration framework.
Invex Imaging Examples, Algorithms and Convergence Analysis
In this section, we demonstrate the use of invex regularizers to improve some advanced imaging methodologies.
To benefit both practitioners and theory development, we present practical invex imaging algorithms and prove their convergence guarantees to global optima which was only possible for convex functions.
4.1 Image Denoising
Image denoising plays a critical role in modern signal processing systems since images are inevitably contaminated by noise during acquisition, compression, and transmission, leading to distortion and loss of image information [56].
Plenty of denoising methods exist, originating from a wide range of disciplines such as probability theory, statistics, partial differential equations, linear and nonlinear filtering, spectral and multiresolution analysis, also classical machine learning and deep learning [57, 58, 56].
All these methods rely on some explicit or implicit assumptions about the true (noise-free) signal in order to separate it properly from the random noise.
One of the most successful assumptions is that a signal can be well approximated by a linear combination of few basis elements in a transform domain [59, 60].
Under this assumption, a denoising method can be implemented as a two-step procedure: i) to obtain high-magnitude transform coefficients that convey mostly the true-signal energy, ii) to discard the transform coefficients which are mainly due to noise.
Typical choices for the first step are the wavelet, cosine transforms, and principal component analysis (PCA) [59, 60, 61].
The second step is seen as a filtering procedure that is formally modelled as a proximal optimization problem [62]
[Equation]
where g ( x ) acts as a regularization term, and u represents the noisy transform coefficients.
In fact, the usefulness of Eq.
(11) is not just limited to denoising, but other imaging problems like computer tomography [63], optical imaging [64], biomedical and spectral imaging [65].
In general, global optima guarantees in Eq.
(11) is restricted to convex g ( x ) , e.g., ℓ 1 -norm.
We improve this important proximal operator by incorporating invex regularizers.
Specifically, using those invex functions g ( x ) as listed in Table 2, global minimization is achieved in Eq.
The result is presented in the following theorem:
Theorem 3 ( Invex Proximal ) .
Consider the optimization problem in Eq.
(11) for all functions in
Table 2.
Then the following holds:
The function h ( x ) =
g ( x ) + 12 ∥ x − u ∥ 22 is convex (therefore invex).
The resolvent operator of the proximal is ( I + ∂g ) − 1 and it is treated as a singleton because it always maps to a global optimizer.
It is classically known that the sum of two invex functions is not necessarily invex in general [46].
Therefore, presenting examples like above, where the sum of f ( x ) and g ( x ) is invex, is important to both invexity and imaging communities.
We present the proof of Theorem 3 and provide the solution to Eq.
(11) for each function in Table 2 in Appendix C of supplementary material.
4.2 Image Compressive Sensing
Image compressive sensing has been extensively exploited in areas such as microscopy, holography, optical imaging and spectroscopy [66, 67, 68].
It is an inverse problem that aims at recovering an image f ∈ R n from its measurement data vector b = Φ f , where Φ ∈ R m × n is the image acquisition matrix ( m < n ).
Since m < n , compressive sensing assumes f has a k -sparse representation x ∈ R n ( k ≪ n non-zero elements) in a basis Ψ ∈ R n × n , that is f =
Ψ x , in order to ensure uniqueness under some conditions.
Examples of this sparse basis Ψ in imaging are the Wavelet (also Haar
Wavelet) transform, cosine and Fourier representations [69].
Hence, one can work with the abstract model b = ΦΨ x =
Hx , where H encapsulates the product between Φ , and Ψ , with ℓ 2 -normalized columns [66, 70].
Under this setup, compressive sensing enables to recover x using much lesser samples than what are predicted by the Nyquist criterion [70].
The task formulation is minimize
[Equation]
(12) where λ ∈ (0 , 1] is a typical choice in practice.
When the regularizer g ( x ) takes the convex form of ℓ 1 -norm, and when the sampling matrix H satisfies the restricted isometry property (RIP) for any k -sparse vector x ∈ R n , i.e., (1 − δ 2 k ) ∥ x ∥ 22 ≤ ∥ Hx ∥ 22 ≤ (1 + δ 2 k ) ∥ x ∥ 22 for δ 2 k < 13 [69, Theorem 6.9], it has been proved that x can be exactly recovered by solving Eq.
We are interested in invex regularizers.
It has been proved that, when g ( x ) takes the particular invex form in Eq.
(6), x can be exactly recovered by solving Eq.
Below we further generalize this result to all the invex functions as listed in Table 2.
The generalized result is presented in Theorem 4.
Theorem 4 ( Invex Image Compressive Sensing ) .
Assume Hx = b , where x ∈ R n is k -sparse, the matrix
H ∈
R m × n ( m < n ) with ℓ 2 -normalized columns that satisfies the RIP condition for any k -sparse vector, and b ∈ R m is a noiseless measurement vector.
If g ( x ) in Eq.
(12) takes the form of the functions in Table 2, then the following holds:
The objective function 12 [-(cid:107) Hx − b (cid:107) 
-]∥
Hx − b ∥ 22 + λg ( x ) is invex.
x can be exactly recovered by solving Eq.
(12) i.e. only global optimizers exist.
When g ( x ) takes the form of Eq.
(9), extra mild conditions on x are needed.
We clarify that if H satisfies the mentioned RIP, then each sub-matrix with k -columns of H , selected according to indices of the nonzero elements of the k -sparse signal is a full row-rank matrix.
This result is important to invex community, because it supports the validity of Lemma 2 to build invex functions with affine mappings.
Additionally, we present another proved form of function sum that can result in an invex function, i.e., the sum of g ( x ) and the ℓ 2 -norm composed with the affine mapping Hx − b .
The complete proof is provided in Appendix E of supplementary material.
Next, we present different algorithms to solve Eq.
(12) using invex g ( x ) as in Table 2.
We select a few of the most important and successful image reconstruction techniques to start from, and develop their invex extensions.
Taking advantage of the invex property, we prove convergence to global minimizers for each extended algorithm, which is unexplored up to date.
[Equation]
[Equation] [Equation]
[Equation] [Equation] √ 4( r t ) 2 +1+  8: x ( t +1) =
(cid:26) z ( t +1) , if f ( z ( t +1) )
+ λg ( z ( t +1) )
≤ f ( v ( t +1) )
+ λg ( v ( t +1) ) v ( t +1) , otherwise 9: end for 10: return: x
( T )
4.2.1 Accelerated Proximal Gradient Algorithm
The accelerated proximal gradient (APG) method [72] has been shown to be effective solving Eq. (12), achieving better imaging quality in less iterations than its predecessors [13, 36, 37, 38, 73], and been frequently used by recent imaging works [55, 74, 75, 76].
Its convergence to global optima is only guaranteed for convex loss [72].
For non-convex cases, convergence to a critical point has been stated [72].
Its pseudo-code for solving Eq.
(12) is provided in Algorithm 1.
Taking advantage that the loss function f ( x ) + λg ( x ) in Eq.
(12) is invex, and the uniqueness result in Theorem 3, we formally extend APG in the following lemma stating that the sequence (cid:8) x ( t +1) (cid:9) generated by Algorithm 1 converges to a global minimizer of Eq.
Lemma 3 ( Invex APG ) .
Under the setup of Theorem 4 and using L = σ 1 (cid:16) H T H (cid:17) (maximum singular value), the sequence (cid:8) x ( t ) (cid:9) T − 1 t =0 generated by Algorithm 1 converges to a global minimizer.
To prove Lemma 3, we apply the Statement 2 of Theorem 2 to Eq.
(12) and the unicity of the proximal operators for functions in Table 2.
The proof is provided in Appendix F of supplementary material.
4.2.2 Plug-and-play with Deep Denoiser Prior
Plug-and-play (PnP) is a powerful framework for regularizing imaging inverse problems [65] and has gained popularity in a range of applications in the context of imaging inverse problems [29, 65, 77, 78, 79].
It replaces the proximal operator in an iterative algorithm with an image denoiser, which does not necessarily have a corresponding regularization objective.
This implies that the effectiveness of PnP goes beyond standard proximal algorithms such as primal-dual splitting [80, 81, 82].
It has guarantees to a fixed point only when convex objective functions are employed [81].
To apply the PnP framework, we modify Algorithm 1 by replacing the proximal operator (Line 6 in its pseudo-code) with a neural network based denoiser Noise2Void [58], resulting in
[Equation]
The complete pseudo-code is presented in Algorithm 3 of Appendix G in supplemental material.
We remark that in Algorithm 3, Line 5 of the Algorithm 1 is retained to allows the comparison between regularizers (invex and convex).
More specifically, Line 5 computes the proximal step, while
Line 6 relies on a neural network for the same purpose (13).
This offers an avenue for simultaneously exploiting both the model-based and data-driven approaches.
The output of Algorithm 3 is a close estimation to the solution of Eq.
The benefit of using this denoiser is that it does not require clean target images in order to be trained.
We present the following convergence result for this modified algorithm under the assumption of f ( x ) in Eq.
(12) being invex which is a generalization of [81] (restricted to convex functions only).
Lemma 4 ( Invex Plug-and-play ) .
Assume f
( x ) in Eq.
(12) is invex with Lipschitz continuous gradient, and a denoiser d : R n → R .
Under the setup of Theorem 4 and some mild conditions on d , the sequence (cid:8) x ( t ) (cid:9) T t =0 generated by Algorithm 1 satisfies
[Equation]
for any x ∗ = d ( x ∗ − α 1 ∇ f ( x ∗ )) (fixed point) and for some κ ∈ (0 , 1) .
Eq.
(14) guarantees that (cid:8) x ( t ) (cid:9) T t =0 is arbitrarily close to the set of fixed points of d ( · ) , which is considered a close estimation to the solution of Eq.
Its proof is provided in Appendix G of supplementary material.
Eq. (9) is an example satisfying assumption required in Lemma 4.
4.2.3 Unrolling
The unrolling or unfolding framework is another imaging strategy for solving Eq.
It offers a systematic connection between iterative algorithms used in signal processing and the neural networks [31, 32, 83].
Unrolled neural networks become popular due to their potential in developing efficient and high-performing network architectures from reasonably sized training sets [84, 85].
A folded version of the proximal gradient algorithm is presented in Algorithm 2.
Particularly, existing works [86, 87] have shown that the efficiency of Algorithm 2 can be improved by simulating a recurrent neural network so that its layers mimic the iterations in Line 4 of Algorithm 2.
Specifically, each x ( t +1) constitutes one linear operation which models a layer of the network, followed by a proximal operation that models the activation function.
Thus, one forms a deep network by mapping each iteration to a network layer and stacking the layers together to learn H , α t , and x ( t ) for all t which is equivalent to executing an iteration of Algorithm 2 multiple times.
Their study was conducted only for g ( x ) in the form of ℓ 1 -norm.
Convergence guarantees to global optima for Algorithm 2 has been established in [13], but it is restricted to convex objective functions.
Therefore, due to the success and importance of unrolling we aim to extend the global optima guarantees of Algorithm 2 to invex objectives, and present the results in the following lemma:
Lemma 5 ( Invex Unrolling ) .
Under the setup of Theorem 4 and using L = σ 1 (cid:16) H T H (cid:17) (maximum singular value) and α t < 2 L +2 , the sequence (cid:8) x ( t ) (cid:9) T − 1 t =0 generated by Algorithm 2 converges to a global minimizer.
The key to proving Lemma 5 relies on the uniqueness result of the proximal operator for functions in Table 2 as stated in Theorem 3.
The proof is presented in Appendix H of supplementary material.
Such results confirm that the invex unrolled network of Algorithm 2, which uses the proximal operators of invex mappings as the activation functions, can reach the optimal solution during training.
Experiments and Results
A number of datasets have been merged to formulate one unique dataset for our training and evaluation purposes.
These are DIV2K super-resolution [88], the McMaster [89], Kodak [90],
Berkeley Segmentation (BSDS 500) [91], Tampere Images (TID2013) [92] and the Color BSD [93] datasets.
We conduct various experiments to study the performance of those invex regularizers as listed in Table 2 in non-ideal conditions.
We compare them against the state-of-the-art methods originally developed for convex regularizers ( ℓ 1 -norm) ensuring global optima.
When neural network training is involved, we take a total of 900 images which are randomly divided into a training set of 800 images, a validation set of 55 images, and a test set of 45 images.
For all the experiments, the images are scaled into the range [0 , 1] .
For the invex regularizer in Eq.
(6), we vary the value of p .
5.1 Image Compressive Sensing Experiments
We assess signal reconstruction, in these experiments, by averaging the peak-signal-to-noise-ratio (PSNR) in dB over the testing image set.
We consider additive white Gaussian noise in the measurements data vector with three different levels of SNR (Signal-to-Noise Ratio)
= 20, 30, and ∞ (noiseless case).
For Algorithm 1 and its plug-and-play variant, the parameters λ, α 1 , and α 2 were chosen to be the best for each analyzed function determined by cross validation, and the initial point x (0) was the blurred image b .
The results are summarized in Table 3, where the best and least efficient among invex functions is highlighted in boldface and underscore, respectively.
Additional results are reported in Appendix I of supplemental material for each experiment, using the structural similarity index measure to assess imaging quality.
Experiment 1 studies the effect of different invex regularizers, the Smoothly Clipped Absolute
Deviation (SCAD) [94], and the Minimax Concave Penalty (MCP) [24], under Algorithm 1.
A deconvolution problem is studied to formulate Eq.
(12) which is an important problem in signal processing due to imperfect artefacts in physical setups such as mismatch, calibration errors, and loss of contrast [95].
To compare, the used state-of-the-art methods that employ convex regularization are the Total Variation Minimization by Augmented Lagrangian (TVAL3) [96], and the fast iterative shrinkage-thresholding algorithm (FISTA) [13] which ensures global optima.
Further, to comparing with convolutional neural networks methodologies, the non-iterative reconstruction methodology
ReconNet [97] is used.
To model this problem, all pixels of the testing set are fixed to 256 ×pixels.
The images went through a Gaussian blur of size 9 × 9 and standard deviation 4 , followed by an additive zero-mean white Gaussian noise.
The sensing matrix H is built as H = ΦΨ (for all methods except ReconNet), where Φ represents the blur operator over the images and Ψ is the inverse of a three stage Haar wavelet transform.
This experiment is extremely ill-conditioned, where the condition number of H T H is significantly higher than 1.
This means that in practice the RIP condition is not guaranteed.
To achieve a fair comparison, the number of iterations was fixed for all functions as T = 800 .
The deconvolution problem follows a compressive sensing setup because the
Gaussian filter remove high frequency information of the input image.
In the case of ReconNet, we follow existing setting in [97].
For the learning of ReconNet, we extract patches of size 33 × 33 from the noisy blurred training image set, and we train it using the Adam optimization algorithm and a learning rate 5 × 10 − 4 for 512 epochs with a batch size of 128 .
Experiment 2 studies the invex regularizers under the plug-and-play modification of Algorithmas described in Section 4.2.2 2 [58].
The same deconvolution problem as in Experiment 1 is used.
The interesting aspect of this scenario is that Algorithm 1 has a proximal step in Line 5 that allows to compare between regularizers (invex and convex) while using neural networks in Line 6 (see
Algorithm 3 in Appendix G of Supplemental material).
Noise2Void is trained by randomly extracting patches of size 64 × 64 pixels from the training images where zero-mean white Gaussian noise was added for SNR = 20 , 30 dB. Data augmentation on the training dataset is used, by rotating each image three times by 90 and also added all mirrored versions.
The learning rate is fixed as 0 . 0004 .
Experiment 3 compares the invex regularizers but under the unrolling framework as described in
Section 4.2.3.
The gold standard convex regularizations to compare with are the learned iterative shrinkage and thresholding algorithm (LISTA) [87], and the Interpretable optimization-inspired deep network (ISTA-Net)[98].
Also, to comparing with convolutional neural networks methodologies, [Table] the non-iterative reconstruction methodology ReconNet [97] is used.
We follow the existing setting for LISTA in [86] 3 , and for ISTA-Net in [87].
For the training stage we extract 10000 patches b ∈ R 16 × 16 at random positions of each image, with all means removed.
We then learn a dictionary
D ∈ R 256 × 512 from the extracted patches, using the same strategy as in [86].
Gaussian i.i.d sensing matrices Φ ∈ R m × 256 are created from the standard Gaussian distribution, Φ [ i, j ] ∼ N (0 , 1 /m ) and then normalize its columns to have the unit ℓ 2 -norm, where m is selected such that m {+
+} = 0 .
The matrix H is built as H = ΦΨ with T = 16 (number of layers).
We follow the same two-step strategy in [86] to train a recurrent neural network.
First, perform a layer-wise pre-training solving Eq. (12) for each extracted patch b by fixing H = Ψ .
Second, append a learnable fully-connected layer at the end of the network structure, initialized by Ψ .
Then, perform an end-to-end training solving
Eq.
(12) where H in this case is learnt by updating the initial matrix Ψ .
For each testing image, we divide it into non-overlapping 16 × 16 patches.
When g ( x ) is the the ℓ 1 -norm, we recover [86].
In the case of ISTA-Net, and ReconNet, for their learning stage we extract patches from the training image set of size 33 × 33 .
Gaussian i.i.d sensing matrices Φ ∈ R m × 1089 are created with ℓ 2normalized columns as for LISTA, where m is selected such that m 1089 = 0 .
The optimizer employed was Adam algorithm and a learning rate 1 × 10 − 4 for 200 and 512 epochs for ISTA-Net and ReconNet respectively, with a batch size of 64 for both networks.
For ISTA-Net T = 16 (number of unrolled iterations).
We recall that when g ( x ) is the the ℓ 1 -norm in ISTA-Net, we recover [98].
5. Image Denoising Experiment
Two image datasets, which we merge ( 80 images in total), are used for this experiment comes from a neutron image formation phenomenon 4 .
These type of images contain the neutron attenuation properties of the object which helps analyze material structure.
Performance is assessed by averaging along all the images the experimental SNR in dB given by SNR = 20 log (cid:16) ∥ z ∥ 2 ∥ ˆ z − z ∥ 2 (cid:17) , where z and ˆ z stand for the noisy and the denoised image, respectively, and the structural similarity index measure (SSIM) computed between z and ˆ z .
Taking advantage of results observed from previous experiments, we compare the top three regularizers in Eqs.
(6), (8), and (10) with two state-of-the-art denoising techniques including the block-matching and 3-D filtering (BM3D) [59] using ℓ 1 -norm regularizer and the deep learning technique Noise2Void (trained as in Experiment 2) [58].
We follow the two-step denoising procedure described in Section 4.1.
In the first step, the transform domain is built using
PCA as in [61].
To build this transform we extract patches of 16 × 16 from the noisy image that are then used to adaptively construct a tight frame (nearly orthogonal matrix) tailored to the given noisy data 5 .
Results are summarized in Table 3.
We report examples of denoised images obtained by Eqs. (6), (8), (10), BM3D, and Noise2Void are illustrated in Appendix J of supplementary material, along with the algorithm used for the invex regularizers to denoise these images.
Discussion, Limitations and Conclusion
Application advancement of invex theory has paused for decades due to the lack of practical examples, which has caused a significantly reduced interest in invexity research.
To address this issue, we present for the first time a list of invex regularizers for image reconstruction applications, and formulate corresponding optimization problems.
Particularly, for image compressive sensing, we improve three advanced imaging techniques using the listed functions in Table 2 as invex regularizers.
We present their solution algorithms and develop theoretical guarantees on their convergence to global minimum.
We also conducted various image compressive sensing and denoising experiments to demonstrate the effectiveness of invex regularizers under practical scenarios that are non-ideal with noisy data observed and RIP condition not guaranteed.
Significant benefit of using invex regularizers have been proved from both theoretical and empirical aspects.
In fact, Table 3 and theoretical results in Section 4 revive the potential of exploring invex theory in practical applications.
The numerical results presented in Table 3 confirm performance improvement by using invex regularizers over the ℓ 1 -norm-based methods (e.g FISTA, TVAL3) in unexplored scenarios.
These tables and theoretical results in Section 4 revive the potential of exploring invex theory in practical applications.
The best result is obtained with Eq.
(6), and Eq.
(9) is the least efficient.
The intuition behind the superiority of Eq.
(6) comes from the possibility of adjusting the value of p in data-dependent manner [49].
This means that when the images are strictly sparse, and the noise is relatively low, a small value of p should be used.
Conversely, when images are non-strictly sparse and/or the noise is relatively high, a larger value of p tend to yield better performance (which seems to be the case for the selected image datasets).
We believe that the remaining invex, SCAD, and MCP regularizers have a lesser performance than Eq.
(6) as they do not have the flexibility of adjustment to the sparsity of the data.
In fact, Eq.
(9) shows the poorest performance because in the proof of Theorem 4, we theoretically guarantee that Eq.
(9) cannot sparsify all images.
Therefore, this analysis leads to the conclusion that the invex function Eq.
(6) offers the best performance for the metrics concerns and the imaging problems studied here.
Although, we have presented theoretical results with global optima using invexity for some of most important and successful image reconstruction techniques, we highlight several limitations of our analysis.
Specifically, we focused on reconstructed the image of interest in an ideal scenario, that is, without the present of noise (Theorem 4).
Additionally, we have limited our numerical results to tasks like denoising, and deconvolution.
And, the convergence guarantees for the plug-and-play result only ensures a close estimate of the solution (Lemma 4).
Therefore, we see there are a number of future directions this research can be taken further improving the results even further.
One aspect is to explore avenues for improving convergence guarantees to global optima the plug-and-play framework.
Another direction is the study of inclusion of noise in the analysis of imaging applications, which may be an enabler to improve downstream tasks like invex robust image reconstruction.
Finally, we feel that the application domains for invex functions can go well beyond denoising, and deconvolution imaging problems, especially around deep learning research, which can improved a number of downstream applications.
Broader Impact
We believe that the presented mathematical and empirical analysis over the studied regularizers has the potential to unlock the benefits of invexity for further applications in signal and image processing.
This may be an enabler to improve downstream tasks like deep learning for imaging, and to provide more robust image reconstruction algorithms.