Learning Spectral Regularizations for Linear Inverse Problems
Abstract
One of the main challenges in linear inverse problems is that a majority of such problems are ill-posed in the sense that the solution does not depend on the data continuously.
To analyze this effect and reestablish a continuous dependence, classical theory in Hilbert spaces largely relies on the analysis and manipulation of the singular values of the linear operator and its pseudoinverse with the goal of, on the one hand, keeping the singular values of the reconstruction operator bounded, and, on the other hand, approximating the pseudoinverse sufﬁciently well for a given noise level.
While classical regularization methods manipulate the singular values via explicitly deﬁned functions, this paper considers learning such parameter choice rules in such a way, that one obtains higher quality reconstruction results while still remaining in a setting of provably convergent spectral regularization methods.
We discuss different ways of parametrizing our spectral regularization methods via neural networks, interpret existing feed forward networks in the setting of spectral regularization which can become provably convergent via an additional projection, and ﬁnally demonstrate their superiority in 1d numerical examples.
Introduction
The classical theory for linear inverse problems considers a linear operator A : X → Y between
Hilbert spaces X and Y and asks the question how to reconstruct an unknown x ∈ X from an observation y δ ∈ Y given as y δ
=
Ax + n δ , where n δ ∈ Y represents noise of magnitude (cid:107) n δ (cid:107) =
δ .
As soon as A is a compact linear operator with inﬁnite dimensional range, A admits a singular value decomposition (SVD)
[Equation] (1) with singular vectors u n ∈ X and v n ∈ Y and zero being an accumulation point of the corresponding singular values σ n .
Thus, the pseudoinverse A † of A becomes
[Equation]
[Equation]
Since zero is an accumulation point of the σ n , the pseudoinverse is unbounded, i.e., discontinuous.
The idea of spectral regularization is to replace the pseudoinverse A † by a family of operators
R α :
Y →
X given by
[Equation]
in such a way that R α is a continuous on the entire space Y and that it converges pointwise to A † as the regularization parameter α goes to zero.
A suitable choice of α as a function of δ (a-priori choice) or of ( δ, y δ ) (a-posteriori choice) allows to reestablish the continuous dependence of the solution on the data in the sense that
[Equation]
In this paper we investigate learning a function N parameterized by θ such that
[Equation]
provably satisﬁes (4).
We discuss a-priori as well as a-posteriori choices of the regularization, demonstrate how learning improves the results over classical choices, and illustrate the performance boost when turning to nonlinear reconstruction operators.
Related Work
Classically, the spectral regularization approaches have been deﬁned manually, e.g. via functions g α of the form
[Equation]
Truncated SVD
[Equation]
Lavrentiev
[Equation]
along with suitable parameter choice rules α , see e.g. [8].
The fact that Tikhonov regularization is equivalent to solving a minimization problem with a quadratic penalty, subsequently gave rise to nonlinear variational methods for which convergent regularization methods similar to (4) can be guaranteed in different measures of distances, see e.g. [4, 2].
Due to the rise of deep learning techniques many benchmarks are currently dominated by directly learning a mapping from y δ to a desired solution x via a neural network.
Such techniques are, however, largely lacking a theoretical understanding and rarely even take the noise level δ into account explicitly (see e.g. [20] for an exception).
Several works have considered hybrid methods between regularization techniques that allow for detailed analysis and learning based approaches:
For instance, [6, 1, 10] use minimization/regularization algorithms as a template for network architectures, [3, 12, 19, 11] optimize over a learned latent space that contains mostly realistic solutions, [20] consider regularization by parametrization via deep neural networks, and algorithmic schemes that replace the proximal operator of a regularizer with a neural network have been studied in [15, 13].
Such techniques do, however, not correspond to minimization problems anymore unless the network possesses very speciﬁc properties, see [16].
Beyond this, safeguarding techniques such as [14] or bilevel optimization problems that learn a parameterized variational regularization (e.g. [18, 7, 5, 9]) are the only way to remain in the regime of energy minimization methods.
The additional difﬁculty of deﬁning networks in such a way that they act on continuous functions rather than their ﬁxed discretizations, makes a convergence analysis in the sense of (4) difﬁcult and rare.
On a related note, the work [17] considers learning itself as an ill-posed inverse problem to derive convergence properties similar to (4).
Learned Spectral Regularizers 3.1 Architectures
For learning spectral regularizations we consider different types of parameterized functions to represent g α :
• A-priori parameter choices:
We parametrize N ( σ, δ, y δ ; θ ) by two classical approaches, namely a learned Lavrentiev and a learned Tikhonov regularization given via
[Equation]
with p ≤ 1 , q ≤ 2 , and a network
[Equation]
with a 2-layer fully connected network FCN , and one additional scale parameter θ scale .
To also make a comparison to classical (but noise-level optimal) regularization choices, we additionally drop the dependence of ˜ N on σ in (7) and refer to these methods as the classical
Lavrentiev and Tikhonov regularizations.
• A-posteriori parameter choice: Many papers have demonstrated great success in directly predicting a solution ˆ x for given data y δ by exploiting spatial regularity of x , e
g. through convolutional neural networks in imaging applications.
While the straight forward application of such networks lacks any kind of convergence guarantee, such techniques can be converted to a-posteriori spectral regularizations with convergence guarantees via suitable projectors.
Let ˆ x = G
( y δ ; θ ) be some prediction of a neural network G designed to solve the underlying problem directly.
Then the choice
[Equation]
results in the spectral regularization yielding G ( y δ ; θ ) as a reconstruction result (assuming that (cid:104) v n , y δ (cid:105) (cid:54) = 0 , and that all σ n are different).
To ensure convergence, we modify such a prediction via the
following projection
[Equation]
with learnable parameters θ l and θ u .
The
projection can be interpreted as remaining in between a strong and a weak Tikhonov regularization ( α u << α l ) up to some tolerance with deceases to zero as δ → 0 .
For G we choose an architecture loosely motivated by unrolling a proximal gradient descent algorithm, see A.2 in the appendix for details.
With the above choices, we can state the following convergence result:
Proposition 1 (Convergent spectral regularization methods) .
Both learning based approaches, the a-priori choice (6) and the a-posteriori choice (9) , are convergent regularization methods in the sense of (4) provided that q < 2 and p < 1 .
Proof.
See appendix A.1.
Numerical Experiments
To compute the solution of inverse problems numerically, we need to discretize any inﬁnite dimensional problem to a ﬁnite one, e.g., by considering a suitable subspace.
While this step alone reintroduces regularity as ﬁnite dimensional linear operators can never be discontinuous, the resulting problem still remains ill-conditioned due to a quick decay of the singular values σ n .
To investigate the behavior of our regularization strategy, we consider two inverse problems: The differentiation as well as deblurring of a function y : [0 , 1] → R , giving rise to the linear operators
[Equation]
for g being a Gaussian kernel.
In both cases we discretize x by evaluating it at positions ( x n ) 1 ≤ n ≤ N and approximate the integrals of the operators by simple summations.
Further details on the training can be found in the appendix.
Figure 1 (left and middle) shows the learned Tikhonov and Lavrentiev regularizations along with the classical Tikhonov regularization and a naive approach, in which g α is directly parameterized as a fully connected network depending on σ and δ , for two different noise levels δ as a function of σ .
As we can see, all learned regularizers choose a Tikhonov-type shape of g α , but allow much larger g α ( σ ) for σ on a medium scale.
In particular, the learned Tikhonov and direct (naive) learning yield remarkably similar shapes of g spectral
[Figure]
[-δ 2 .5 .
-]
plot in Fig.
1) as a function of δ , one can see that the Tikhonov, learned Tikhonov and Lavrentiev regularizers yield a value of g α ( σ )
= 1 /σ for δ = 0 .
The direct (naive) approach, however, fails to yield such a value, which directly implies that this method is not a convergent regularization in the sense of (4).
We conclude that the choice of the architecture with built-in behavior is crucial for obtaining theoretical guarantees.
Naive
Lav.
Tik.
Learned Lav.
Learned Tik.
A-Post.
Deblur training 29.
test 29.
Diff.
training 28.
test
[Table] As for the overall performance of each approach, Table 4 shows the PSNR values for all methods over training and testing in both applications for a ﬁxed discretization of N = 50 .
As we can see, the learning-based methods clearly outperform the classical approach while still remaining provably convergent in the sense of Prop. 1.
Moreover, the learned Tikhonov parametrization is superior to its Lavrentiev counterpart.
The naive approach does yield good PSNR values (although with a high initialization-depending variance), but does not yield a convergent regularization, i.e., does not yield faithful results for small noise levels.
The best results by far are obtained by the learned a-posteriori approach, which converts a direct prediction of the solution x to a spectral regularization.
[Table]
Shown in the inset ﬁgure are two curves of g α = N ( · , δ, y δ ; θ ) for the same y and δ , such that merely the realizations of the noise differ.
As we can see, the learned a-posteriori choice is non-monotone, does not yield smooth curves such as in Fig. 1, and differs signiﬁcantly for different realizations of noisy data.
Yet, the PSNR values of this approach is signiﬁcantly higher, indicating the importance of non-linear regularizations that vary for different y δ .
Conclusions
In this paper we studied spectral regularization methods for linear inverse problems and successfully learned provably convergent regularization methods that outperform their classical counterpart.
By considering a-posteriori choice rules in spectral regularizations, we turned to non-linear reconstruction techniques, which yield even better reconstruction results and ultimately raise the quest for establishing provably convergent learned regularization methods in other notions of distance, e.g. resembling the analysis of variational methods using Bregman distances.
A Supplementary Material A.1 Proof of Proposition
The techniques used for proving our proposition are well known and follow the arguments in [8].
For the sake of completeness, we will still give an overview of the proof.
The central idea is to show the following results that ensure a convergent spectral regularization method: For a regularization of the form (3) the following criteria ensure (4).
[Equation]
Proof.
The condition (11a) ensures that R α is a continuous linear operator for any ﬁxed δ >because (cid:107) R α (cid:107)
=
C α .
For y in the domain of A † , y ∈ D ( A † ) , and y δ with (cid:107) y − y δ (cid:107) ≤ δ we estimate
[Equation]
Now using (cid:107) R α (cid:107) =
C
α condition (11d) ensures that the second term in the above estimate converges to zero for δ → 0 .
As for the ﬁrst term we ﬁnd
[Equation]
Due to condition (11c) the above sum remains bounded independent of α and therefore is uniformly convergent, such that summation and a limit of δ → 0 can be exchanged.
Finally, condition (11b) ensures that the above term converges to zero.
Left to verify is that our learnable architectures satisfy the conditions (11).
For both a-priori choice rules (11a) holds due to the sigmoid function being strictly greater than zero for all inputs.
Condition (11b) holds for p, q > 0 since the sigmoid function is bounded by 1.
Condition (11c) holds with ˜ C = 1 , and (11d) holds if p < 1 and q < 2 .
Similarly, the projection operator of the a-posteriori choice (9) ensures the conditions (11) to be met.
A.2 Details on Network Architectures and Training
The fully-connected networks in the learned Lavrentiev and learned Tikhonov models consist of two fully-connected layers with hidden dimension 10.
The naive network consists of 3 fully-connected layers with hidden dimension 2 → 100 → 100 → 1 .
For the a-posteriori prediction ˆ x =
G
( y δ ; θ ) we utilize an iteration inspired by proximal gradient descent
[Equation]
(15) where x 0 = 0 and τ = 1 .
The networks G i themselves are 3 -layer convolutional networks with hidden layer sizes of 10.
For the convoultions we use zero padding and we ﬁx the minimum kernel size such that the receptive ﬁeld of the output covers the entire input x i − 1 .
A.2.1 Training
For each resolution N ∈ { 10 , 11 , . . .
, 50 } we generate R = 5000 many training examples x i,N by sampling random superpositions of sine and cosine functions, applying the discretized operator A and adding zero-mean Gaussian noise of different standard deviations δ i,N with δ i,N ∈ [0 , 0 . 2] to obtain simulated data y δ i,N i,N .
Then we train our networks in a supervised way via min θ
[Equation]
The superpositions of sine and cosine functions are realized by
[Equation] [Equation]
where ω 1 and ω 2 are drawn from a standard normal distribution and γ is sampled uniformly from the interval [ − 1 , 1] .