Statistical Learning and Inverse Problems: A Stochastic Gradient Approach
Anonymous Author(s)
AffiliationAddress
Abstract
Inverse problems are paramount in Science and Engineering.
In this paper, we consider the setup of Statistical Inverse Problem (SIP) and demonstrate how Stochastic
Gradient Descent (SGD) algorithms can be used in the linear SIP setting.
Weprovide consistency and finite sample bounds for the excess risk.
We also proposea modification for the SGD algorithm where we leverage machine learning methods to smooth the stochastic gradients and improve empirical performance.
Weexemplify the algorithm in a setting of great interest nowadays: the Functional
Linear Regression model.
In this case we consider a synthetic data example andexamples with a real data classification problem.
Introduction
Inverse Problems (IP) might be described as the search of an unknown parameter (that could be a function) that satisfies a given, known equation.
Considering the notation:
[Equation]
where f and y are elements of given Hilbert spaces, we would like to compute (or estimate)
f giventhe data y for some level of noise.
Typically, IPs are ill-posed in the sense that the solution doesnot depend continuously on the data.
There are several very important and impressive examples of
IPs in our daily lives.
Medical imaging has been using IPs for decades and it has shaped the area,as for instance, Computerized Tomography (CT) and Magnetic Resonance Imaging (MRI).
For anintroductory text, see Vogel [2002].
A vast literature of IPs is devoted to deterministic problems where the noise term is also a element ofa Hilbert space and commonly assumed small in norm, which is not usually verified in practice.
Inthis work, we will take a different avenue, known as Statistical Inverse Problems (SIP).
This approachis a formalization of IPs within a probabilistic setting, where the uncertainty of all measurements areproperly considered.
Our focus in this work is to propose a direct and practical method for solving
SIP problems and, at the same time, provide theoretical guarantees for the excess risk performance ofthe algorithm we develop.
Our algorithm is based on a gradient descent framework, where stochasticgradients (or base learners that approximate the stochastic gradients) are used to estimate generalfunctional parameters.
The paper is organized as follows.
We finish this section contextualizing our paper in the broadliterature and stating our main contributions.
In Section 2, we formally introduce the learningproblem that we analyze.
In Section 3, we provide examples of practical problems that fits withinour formulation.
In Section 4 we provide our main results and algorithms.
Finally, in Section 5 weprovide numerical examples and a real data application for a Functional Linear Regression problem(FLR).
Due to space constraints, some of the figures, proofs and experiments were moved to thesupplementary material.
1.1 Contribution
We provide a novel numerical method to estimate functional parameters in SIP problems usingstochastic gradients.
More precisely, we extend the properties and flexibility of SGD and boostingalgorithms to a broader class of problems by bridging the gap between the IP and machine learningcommunities.
Whereas most of the IPs methods focus on regularization strategies to “invert" the operator A , wepropose a gradient descent type of algorithm to estimate the functional parameter directly.
Ouralgorithm works in the same spirit as Stochastic Gradient Descent algorithms with sample averaging.
While results of SGD are well understood in the context of regression problems in finite and infinitedimensions and SGD is well understood in deterministic IPs, SGD have not yet been consideredunder the SIP formulation.
We show that our procedure also ensures risk consistency in expectation and high probability underthe statistical setting.
Furthermore, we propose a modification in our algorithm to substitute thestochastic gradients by base learners similarly to boosting algorithms, Mason et al.
[1999], Friedman[2001].
This modification improve a common challenge faced by SIP problems: the discretizationprocedures of the operator A that arises in SIP.
1.2 Literature Review
Historically, SIP was first introduced in Sudakovand and Khalfin [1964] where IPs from Mathematical
Physics were recast into a statistical framework.
For a more structured introduction, we forward thereader to Kaipio and Somersalo [2004].
Several advances were made in the parametric approach to
SIP, where the unknown function is assumed to be completely described by an unknown parameterliving in a finite dimensional space, see for instance Evans and Stark [2002].
In our paper, however,we will consider the nonparametric framework as described in Cavalier [2008].
In this setting, we seethe IP as a search of an element of an infinite dimensional space.
When considering IPs (and SIP, in particular), there are several ways to regularize the problem in orderto deal with its ill-posedness.
For instance, one could consider roughness penalty or a functional basisas in Tenorio [2001].
Additionally, one could examine Tikhonov and spectral cut-off regularizationsas in Bissantz et al.
For many of those standard approaches, consistency under the SIP settingand rates of convergences were established.
See for instance Bissantz et al.
[2004], Bissantz and
Holzmann [2008].
A thoroughly discussion of stochastic gradient algorithms is outside the scope ofthis work
and we refer the reader to Zinkevich [2003], Nesterov et al.
[2018] and references therein.
There have been numerous applications of Machine Learning (and Deep Learning, in particular) tosolve IPs, in recent years. However, in our opinion, these applications are akin of the capabilitiesthat these new techniques could bring to this area of research.
Some attention have been given toimaging problems as in Jin et al.
[2017] and Ongie et al.
Under deterministic IP, the paper Li et al.
[2020] studies the regularization and convergence rates of penalized neural networks whensolving regression problems.
See also Adler and Öktem [2017] and Bai et al.
Other importantreferences regarding SGD for deterministic IP are Jin et al.
[2021], Tang et al.
[2019], Jin et al.
The main examples we bring in our paper is the class of Functional Linear Regression (FLR).
Thisproblem has drawn the attention of the statistical, econometric and computer science communities inthe past decade, see Cai and Hall [2006], Yao et al.
[2005], Hall and Horowitz [2007].
The usualmethodology applied to this problem is the well-known FDA.
For example, one could consider aprespecified functional basis to regularize the regression problem Goldsmith et al.
[2011] or one coulduse the Functional Principal Component (FPC) basis, Morris [2015].
More recently, methods inspiredin machine learning for standard linear regression problems were also extended to the FLR setting, seefor instance James et al.
[2009], Fan et al.
[2015] for methods that are suitable for high dimensionalcovariates or interpretable in the LASSO sense.
In this work we show how our modification to the
SGD algorithm can be seen as an averaging of boosting estimators and can also be used to estimate
FLR models in the high-dimensional setting.
Problem Formulation
We start by fixing a probability space (Ω , A , P ) and a vector space X of inputs.
We denote therandom input by X ∈ L 2 (Ω , A , P ) taking values in X and consider the space 1 L 2 ( X , B ( X ) , µ X ) ,henceforth referred to as L 2 ( X ) , of functions g : X −→ R d with inner product ⟨ g 1 , g 2 ⟩ L 2 ( X ) =
E [ ⟨ g 1 ( X ) , g 2 ( X ) ⟩ ] and norm ∥ g ∥ 2 L 2 ( X ) =
E [ ∥ g ( X ) ∥ 2 ] < + ∞ , where ⟨· , ·⟩ and ∥ · ∥ are the innerproduct and norm of R d .
We also consider a Hilbert space H with inner product ⟨· , ·⟩ H .
Finally, we consider an operator A : H −→ L 2 ( X ) .
This operator defines a direct problem and we assume that it is known.
Givenf ∈ H , we use the notation
A [ f ] ∈ L 2 ( X ) , i.e. A [ f ] is a square-integrable function
A [ f ] : X −→ R d .
We are interested in solving the statistical inverse problem related to A : jointly to observing samplesof X taking values in X , we observe noisy samples of A [ f ◦ ]( X ) , for some fixed, unknown f ◦ ∈ H ,which we denote by Y :
[Equation]
where ϵ is a zero-mean random noise.
The problem we will pore over in this paper is the estimationof f ◦ based on this given sample.
Let ℓ : R d × R d → R + be a point-to-point loss function as, for example, the squared loss ℓ ( y , y ′ )
=12 ∥
y − y ′
∥ 2 , for regression, or the logistic loss function ℓ ( y , y ′ ) = log(1 + e − y · y ′ ) , for classification.
We define the populational risk as:
[Equation]
and we would like to solve:
[Equation]
where F ⊂ H with f ◦ ∈ F .
We will denote by ∂ 2 the partial derivative with respect to the secondargument.
Given a sample, we will study how to control the excess risk of a functional estimator ˆ f of f ◦ :
[Equation]
Instead of taking the standard route of solving the Empirical Risk Minimization problem and laterestablishing results for (3), in Section 4 we show how our algorithms allow us to tackle (3) directlyby constructing stochastic gradients directly for the populational risk.
Examples: motivation
Before we formalize our results, we first motivate the study of Eq.
(1) with a few of applications.
Each of those problems have a myriad of solutions on their own.
For more information on those IPs,see, for instance, Vogel [2002].
Deconvolution .
This type of inverse problems relate the values of Y and X through the following convolution equation:
[Equation]
where X = W = R d , H = L 2 ( W , B , µ ) and the kernel k : R d −→ R is known.
In this case, we define the operator A as:
[Equation]
Functional Linear Regression .
Consider the scalar, multivariate functional linear regression: let 110 X
=
D ([0 , T ]) (the space of right-continuous with left limits functions taking values in R d × k with the sup norm) so that L 2 (Ω) is the space of stochastic processes with sample paths in D ([0 , T ]) andnorm
[Equation]
Moreover, H = L 2 ([0 , T ]) taking values in R k and Y is given by the following model
[Equation] where f ∈ H . Here we changed the notation from w to s in order to keep the classical notation from FLR.
In this case,
[Equation]
The model can be easily extended to deal with Y taking label values such as in a classificationproblem as we will show in the numerical studies.
Due to space constraints, we provide examples in the FLR setting.
In the supplementary material,we demonstrate how the algorithms presented in Section 4.3 can also be applied in deconvolutionproblems.
Theoretical Results and Algorithms
In this paper, we consider the following set of assumptions.
Assumption 4.1.
A : H −→ L 2 ( X ) is a linear, bounded operator;2.
ℓ is a convex and C 2 function in its second argument;
There exists θ 0 > 0 such that, for all f, g ∈ H , sup | θ |≤ θE [Equation] A
[Equation]
X ) , [Equation] [Equation] Y, A [ f ]( X ) +
[Equation] [ g ]( X ) (cid:1) A [ g ]( X ) (cid:69)(cid:21) < ∞ ; f ◦ ∈ arg min f ∈F R A ( f ) and R A ( f ◦ ) >
−∞ ;
sup f,f ′ ∈F
∥ f − f ′
∥
L 2 ( W ) =
D < ∞ .
Assumption 1 is our strongest one, since it imposes that our operator is linear and bounded.
Nevertheless, linear SIPs encompass a wide class of problems of practical and theoretical interest forengineering, statistics and computer science communities among others, a few of them presented in
Section 3.
Moreover, the nonlinear case could be similarly studied with more cumbersome notationand assumptions.
Assumption 2 is standard for gradient based algorithms and is commonly assumedin many learning problems.
Assumption 3 is a mild integrability condition of the loss function commonly satisfied in many practical situations.
For instance, in the squared loss case, this assumptionbecomes E [ ∥ A [ g ]( X ) ∥ 2 ] < ∞ , which is automatically satisfied since A [ g ] ∈ L 2 ( X ) .
Assumption 4is needed so the problem we analyze indeed has a solution.
Assumption 5 is stating that the diameterof the set F is finite.
One should notice that our set of assumptions does include the class of ill-posed (linear) inverseproblems since we do not need to assume that A is bijective.
If that were the case, it is known thatthen
A would have a bounded inverse, and then, the IP would not be ill-posed.
In the next sections we provide our theoretical results.
Instead of following the common approach ofminimizing the Empirical Risk Minimization problem, we show how to compute stochastic gradientsin order to control directly for the excess risk (3) both in expectation and in probability.
4.1 Preliminaries
Our first result allows us to compute the gradient of the populational risk at a given functional parameter f .
Before we present it, note that, by linearity, A : H → L 2 ( X ) is differentiable and, for every f, g ∈ H , we have that the directional derivative of A [ f ] in the direction g is given by
[Equation]
Note that the directional derivative does not depend on the point f that we are evaluating the gradient.
Let A ∗ denote the adjoint operator of A defined as the linear and bounded operator A ∗ : L 2 ( X ) −→ H such that
[Equation]
= ⟨ f, A ∗ [ h ] ⟩ H , for all f ∈ H and h ∈ L 2 ( X ) .
The following lemma holds true:
Lemma 4.2.
Under 1, 2 and 3 of Assumption 4.1 we have that
[Equation]
where ϕ f ( x ) =
E [ ∂ 2 ℓ ( Y , A
[ f ]( x )) | X = x ] .
Proof.
Firstly, define, for fixed ( X , Y ) ,
[Equation]
Then, we get the directional derivative of the risk function in direction g by applying the Taylorformula for ψ as a function of δ around δ = 0 :
[Equation]
where θ comes from the Taylor formula and it is between − θ
and θ 0 , for some fixed θ 0 > 0 .
Hence,by Assumption 3, we find
[Equation]
By the definition of ϕ and by conditioning in X , we find
[Equation]
Finally, we get that the descent direction ∇R A ( f ) is given by A ∗ [ ϕ f ] ∈ H .
4.2 Unbiased Estimator of the Gradient
In order to define an unbiased estimator of the gradient ∇R A , we consider the following assumption:
Assumption 4.3.
H is a Hilbert space of functions from W to R k ; There exists a kernel Φ : X × W −→ R k
× d such that [Equation] for all w ∈ W and h ∈ L 2 ( X ) .
Several examples, including the Functional Linear Regression, as we will verify in Section 5, satisfythis assumption,. Additionally, there are two situations that encompass many important SIPs. Thefirst one is a restriction of the Hilbert space H without restrictions on the operator A : 2 The adjoint of a linear, bounded operator always exists.
Lemma 4.4.
Assumption 4.3 is verified if H is a Reproducing Kernel Hilbert Space (RKHS).
Proof.
For simplicity of notation, we assume k = 1 . Notice that, by the RKHS assumption, φ w : L 2 ( X ) −→ R defined as φ w ( h ) =
A ∗ [ h ]( w ) , for h ∈ L 2 ( X ) , is an element of the dual of L 2 ( X ) , i.e. there existis M w < + ∞ such that [Equation]
Hence, by the Riesz Representation Theorem, there exists a kernel Φ( · ; w ) : X −→ R d such that
, forall h ∈ L 2 ( X ) ,
[Equation]
as desired.
Remark 4.5 .
If the RKHS H has kernel K , then Φ is given by Φ( x , w ) =
A [ K ( · , w )]( x ) .
Indeed, by the definition of kernel in the RKHS and the definition of A ∗ , we find
[Equation]
The second situation considers a different (and somewhat less restrictive) assumption for the Hilbertspace H and a particular, yet very general, class of operators A , called integral operators.
Lemma 4.6.
If H =
L 2 ( W , B , µ ) taking values in R k and A is a integral operator of the form:
[Equation]
where φ is a kernel taking values in R d × k such that φ ( x , · ) f ∈ L 1 ( W , B , µ ) , then Assumption 4.3 isverified.
Proof.
By the definition of the adjoint operator, we find
[Equation]
Hence, we conclude A ∗ [ h ]( w ) = E (cid:2) φ ( X , · ) T h ( X ) (cid:3) , which implies Assumption 4.3 with kernelΦ( x , w ) = φ ( x , w ) T . Remark 4.7 .
The class of integral operators delivers several of the most important linear IPs. Additionally, using Green’s function formulation, some PDEs IPs could also be recast as integral equations. For instance, the recovery the initial condition of a linear PDE with known Green function andobserving the solution of the PDE at some future, fixed time.
Under Assumption 4.3, Lemma 4.2 implies the following very useful result that is the cornerstone ofour method.
Corollary 4.8.
If Assumption 4.3 is verified, then the gradient of the risk function with respect to f isgiven by
[Equation]
Because of the results above, it is possible to construct an unbiased estimator for the gradient for therisk function for any f .
In fact, for a given sample ( x , y ) and a fixed function f , we define, for anyw ∈ W ,
[Equation]
Therefore, conditioning in X , we find
[Equation]
The main benefit is that with a single observation of ( X , Y ) , we are able to compute an unbiasedestimator for the gradient of the risk function under the true distribution.
4.3 Proposed Algorithms
Inspired by Corollary 4.8, we propose the following SGD algorithm for SIP problems that we called
SGD-SIP: given an initial guess f 0 , for each step i , we compute, following Eq.
(5), an unbiasedestimator u i for the gradient of the loss function.
Next, we update an accumulated functionalparameter by taking a stochastic gradient step in the direction of u i with step size α i .
In the laststep, we average all the accumulated gradient steps in the same spirit as Polyak and Juditsky [1992].
The choice of the step size needs to satisfy two criteria: (cid:80) ni
=
α i sublinear in n , and nα n → ∞ asn → + ∞ .
We formally justify those desired properties in Theorem 4.9.
input : sample { x i , y i } ni =1 , operator A , initial guess foutput :
ˆ f n ˆ g 0 = f 0 ; for 1 ≤ i ≤ n do
Compute u
i ( w ) =
Φ( x i , w ) ∂ 2 ℓ ( y i , A [ˆ g i − 1 ]( x i )) ; ˆ g i
= ˆ g i − 1 − α
i u i ; end
Set
ˆ f n = 1 n (cid:80) ni
=
ˆ g i ;
Algorithm 1 uses only one sample at a time in order to estimate the gradient of the true risk function.
In order to preserve this property, we make the number of iterations equal to the sample size; it cannotbe larger.
This connects with the stopping rules in iterative algorithms in IP.
Algorithm 1 has a limitation common to many approaches to Inverse Problems: one cannot hope tocompute u i for every possible w i and some discretization of the operator A is needed, see Kaipioand Somersalo [2007].
Since the SGD-SIP algorithm only computes the stochastic gradient in thepoints of discretization, it risks overfitting the data and provides non-smooth estimators.
Next, wemotivate Algorithm 2 in order to overcome the discretization problem by leveraging machine learningmethods.
Consider that the space W was discretized in a grid of size n w .
In order to fully estimate the functionˆ f n ( w ) for every w ∈ W , we consider a hypothesis class H and, in each step, we fit a functionˆ h ⋆i ∈ H on the stochastic gradient u i in the discretized grid of W .
Note that in this case, F will begiven by the linear span of the class H .
Each of these functions h ⋆i can be seen as a base-learner inthe same spirit of Boosting estimators, widely used in standard regression problem in the context of SIP Mason et al.
[1999], Friedman [2001].
Next we present our algorithm ML-SGD.
1 ≤ i ≤ n do do end end
Set
The goal of ML-SGD is twofold.
First, it allows us to interpolate the function
h
⋆j to points wnot used in the discretization grid.
Second, the ML procedure smooths the noise in each gradientstep calculation leading to smoother approximations that helps avoiding over-fitting.
We show in
Section 5 and in the supplementary material the benefits of such an approximation when estimatingthe functional parameter f ◦ in both simulated and empirical examples.
4.4 Main Result
Our main result is a finite sample bound for the expected excess risk of Algorithm ?? .
The result alsoextends to Algorithm ?? in the case where the base learner are also unbiased estimators.
Theorem 4.9.
Under Assumptions 4.1 and 4.3 and if the kernel Φ satisfies C = sup x ∈ X ∥ Φ( x , · ) ∥ 2 <+ ∞ 3 , we have the following performance guarantee for Algorithm 1:
[Equation]
where M ( A, F ) =
C ( E [ ∥ Y ∥ 2 ] + ∥ A ∥ 2 D 2 ) < ∞ .
The proof of the theorem is provided in the supplementary material.
Theorem 4.9 implies that ifwe pick the decreasing sequence {
α i } ni
=1 so that nα n → ∞ ( α n cannot decrease too fast) but fastenough so that 1 n (cid:80) ni
=
α i → 0 , then we get the convergence result.
For instance, one could takeα i =
η/ √ i for some fixed number η normally taken to be in (0 , 1) .
In this case, the excess riskdecreases in expectation with rate O (1 / √ n ) .
Theorem 4.9 also implies that the excess risk converges to zero in probability.
For α i =
η/ √
i it is straightforward to check that
[Equation]
Finite sample bounds with high probability can also be provided under stronger assumptions aboutthe stochastic gradients.
See for instance Nemirovski et al.
Functional Linear Regression: numerical studies
In this section, we provide two applications of the Functional Linear Regression problem.
We firstdemonstrate the performance of both algorithms in simulated data and next we provide an examplefor generalized linear models, applied to an classification problem using bitcoin transaction data. In Appendix B and Appendix C we also provide an additional numerical study in a different type of
Inverse Problem: The deconvolution problem.
As we have seen in Section 3, the operator in the FLR case is given by
[Equation]
Remember that in this example we are denoting w by s .
Hence, by Lemma 4.6, we find A ∗ [ g ]( s ) = 234 E (cid:2) X T ( s ) g ( X ) (cid:3) .
One can easily verify that A ∗ [ g ] ∈ H by the assumption that the norm (4) of X isfinite.
Therefore, we have Φ( x , s ) =
x T ( s ) , and we find, as in Eq.
[Equation]
5.1 Synthetic Data
We will consider the simulation study presented in González-Manteiga and Martínez-Calvo [2011].
Specifically, we set W = [0 , 1] , f ◦ ( z ) = sin(4 πz ) , and X simulated accordingly a Brownian motionin [0 , 1] .
We also consider a noise-signal ratio of 0.2.
We generate 100 samples of X and Y with theintegral defining the operator A approximated by a finite sum of 1000 points in [0 , 1] . We test forthe same specification when f ◦ ( z ) oscillates between 1 and − 1 in the points 0 .
Forthe observed data used in the algorithm procedure, we consider a coarser grid where each functionalsample is observed at only 100 equally-spaced times.
For the ML-SGD algorithm, we used smoothingsplines and regression trees as base learners.
We compared the results with Penalized Functional Linear Regression (PFLR) with cubic splines and cross-validation to select the number of basisexpansion; we also compare with Landweber iteration method.
In order to fit the PFLR model, we used the package refund Goldsmith et al.
[2021] available in R .
Detailed numerical results with errorbars are displayed in Appendix C showing that the ML-SGD algorithm is at least as competitive as astate-of-the-art tailored specifically to FLR problems and superior to Landweber iterations, a generalmethod for Inverse Problems. In Figure 1, we can see that both SGD-SIP and Landweber iterationsachieve similar fit performance.
PFLR with cubic splines and degree of freedom of 20 achieveda similar performance than ML-SGD with splines with 10 degrees of freedom. Both essentiallyrecovers the true function c ◦ perfectly. Despite the smoothness of f ◦ , ML-SGD with regression treeswith 30 terminal nodes was also able to approximate f ◦ well. Here we focus only on the methodswith best performance.
ML-SGD with both regression trees and splines with 20 degrees of freedomwere able to recover the true function as well as PFLR with cubic splines and 20 degrees of freedom.
In the appendix, we provide bar plots with the MSE under both scenarios with error bars for differentsimulations. Both Landweber iterations and the SGD algorithm are noisier and seems to overfit thedata. We refer the reader to Appendix C for a detailed comparison among the methods.
[Figure] 5.2 Real Data Application
Next we consider a classification problem in the FLR setting.
The data set contains 3000 bitcoinaddresses spanning from April 2011 and April 2017 and their respective cumulative credit, which isdescribed as 501 equally spaced observations for the first 3000 hours of each address, normalizedin the interval [0 , 1] .
For each address, we also have a label describing if the address was used forcriminal activity, commonly called darknet addresses .
In Table 1 we present a summary of thedata.
We refer the reader to Appendix A for more information about the data set used that we makeavailable online. [Table]
Here we use the cumulative credit curve at each point in time as the explanatory variables X ∈ 269 X
= D ([0 , 1]) and Y ∈ {− 1 , 1 } as the predicted outcome for the indicator variable that thecategory is darknet (addresses associated with illegal activities). We propose the following model:log P ( Y =1 | X ) P ( Y = − 1 | X ) = (cid:82) T 0 f ( s ) X
( s ) ds. By using the log-likelihood of the negative binomial, one caninclude its gradient with respect to the functional parameter directly in Equation (7) in order to useour framework.
Other type of classification loss functions can also be used in the same spirit.
We compare the SGD-SIP and ML-SGD algorithm with PFLR.
For the ML-SGD algorithm, we usetwo types of base learner, regression trees and cubic splines.
The step sizes are taken to be equal ofthe form O (1 / √ i ) , where i = 1 , · · · , n is the current step of the algorithm and n is the total numberof steps/sample.
For the PFLR algorithm, we use cubic splines with different degrees of freedom andquadratic penalty term.
We highlight that those choices of splines and penalty term are widely used inthe literature, see, for instance, Goldsmith et al.
In Table 2 we provide 3-fold cross validationfor the accuracy and kappa metrics.
The SGD-SIP Algorithm achieved the best average performancein terms of accuracy.
The same performance is achieved by the Functional PLR with cubic splineswith number of knots equal to 30 and penalization for the derivative of the estimate.
The ML-SGDalgorithm with smooth splines with 30 degrees of freedom also achieved similar performance with asmoother estimator.
Although the benchmark is as good as the ML-SGD algorithm, we highlighthere that PFLR is tailored to Functional Data Analysis problem, while our approach is flexible formany different types of Linear SIP problems. Moreover, our algorithm can make use of only onesample at each iteration, which makes it suitable also for online applications.
We refer the reader tothe supplementary material for results under different choices of step size, number of knots and basefunctions for the PFLR model, other metrics and confusion matrices.
In order to fit the PFLR mode,we used the package refund Goldsmith et al.
[2021] available in R .
[Table]
Conclusion
In this work, we provided a novel numerical method to solve SIP based on stochastic gradients withtheoretical guarantees for the excess risk.
Moreover, we have shown how one can improve algorithmicperformance by estimating base-learners for each stochastic gradient in the same spirit as boostingalgorithms.
Our framework can be applied in a variety of settings ranging from deconvolutionproblems, Functional Data analysis in both regression and classification problems, integral equationsand other linear IPs related to PDEs.
We demonstrate the performance of our method with numericalstudies and also with a real world application data and comparing with widely used techniques in the FLR setting.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We did both in the problem formulation
and in the numerical studies.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do not believe
that the methods proposed here could cause a negative impact in the society of any kind and thisdiscussion was not necessary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please refer to
(b) Did you include complete proofs of all theoretical results?
[Yes] Proofs are in the main text andin the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] The codes and data used are
available online and in the supplementary material.
Please, refer to the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to Section 5 and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes] In the main text we reported results with 3folds cross-validation.
In the
online supplementary material we report results with synthetic data with error bars for the seed
and additional validation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
internal cluster, or cloud provider)?
[No] The algorithms are simple and all the methods run in
standard computers.
Cloud computer and GPUs were not used.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.
[Table]
) Did you discuss whether and how
[Table]
was obtained from people whose data you’re us-
[Table]
ing/curating?
[No] The assets used are already available for
[Table]
) Did you discuss whether the data you are using/curating contains personally identifiable infor- [Table]
mation or offensive content?
[No] The data is not identifiable or offensive.
Please, refer to the
[Table] description provided in Section [Table]
If you
[Table]
crowdsourcing or conducted research with human subjects...
[Table]
you
[Table]
participants and
[Table]
, if applicable
[Table]
you
[Table]
[Table]
, if
[Table]
[Table]
the
[Table] and the [Table] on
[Table] [Figure] [Equation]
[Figure]
A link for a github webpage with the data will be provided. The link is omitted at the moment tonot reveal the authors identity. B Functional Gradient for the Deconvolution Problem Remember that the operator A is given by
[Equation]
Hence,
[Equation]
where
A ∗ [ g ]( w ) =
E [ k ( X − w ) g ( X )] .
Therefore, we have Φ( x , w ) =
k ( x − w ) , and we find, as in Eq.
(5), u i ( w ) =
k ( x i − w ) ∂ 2 ℓ ( y i , A
[ˆ g i − 1 ]( x i )) .
We highlight here the need to use each observation only once in order to compute the stochasticgradient so we can have precisely n steps for the SGD-SIP/ML-SGD algorithm.
In this case, thesamples can be used to provide unbiased estimators for the gradient of the risk function under thepopulational distribution.
C Numerical Studies: Synthetic Data
In this section we present the numerical studies of our proposed algorithms with standard benchmarksfrom the literature.
We studied both the Functional Linear Regression problem and the Deconvolutionproblem.
We remind the reader that the same framework can also be used to solve different types ofinverse problems under a statistical framework, such as ODEs and PDEs.
C.1 Functional Linear Regression
Recall Section 5 where for the FLR problem our goal is to recover f ◦ when we have access to observations of the form
[Equation]
where the operator A is given by
[Equation]
Recall the data generating process described in 5.1.
We set W = [0 , 1] , f ◦ ( w ) = sin(4 πw ) , and Xsimulated accordingly a Brownian motion in [0 , 1] .
We also consider a noise-signal ratio of 0.2.
Next,we study also the case where f ◦ oscillates between 1 , − 1 in the points w = 0 .
Wegenerate 3000 samples of X and Y with the integral defining the operator A approximated by a finitesum of 1000 points in [0 , 1] .
For the observed data used in the algorithm procedure, we consider acoarser grid where and each functional sample is observed at only 100 equally-spaced times.
For the
ML-SGD algorithm, we used smoothing splines as base learners.
We compare our algorithm withthe Landweber method, which is a Gradient Descent version for deterministic Inverse Problems and
Functional Penalized Linear Regression (FPLR).
For the ML-SGD, SGD and Landweber method,the step sizes were taken fixed to be O (1 / √ N ) (which satisfy the requirements discussed after 4.9).
We simulate the data generating process 10 times in order to compute the metrics performance.
Wecompare the methods in terms of Mean Square Error of the recovered function f ◦ .
In Figure 3 we present the Mean Squared Error and with Error Bars representing 2 standard deviations.
In this case, we can see that PFLR with different specifications out-perform our propposed algorithms,which achieves similar performance as Landweber iterations.
It is important to note here, that while
PFLR methods are tailored for this type of problems, ours, as well as Landweber iterations, are not.
Nevertheless, we can see in Figure 1a that essentially all the algorithms are capable of recovering thetrue underlying function f ◦ .
In Figure 4 we have a similar setup in a harder problem, where the underlying f ◦ is not as smooth asbefore.
In this case, the advantage of the PFLR reduces and the performance of all the methods arevery similar.
It is important to note that our approach makes use of only one sample at each iteration ofour proposed algorithms.
One can improve the stability and convergence of the estimated algorithmsby simply using more samples at each time.
In case one uses all the samples in each iteration (suchas what is commonly done in Landweber iteration or boosting procedures in standard regressionproblems), Theorem 4.9 cannot be applied directly but empirically the methods perform well.
Weillustrate this approach in Figure 5, where we make use of all the samples in every iteration of ouralgorithms.
[Figure]
[Figure]
[Figure]
C.2 Deconvolution
For the deconvolution problem we examine the following numerical exercise.
We take two choices offunctional parameters for Eq.
[Figure]
as a peak function:
[Equation]
We consider the kernel to be given by
[Equation]
and the following parameters for the data generating process.
First we discretize the space W =[ − 10 , 10] with increments h = 0 .
We use the same for the space X =
[ − 10 , 10] .
Next, we usethe discretized space to generate the true values
A [ f ] where we approximate the integral by a finitesum.
The second step is to generate the random observations.
For that, we consider a coarser gridfor X , with grid h obs = 0 .
1 , i.e. 10 times less information than the simulation used to generate thetrue observations.
This reproduces the fact that in practice one cannot hope to observe the functionaldata over all points. Moreover, when computing the operator A in our algorithm, we again consider acoarser grid for W , with grid h obs = 0 . We then add iid noise terms N (0 , 2) to the observations A [ f ] collected from the coarse grid.
For the ML-SGD algorithm (Algorithm 2), we used smoothsplines with 5 degrees of freedom as H in order to estimate the stochastic gradients.
We compare ouralgorithms with the well-known landweber iteration, which resambles the standard Gradient Descentalgorithm when ignoring noise and using all the samples available in all the iterations.
{+We start withf 
+}
{+( z ) = 
+}
in all the algorithms. In Figure 6a we can see that ML-SGD outputs a smooth estimator for the functional parameter f ◦while the other two methods tends to overfit the data. Nevertheless, this apprently instability seems to allow both the SGD-SIP and Landweber to better estimate the function in the peak, which compensatein the Mean Square Error estimator despite the increase in the volatility of the estimator. In Figure 6bwe present the Mean Squared Errors and Error Bars with two standard deviations.
[Figure]
D Proof of Theorem 4.
Proof.
First, it is straightforward to check that R A is convex in F : if f, g ∈ F and λ ∈ [0 , 1] , then R [Equation] ( λf
[Equation] [Equation] E [
[Equation] ( Y , A [ λf + (1 − λ ) g ]( X ))] =
E [
[Equation] ( Y , λA [ f ]( X ) + (1 − λ ) A [ g ]( X ))] ≤ E [ λℓ ( Y , A
[ f ]( X ))]
+ E
[(1 − λ ) ℓ ( Y , A [ g ]( X ))]
= λ R A
( f ) + (1 − λ )
R A ( g ) .
For simplicity of notation we will denote the norm and inner product in L 2 ( W ) by ∥ · ∥ and ⟨· , ·⟩ .
By the Algorithm 1 procedure, we have that
[Equation]
where the last inequality follows from convexity of the loss function (Assumption 2).
Rearranging terms we get
[Equation]
Summing over i leads ton (cid:88)
[Equation]
For the first term, by Assumption 5, we findn (cid:88) i =
[Equation]
since ˆ g i ∈ F for all i = 1 , . . . , n .
{+To bound the second term, notice that 
+}
[Equation]
Hence, if we take C =
sup x ∈ X ∥ Φ( x , · )
{+∥ 2 < + ∞ , we find 
+}
[Equation]
Finally, for the third term, note that, after taking expectation, the tower property and the fact that u i isan unbiased estimator of the gradient of R A (see Eq.
(5)) give that
[Equation]
where D i − 1 denotes the σ -algebra generated by the data { x k , y k } i − 1 k =1 .Again, by convexity of the risk function, R A ( ˆ f n ) ≤ 1 n (cid:80)
ni
=1 R A (ˆ g i ) .
Therefore,
[Equation]
and the theorem is proved.