Statistical Learning and Inverse Problems: A Stochastic Gradient Approach
Abstract
Inverse problems are paramount in Science and Engineering.
In this paper, we consider the setup of Statistical Inverse Problem (SIP) and demonstrate how Stochastic
Gradient Descent (SGD) algorithms can be used to solve linear SIP.
We provide consistency and finite sample bounds for the excess risk.
We also propose a modification for the SGD algorithm where we leverage machine learning methods to smooth the stochastic gradients and improve empirical performance.
We exemplify the algorithm in a setting of great interest nowadays: the Functional Linear
Regression model.
In this case we consider a synthetic data example and a classification problem for predicting the main activity of bitcoin addresses based on their balances.
Introduction
Inverse Problems (IP) might be described as the search of an unknown parameter (that could be a function) that satisfies a given, known equation.
Considering the notation:
[Equation]
where f ◦ and y are elements of given Hilbert spaces, we would like to compute (or estimate) f ◦ given the data y for some level of noise.
Typically, IPs are ill-posed in the sense that the solution does not depend continuously on the data.
There are several very important and impressive examples of IPs in our daily lives.
Medical imaging has been using IPs for decades and it has shaped the area, as for instance, Computerized Tomography (CT) and Magnetic Resonance Imaging (MRI).
For an introductory text, see Vogel [2002].
A vast literature of IPs is devoted to deterministic problems where the noise term is also a element of a Hilbert space and commonly assumed small in norm, which is not usually verifiable in practice.
In this work, we will take a different avenue, known as Statistical Inverse Problems (SIP).
This approach is a formalization of IPs within a probabilistic setting, where the uncertainty of all measurements are properly considered.
Our focus in this work is to propose a direct and practical method for solving
SIP problems and, at the same time, provide theoretical guarantees for the excess risk performance of the algorithm we develop.
Our algorithm is based on a gradient descent framework, where stochastic gradients (or base learners that approximate the stochastic gradients) are used to estimate general functional parameters.
The paper is organized as follows.
We finish this section contextualizing our paper in the broad literature and stating our main contributions.
In Section 2, we formally introduce the learning problem that we analyze.
In Section 3, we provide examples of practical problems that fits within our formulation.
In Section 4 we provide our main results and algorithms.
Finally, in Section 5 we provide numerical examples and a real data application for a Functional Linear Regression problem (FLR).
Due to space constraints, some of the figures, proofs and experiments were moved to the supplementary material.
1.1 Contribution
We provide a novel numerical method to estimate functional parameters in SIP problems using stochastic gradients.
More precisely, we extend the properties and flexibility of SGD and boosting algorithms to a broader class of problems by bridging the gap between the IP and machine learning communities.
Whereas most of the IPs methods focus on regularization strategies to “invert" the operator A , we propose a gradient descent type of algorithm to estimate the functional parameter directly.
Our algorithm works in the same spirit as Stochastic Gradient Descent algorithms with sample averaging.
While results of SGD are well understood in the context of regression problems in finite and infinite dimensions and SGD is well understood in deterministic IPs, SGD have not yet been considered under the SIP formulation.
We show that our procedure also ensures risk consistency in expectation and high probability under the statistical setting.
Furthermore, we propose a modification in our algorithm to substitute the stochastic gradients by base learners similarly to boosting algorithms, Mason et al.
[1999], Friedman [2001].
This modification improve a common challenge faced by SIP problems: the discretization procedures of the operator A that arises in SIP.
1.2 Literature Review
Historically, SIP was first introduced in Sudakovand and Khalfin [1964] where IPs from Mathematical
Physics were recast into a statistical framework.
For a more structured introduction, we forward the reader to Kaipio and Somersalo [2004].
Several advances were made in the parametric approach to
SIP, where the unknown function is assumed to be completely described by an unknown parameter living in a finite dimensional space, see for instance Evans and Stark [2002].
In our paper, however, we will consider the nonparametric framework as described in Cavalier [2008].
In this setting, we see the IP as a search of an element of an infinite dimensional space.
When considering IPs (and SIP, in particular), there are several ways to regularize the problem in order to deal with its ill-posedness.
For instance, one could consider roughness penalty or a functional basis as in Tenorio [2001].
Additionally, one could examine Tikhonov and spectral cut-off regularizations as in Bissantz et al.
For many of those standard approaches, consistency under the SIP setting and rates of convergences were established.
See for instance Bissantz et al.
[2004], Bissantz and
Holzmann [2008].
A thoroughly discussion of stochastic gradient algorithms is outside the scope of this work and we refer the reader to Zinkevich [2003], Nesterov et al.
[2018] and references therein.
There have been numerous applications of Machine Learning (and Deep Learning, in particular) to solve IPs, in recent years.
However, in our opinion, these applications are akin of the capabilities that these new techniques could bring to this area of research.
Some attention have been given to imaging problems as in Jin et al.
[2017] and Ongie et al.
Under deterministic IP, the paper
Li et al.
[2020] studies the regularization and convergence rates of penalized neural networks when solving regression problems.
See also Adler and Öktem [2017] and Bai et al.
Other important references regarding SGD for deterministic IP are Jin et al.
[2021], Tang et al.
[2019], Jin et al.
The main examples we bring in our paper is the class of Functional Linear Regression (FLR).
This problem has drawn the attention of the statistical, econometric and computer science communities in the past decade, see Cai and Hall [2006], Yao et al.
[2005], Hall and Horowitz [2007].
The usual methodology applied to this problem is the well-known FDA.
For example, one could consider a prespecified functional basis to regularize the regression problem Goldsmith et al.
[2011] or one could use the Functional Principal Component (FPC) basis, Morris [2015].
More recently, methods inspired in machine learning for standard linear regression problems were also extended to the FLR setting, see for instance James et al.
[2009], Fan et al.
[2015] for methods that are suitable for high dimensional covariates or interpretable in the LASSO sense.
In this work we show how our modification to the
SGD algorithm can be seen as an averaging of boosting estimators and can also be used to estimate
FLR models in the high-dimensional setting.
Problem Formulation
We start by fixing a probability space (Ω , A , P ) and a vector space X of inputs.
We denote the random input by X ∈ L 2 (Ω , A , P ) taking values in X and consider the space 1 L 2 ( X , B ( X ) , µ X ) , henceforth referred to as L 2 ( X ) , of functions g : X −→ R d with inner product ⟨ g 1 , g 2 ⟩ L 2 ( X ) =
E [ ⟨ g 1 ( X ) , g 2 ( X ) ⟩ ] and norm ∥ g ∥ 2 L 2 ( X ) =
E [ ∥ g ( X ) ∥ 2 ] < + ∞ , where ⟨· , ·⟩ and ∥ · ∥ are the inner product and norm of R d .
We also consider a Hilbert space H with inner product ⟨· , ·⟩ H .
Finally, we consider an operator
A : H −→ L 2 ( X ) .
This operator defines a direct problem and we assume that it is known.
Given f ∈ H , we use the notation A [ f ] ∈ L 2 ( X ) , i.e. A [ f ] is a square-integrable function A [ f ] : X −→ R d .
We are interested in solving the statistical inverse problem related to A : jointly to observing samples of X taking values in X , we observe noisy samples of A [ f ◦ ]( X ) , for some fixed, unknown f ◦ ∈ H , which we denote by Y :
[Equation]
where ϵ is a zero-mean random noise.
The problem we will pore over in this paper is the estimation of f ◦ based on this given sample.
Let ℓ : R d × R d → R + be a point-to-point loss function as, for example, the squared loss ℓ ( y , y ′ )
= 12 ∥
y − y ′
∥ 2 , for regression, or the logistic loss function ℓ ( y , y ′ )
=
log(1 + e − y · y ′ ) , for classification.
We define the populational risk as:
[Equation]
and we would like to solve:
[Equation]
) where F ⊂ H with f ◦ ∈ F .
We will denote by ∂ 2 the partial derivative with respect to the second argument.
Given a sample, we will study how to control the excess risk of a functional estimator ˆ f of f ◦ :
[Equation]
Instead of taking the standard route of solving the Empirical Risk Minimization problem and later establishing results for (3), in Section 4 we show how our algorithms allow us to tackle (3) directly by constructing stochastic gradients directly for the populational risk.
Examples: motivation
Before we formalize our results, we first motivate the study of Eq.
(1) with a few of applications.
Each of those problems have a myriad of solutions on their own.
For more information on those IPs, see, for instance, Vogel [2002].
Deconvolution .
This type of inverse problems relate the values of Y and X through the following convolution equation:
[Equation]
where X = W = R d , H
=
L 2 ( W , B , µ ) and the kernel k : R d −→ R is known.
In this case, we define the operator A as:
[Equation]
Functional Linear Regression .
Consider the scalar, multivariate functional linear regression: let X = D
([0 , T ]) (the space of right-continuous with left limits functions taking values in R d × k with the sup norm) so that L 2 (Ω) is the space of stochastic processes with sample paths in D ([0 , T ]) and norm
[Equation]
Moreover, H =
L 2 ([0 , T ]) taking values in R k and Y is given by the following model
[Equation]
where f ∈ H .
Here we changed the notation from w to s in order to keep the classical notation from
FLR.
In this case,
[Equation]
The model can be easily extended to deal with Y taking label values such as in a classification problem as we will show in the numerical studies.
Due to space constraints, we provide examples in the FLR setting.
In the supplementary material, we demonstrate how the algorithms presented in Section 4.3 can also be applied in deconvolution problems.
Theoretical Results and Algorithms
In this paper, we consider the following set of assumptions.
Assumption 4.1.
A : H −→ L 2 ( X ) is a linear, bounded operator; 2.
ℓ is a convex and C 2 function in its second argument; 3.
There exists θ 0 > 0 such that, for all f, g ∈ H , sup | θ |≤ θE (cid:20)(cid:68)
A [ g ]( X ) , ∂ 22 ℓ (cid:0) Y, A
[ f ]( X ) + θA [ g ]( X ) (cid:1)
A [ g ]( X ) (cid:69)(cid:21) < ∞ ; 4.
f ◦ ∈ arg min f ∈F R A ( f ) and R A ( f ◦ ) >
−∞ ; 5.
sup f,f ′ ∈F
∥ f − f ′
∥
L 2 ( W ) =
D < ∞ .
Assumption 1 is our strongest one, since it imposes that our operator is linear and bounded.
Nevertheless, linear SIPs encompass a wide class of problems of practical and theoretical interest for engineering, statistics and computer science communities among others, a few of them presented in
Section 3.
Moreover, the nonlinear case could be similarly studied with more cumbersome notation and assumptions.
Assumption 2 is standard for gradient based algorithms and is commonly assumed in many learning problems.
Assumption 3 is a mild integrability condition of the loss function commonly satisfied in many practical situations.
For instance, in the squared loss case, this assumption becomes E [ ∥ A [ g ]( X ) ∥ 2 ] < ∞ , which is automatically satisfied since A [ g ] ∈ L 2 ( X ) .
Assumptionis needed so the problem we analyze indeed has a solution.
Assumption 5 is stating that the diameter of the set F is finite.
One should notice that our set of assumptions does include the class of ill-posed (linear) inverse problems since we do not need to assume that A is bijective.
If that were the case, it is known that then A would have a bounded inverse, and then, the IP would not be ill-posed.
In the next sections we provide our theoretical results.
Instead of following the common approach of minimizing the Empirical Risk Minimization problem, we show how to compute stochastic gradients in order to control directly for the excess risk (3) both in expectation and in probability.
4.1 Preliminaries
Our first result allows us to compute the gradient of the populational risk at a given functional parameter f .
Before we present it, note that, by linearity, A : H → L 2 ( X ) is differentiable and, for every f, g ∈ H , we have that the directional derivative of A [ f ] in the direction g is given by
[Equation]
Note that the directional derivative does not depend on the point f that we are evaluating the gradient.
Let A ∗ denote the adjoint operator of A defined as the linear and bounded operator A ∗ :
L 2 ( X ) −→ H such that⟨ A [ f ] , h ⟩ L 2 ( X ) = ⟨ f, A ∗ [ h ] ⟩ H , for all f ∈ H and h ∈ L 2 ( X ) .
The following lemma holds true:
Lemma 4.2.
Under 1, 2 and 3 of Assumption 4.1 we have that
[Equation]
where ϕ f ( x ) =
E [ ∂ 2 ℓ ( Y , A
[ f ]( x )) | X = x ] .
Proof.
Firstly, define, for fixed ( X , Y ) ,
[Equation]
Then, we get the directional derivative of the risk function in direction g by applying the Taylor formula for ψ as a function of δ around δ = 0 :
[Equation]
where θ comes from the Taylor formula and it is between − θ
and θ 0 , for some fixed θ 0 > 0 .
Hence, by Assumption 3, we find
[Equation]
By the definition of ϕ and by conditioning in X , we find
[Equation]
Finally, we get that the descent direction ∇R A ( f ) is given by A ∗ [ ϕ f ] ∈ H .
4.2 Unbiased Estimator of the Gradient
In order to define an unbiased estimator of the gradient ∇R A , we consider the following assumption:
Assumption 4.3.
H is a Hilbert space of functions from W to R k ; 2.
There exists a kernel Φ :
X × W −→ R
k
×
d such that
[Equation]
for all w ∈ W and h ∈ L 2 ( X ) .
Several examples, including the Functional Linear Regression, as we will verify in Section 5, satisfy this assumption.
Additionally, there are two situations that encompass many important SIPs.
The first one is a restriction of the Hilbert space H without restrictions on the operator A :
Lemma 4.4.
Assumption 4.3 is verified if H is a Reproducing Kernel Hilbert Space (RKHS).
Proof.
For simplicity of notation, we assume k = 1 .
Notice that, by the RKHS assumption, φ w : L 2 ( X ) −→ R defined as φ w ( h )
=
A ∗ [ h ]( w ) , for h ∈ L 2 ( X ) , is an element of the dual of
L 2 ( X ) , i.e. there existis M w < + ∞ such that
[Equation]
Hence, by the Riesz Representation Theorem, there exists a kernel Φ( · ; w ) : X −→ R d such that, for all h ∈ L 2 ( X ) ,
[Equation]
as desired.
Remark 4.5 .
If the RKHS H has kernel K , then Φ is given by Φ( x , w ) =
A [ K ( · , w )]( x ) .
Indeed, by the definition of kernel in the RKHS and the definition of A ∗ , we find
[Equation]
The second situation considers a different (and somewhat less restrictive) assumption for the Hilbert space H and a particular, yet very general, class of operators A , called integral operators.
Lemma 4.6.
If H =
L 2 ( W , B , µ ) taking values in R k and A is a integral operator of the form:
[Equation]
where φ is a kernel taking values in R d × k such that φ ( x , · ) f ∈ L 1 ( W , B , µ ) , then Assumption 4.3 is verified.
Proof.
By the definition of the adjoint operator, we find
[Equation]
Hence, we conclude A ∗ [ h ]( w ) =
E (cid:2) φ ( X , w ) T h ( X ) (cid:3) , which implies Assumption 4.3 with kernel Φ( x , w ) =
φ ( x , w ) T .
Remark 4.7 .
The class of integral operators delivers several of the most important linear IPs.
Additionally, using Green’s function formulation, some PDEs IPs could also be recast as integral equations.
For instance, the recovery the initial condition of a linear PDE with known Green function and observing the solution of the PDE at some future, fixed time.
Under Assumption 4.3, Lemma 4.2 implies the following very useful result that is the cornerstone of our method.
Corollary 4.8.
If Assumption 4.3 is verified, then the gradient of the risk function with respect to f is given by
[Equation]
Because of the results above, it is possible to construct an unbiased estimator for the gradient for the risk function for any f .
In fact, for a given sample ( x , y ) and a fixed function f , we define, for any w ∈ W ,
[Equation]
Therefore, conditioning in X , we find
E [ u f ( w ; X , Y )] = E [Φ( X , w ) ∂ 2 ℓ ( Y , A [ f ]( X ))] =
E [ E [Φ( X , w ) ∂ 2 ℓ ( Y , A [ f ]( X )) | X ]] =
E [Φ( X , w ) E [ ∂ 2 ℓ ( Y , A
[ f ]( X ))
| X ])]
=
E [Φ( X , w ) ϕ f ( X )] = ∇R A ( f )( w ) . The main benefit is that with a single observation of ( X , Y ) , we are able to compute an unbiased estimator for the gradient of the risk function under the true distribution.
4.3 Proposed Algorithms
Inspired by Corollary 4.8, we propose the following SGD algorithm for SIP problems that we called
SGD-SIP: given an initial guess f 0 , for each step i , we compute, following Eq.
(5), an unbiased estimator u i for the gradient of the loss function.
Next, we update an accumulated functional parameter by taking a stochastic gradient step in the direction of u i with step size α i .
In the last step, we average all the accumulated gradient steps in the same spirit as Polyak and Juditsky [1992].
The choice of the step size needs to satisfy two criteria: (cid:80) ni
=
α i sublinear in n , and nα n → ∞ as n → + ∞ .
We formally justify those desired properties in Theorem 4.9.
input : sample { x i , y i } ni =1 , operator A , initial guess foutput :
ˆ f n ˆ g 0 = f 0 ; for 1 ≤ i ≤ n do
Compute u
i ( w ) =
Φ( x i , w ) ∂ 2 ℓ ( y i , A
[ˆ g i − 1 ]( x i )) ; ˆ g i
= ˆ g −
− α u i ; end
Set ˆ
[-ˆ f n = 1 n (cid:80) ni =
 ˆ g i ;-] Algorithm 1 uses only one sample at a time in order to estimate the gradient of the true risk function.
In order to preserve this property, we make the number of iterations equal to the sample size; it cannot be larger.
This connects with the stopping rules in iterative algorithms in IP.
Algorithm 1 has a limitation common to many approaches to Inverse Problems: one cannot hope to compute u i for every possible w i and some discretization of the operator A is needed, see Kaipio and Somersalo [2007].
Since the SGD-SIP algorithm only computes the stochastic gradient in the points of discretization, it risks overfitting the data and provides non-smooth estimators.
Next, we motivate Algorithm 2 in order to overcome the discretization problem by leveraging machine learning methods.
Consider that the space W was discretized in a grid of size n w .
In order to fully estimate the function ˆ f n ( w ) for every w ∈ W , we consider a hypothesis class H and, in each step, we fit a function ˆ h ⋆i ∈ H on the stochastic gradient u i in the discretized grid of W .
Note that in this case, F will be given by the linear span of the class H .
Each of these functions h ⋆i can be seen as a base-learner in the same spirit of Boosting estimators, widely used in standard regression problem in the context of
SIP Mason et al.
[1999], Friedman [2001].
Next we present our algorithm ML-SGD.
Algorithm ≤ n do end end
The goal of ML-SGD is twofold.
First, it allows us to interpolate the function
h
⋆j to points w not used in the discretization grid.
Second, the ML procedure smooths the noise in each gradient step calculation leading to smoother approximations that helps avoiding over-fitting.
We show in
Section 5 and in the supplementary material the benefits of such an approximation when estimating the functional parameter f ◦ in both simulated and empirical examples.
4.4 Main Result
Our main result is a finite sample bound for the expected excess risk of Algorithm 1.
The result also extends to Algorithm 2 in the case where the base learner are also unbiased estimators.
Theorem 4.9.
Under Assumptions 4.1 and 4.3 and if the kernel Φ satisfies C = sup x ∈ X ∥ Φ( x , · ) ∥ 2 < + ∞ 3 , we have the following performance guarantee for Algorithm 1:
[Equation]
where M ( A, F ) =
C ( E [ ∥ Y ∥ 2 ] + ∥
A ∥ 2 D 2 ) < ∞ .
The proof of the theorem is provided in the supplementary material.
Theorem 4.9 implies that if we pick the decreasing sequence { α i } ni =1 so that nα n → ∞ ( α n cannot decrease too fast) but fast enough so that 1 n (cid:80) ni =
α i → 0 , then we get the convergence result.
For instance, one could take α i =
η/ √ i for some fixed number η normally taken to be in (0 , 1) .
In this case, the excess risk decreases in expectation with rate O (1 / √ n ) .
Theorem 4.9 also implies that the excess risk converges to zero in probability.
For α i =
η/ √
i it is straightforward to check that
[Equation]
Finite sample bounds with high probability can also be provided under stronger assumptions about the stochastic gradients.
See for instance Nemirovski et al.
Functional Linear Regression: numerical studies
In this section, we provide two applications of the Functional Linear Regression problem.
We first demonstrate the performance of both algorithms in simulated data and next we provide an example for generalized linear models, applied to an classification problem using bitcoin transaction data.
In
Appendix B and Appendix C
we also provide an additional numerical study in a different type of
Inverse Problem: The deconvolution problem.
As we have seen in Section 3, the operator in the FLR case is given by
[Equation]
Remember that in this example we are denoting w by s .
Hence, by Lemma 4.6, we find A ∗ [ g ]( s ) = E (cid:2)
X T ( s ) g ( X ) (cid:3) .
One can easily verify that A ∗ [ g ] ∈ H by the assumption that the norm (4) of X is finite.
Therefore, we have Φ( x , s ) =
x T ( s ) , and we find, as in Eq.
[Equation]
5.1 Synthetic Data
We will consider the simulation study presented in González-Manteiga and Martínez-Calvo [2011].
Specifically, we set W = [0 , 1] , f ◦ ( z ) = sin(4 πz ) , and X simulated accordingly a Brownian motion in [0 , 1] .
We also consider a noise-signal ratio of 0.2.
We generate 100 samples of X and Y with the integral defining the operator A approximated by a finite sum of 1000 points in [0 , 1] .
We test for the same specification when f ◦ ( z ) oscillates between 1 and − 1 in the points 0 .
For the observed data used in the algorithm procedure, we consider a coarser grid where each functional sample is observed at only 100 equally-spaced times.
For the ML-SGD algorithm, we used smoothing splines and regression trees as base learners.
We compared the results with Penalized Functional
Linear Regression (PFLR) with cubic splines and cross-validation to select the number of basis expansion; we also compare with Landweber iteration method.
In order to fit the PFLR model, we used the package refund Goldsmith et al.
[2021] available in R .
Detailed numerical results with error bars are displayed in Appendix C showing that the ML-SGD algorithm is at least as competitive as a state-of-the-art tailored specifically to FLR problems and superior to Landweber iterations, a general method for Inverse Problems.
In Figure 1, we can see that both SGD-SIP and Landweber iterations achieve similar fit performance.
PFLR with cubic splines and degree of freedom of 20 achieved a similar performance than ML-SGD with splines with 10 degrees of freedom.
Both essentially recovers the true function c ◦ perfectly.
Despite the smoothness of f ◦ , ML-SGD with regression trees with 30 terminal nodes was also able to approximate f ◦ well.
Here we focus only on the methods with best performance.
ML-SGD with both regression trees and splines with 20 degrees of freedom were able to recover the true function as well as PFLR with cubic splines and 20 degrees of freedom.
In the appendix, we provide bar plots with the MSE under both scenarios with error bars for different simulations.
Both Landweber iterations and the SGD algorithm are noisier and seems to overfit the data.
We refer the reader to Appendix C for a detailed comparison among the methods.
[Figure]
5.2 Real Data Application
Next we consider a classification problem in the FLR setting.
The data set contains 3000 bitcoin addresses spanning from April 2011 and April 2017 and their respective cumulative credit, which is described as 501 equally spaced observations for the first 3000 hours of each address, normalized in the interval [0 , 1] .
For each address, we also have a label describing if the address was used for criminal activity, commonly called darknet addresses .
In Table 1 we present a summary of the data.
We refer the reader to Appendix A for more information about the data set used that we make available online.
[Table]
Here we use the cumulative credit curve at each point in time as the explanatory variables X ∈ X =
D [0 , 1]) and Y ∈ {− 1 , 1 } as the predicted outcome for the indicator variable that the category is darknet (addresses associated with illegal activities).
We propose the following model: log P ( Y =1 | X ) P ( Y = − 1 | X ) =
(cid:82) T 0 f ( s ) X
( s ) ds.
By using the log-likelihood of the negative binomial, one can include its gradient with respect to the functional parameter directly in Equation (7) in order to use our framework.
Other type of classification loss functions can also be used in the same spirit.
We compare the SGD-SIP and ML-SGD algorithm with PFLR.
For the ML-SGD algorithm, we use two types of base learner, regression trees and cubic splines.
The step sizes are taken to be equal of the form O (1 / √ i ) , where i = 1 , · · · , n is the current step of the algorithm and n is the total number of steps/sample.
For the PFLR algorithm, we use cubic splines with different degrees of freedom and quadratic penalty term.
We highlight that those choices of splines and penalty term are widely used in the literature, see, for instance, Goldsmith et al.
In Table 2 we provide 3-fold cross validation for the accuracy and kappa metrics.
The SGD-SIP Algorithm achieved the best average performance in terms of accuracy.
The same performance is achieved by the Functional PLR with cubic splines with number of knots equal to 20 and penalization for the derivative of the estimate.
The ML-SGD algorithm with smooth splines with 20 degrees of freedom also achieved similar performance with a smoother estimator.
Although the benchmark is as good as the ML-SGD algorithm, we highlight here that PFLR is tailored to Functional Data Analysis problem, while our approach is flexible for many different types of Linear SIP problems.
Moreover, our algorithm can make use of only one sample at each iteration, which makes it suitable also for online applications.
We refer the reader to the supplementary material for results under different choices of step size, number of knots and base functions for the PFLR model, other metrics and confusion matrices.
In order to fit the PFLR mode, we used the package refund Goldsmith et al.
[2021] available in R .
[Table]
Conclusion
In this work, we provided a novel numerical method to solve SIP based on stochastic gradients with theoretical guarantees for the excess risk.
Moreover, we have shown how one can improve algorithmic performance by estimating base-learners for each stochastic gradient in the same spirit as boosting algorithms.
Our framework can be applied in a variety of settings ranging from deconvolution problems, Functional Data analysis in both regression and classification problems, integral equations and other linear IPs related to PDEs.
We demonstrate the performance of our method with numerical studies and also with a real world application data and comparing with widely used techniques in the
FLR setting.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We did both in the problem formulation and in the numerical studies.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do not believe that the methods proposed here could cause a negative impact in the society of any kind and this discussion was not necessary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please refer to(b)
Did you include complete proofs of all theoretical results?
[Yes] Proofs are in the main text and in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes and data used are available online and in the supplementary material.
Please, refer to the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to Section 5 and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] In the main text we reported results with 3folds cross-validation.
In the online supplementary material we report results with synthetic data with error bars for the seed and additional validation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] The algorithms are simple and all the methods run in standard computers.
Cloud computer and GPUs were not used.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Please, see Section(b)
Did you mention the license of the assets?
[Yes] Please, see Section(c) Did you include any new assets either in the supplemental material or as a URL? [No] The existing assets are already available online.
(d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[No] The assets used are already available for public use.
(e) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content?
[No] The data is not identifiable or offensive.
Please, refer to the description provided in Section5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Empirical Application: Dataset and Code
From Febrero-Bande et al.
[2022]: “a bitcoin address is a unique string of number and letters stored in the bitcoin’s blockchain that can be the recipient or sender of bitcoins.
A bitcoin transaction can have multiple addresses as inputs and outputs.
One could think the address as the number of a bank account, but with some very distinct properties as public visibility of balance and all transactions, anonymity, cannot hold negative quantities of bitcoins (debt), and each user in this market usually has a large number of addresses.
The user here, though, is not an actual individual, but a company/website/organization that typically holds multiple addresses; we refer to it as “entity”.
The balance of an address is the amount (possibly zero) of bitcoins in that address at a given time.
The entity that a given address belongs to is usually unknown but some companies make some of its addresses public for various reasons.
Based on this public information, one can identify more addresses as belonging to this same entity if they were inputs to the same transaction with one or more inputs from this entity.” The data used in our classification model is the cumulative credits of the first 3000 hours of the address.
In Figure 2 below we show some example of balances over time of some addresses. The dataset and the R code used in this paper can be found here.
[Figure] B Functional Gradient for the Deconvolution Problem
Remember that the operator A is given by
[Equation]
Hence,
[Equation]
where
[Equation]
Therefore, we have Φ( x , w ) =
k ( x − w ) , and we find, as in Eq.
(5), u i ( w [Equation] =
k [Equation] x i − w [Equation]
[Equation]
i [Equation] [Equation] g i − 1 ]( x i )) .
We highlight here the need to use each observation only once in order to compute the stochastic gradient so we can have precisely n steps for the SGD-SIP/ML-SGD algorithm.
In this case, the samples can be used to provide unbiased estimators for the gradient of the risk function under the populational distribution.
C Numerical Studies: Synthetic Data
In this section we present the numerical studies of our proposed algorithms with standard benchmarks from the literature.
We studied both the Functional Linear Regression problem and the Deconvolution problem.
We remind the reader that the same framework can also be used to solve different types of inverse problems under a statistical framework, such as ODEs and PDEs.
C.1 Functional Linear Regression
Recall Section 5 where for the FLR problem our goal is to recover f ◦ when we have access to observations of the form
[Equation]
where the operator A is given by
[Equation]
Recall the data generating process described in 5.1.
We set W = [0 , 1] , f ◦ ( w ) = sin(4 πw ) , and X simulated accordingly a Brownian motion in [0 , 1] .
We also consider a noise-signal ratio of 0.2.
Next, we study also the case where f ◦ oscillates between 1 , − 1 in the points w = 0 .
We generate 3000 samples of X and Y with the integral defining the operator A approximated by a finite sum of 1000 points in [0 , 1] .
For the observed data used in the algorithm procedure, we consider a coarser grid where and each functional sample is observed at only 100 equally-spaced times.
For the ML-SGD algorithm, we used smoothing splines as base learners.
We compare our algorithm with the Landweber method, which is a Gradient Descent version for deterministic Inverse Problems and Functional Penalized Linear Regression (FPLR).
For the ML-SGD, SGD and Landweber method, the step sizes were taken fixed to be O (1 / √ N ) (which satisfy the requirements discussed after 4.9).
We simulate the data generating process 10 times in order to compute the metrics performance.
We compare the methods in terms of Mean Square Error of the recovered function f ◦ .
In Figure 3 we present the Mean Squared Error and with Error Bars representing 2 standard deviations.
In this case, we can see that PFLR with different specifications out-perform our propposed algorithms, which achieves similar performance as Landweber iterations.
It is important to note here, that while PFLR methods are tailored for this type of problems, ours, as well as Landweber iterations, are not.
Nevertheless, we can see in Figure 1a that essentially all the algorithms are capable of recovering the true underlying function f ◦ .
In Figure 4 we have a similar setup in a harder problem, where the underlying f ◦ is not as smooth as before.
In this case, the advantage of the PFLR reduces and the performance of all the methods are very similar.
It is important to note that our approach makes use of only one sample at each iteration of our proposed algorithms.
One can improve the stability and convergence of the estimated algorithms by simply using more samples at each time.
In case one uses all the samples in each iteration (such as what is commonly done in Landweber iteration or boosting procedures in standard regression problems), Theorem 4.9 cannot be applied directly but empirically the methods perform well.
We illustrate this approach in Figure 5, where we make use of all the samples in every iteration of our algorithms.
[Figure]
[Figure] [Figure]
C.2 Deconvolution
For the deconvolution problem we examine the following numerical exercise.
We take two choices of functional parameters for Eq.
[Figure]
as a peak function:
[Equation]
We consider the kernel to be given by
[Equation]
and the following parameters for the data generating process.
First we discretize the space W = [ − 10 , 10] with increments h = 0 .
We use the same for the space X =
[ − 10 , 10] .
Next, we use the discretized space to generate the true values
A [ f ] where we approximate the integral by a finite sum.
The second step is to generate the random observations.
For that, we consider a coarser grid for X , with grid h obs = 0 .
1 , i.e. 10 times less information than the simulation used to generate the true observations.
This reproduces the fact that in practice one cannot hope to observe the functional data over all points.
Moreover, when computing the operator A in our algorithm, we again consider a coarser grid for W , with grid h obs = 0 .
We then add iid noise terms N (0 , 2) to the observations
A [ f ] collected from the coarse grid.
For the ML-SGD algorithm (Algorithm 2), we used smooth splines with 5 degrees of freedom as H in order to estimate the stochastic gradients.
We compare our algorithms with the well-known landweber iteration, which resambles the standard Gradient Descent algorithm when ignoring noise and using all the samples available in all the iterations.
We start [-withf 
-]with f 0 ( z ) =
in all the algorithms.
In Figure 6a we can see that ML-SGD outputs a smooth estimator for the functional parameter f ◦ while the other two methods tends to overfit the data.
Nevertheless, this apprently instability seems to allow both the SGD-SIP and Landweber to better estimate the function in the peak, which compensate in the Mean Square Error estimator despite the increase in the volatility of the estimator.
In Figure 6b we present the Mean Squared Errors and Error Bars with two standard deviations.
[Figure]
D Proof of Theorem 4.
Proof.
First, it is straightforward to check that R A is convex in F : if f, g ∈ F and λ ∈ [0 , 1] , then
R A ( λf + (1 − λ ) g ) =
E [ ℓ ( Y , A [ λf + (1 − λ ) g ]( X ))]
=
E [ ℓ ( Y , λA [ f ]( X ) + (1 − λ )
A [ g ]( X ))]
≤ E [ λℓ ( Y , A
[ f ]( X ))]
+ E
[(1 − λ ) ℓ ( Y , A [ g ]( X ))]
= λ R A
( f ) + (1 − λ )
R A ( g ) .
For simplicity of notation we will denote the norm and inner product in L 2 ( W ) by ∥ · ∥ and ⟨· , ·⟩ .
By the Algorithm 1 procedure, we have that
[Equation]
where the last inequality follows from convexity of the loss function (Assumption 2) [Equation]
Rearranging terms we get
[Equation]
Summing over i leads to n
[Equation]
For the first term, by Assumption 5, we find n (cid:88)
[Equation]
since ˆ g i ∈ F for all i = 1 , . . . , n .
To bound the second term, notice that[-
-]
[Equation]
Hence, if we take C =
sup x ∈ X ∥ Φ( x , [Equation]
) ∥ 2 < + ∞ , we find[-
-]
[Equation]
Finally, for the third term, note that, after taking expectation, the tower property and the fact that u i is an unbiased estimator of the gradient of R A (see Eq.
(5)) give that
E [
[Equation]
u i
[Equation]
∇R A
[Equation] g i − [Equation] where D i − 1 denotes the σ -algebra generated by the data { x k , y k } i − 1 k =1 .Again, by convexity of the risk function, R A ( ˆ f n ) ≤ 1 n (cid:80)
ni
=1 R A (ˆ g i ) .
Therefore,
[Equation]
and the theorem is proved.