S HORT AND S PARSE D ECONVOLUTION — A G EOMETRIC A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Short-and-sparse deconvolution (SaSD) is the problem of extracting localized, recurring motifs in signals with spatial or temporal structure.
Variants of this problem arise in applications such as image deblurring, microscopy, neural spike sorting, and more.
The problem is challenging in both theory and practice, as natural optimization formulations are nonconvex.
Moreover, practical deconvolution problems involve smooth motifs (kernels) whose spectra decay rapidly, resulting in poor conditioning and numerical challenges.
This paper is motivated by recent theoretical advances (Zhang et al., 2017; Kuo et al., 2019), which characterize the optimization landscape of a particular nonconvex formulation of SaSD.
This is used to derive a provable algorithm which exactly solves certain non-practical instances of the SaSD problem.
We leverage the key ideas from this theory (sphere constraints, data-driven initialization) to develop a practical algorithm, which performs well on data arising from a range of application areas.
We highlight key additional challenges posed by the ill-conditioning of real SaSD problems, and suggest heuristics (acceleration, continuation, reweighting) to mitigate them.
Experiments demonstrate the performance and generality of the proposed method.
I NTRODUCTION
Many signals arising in science and engineering can be modeled as superpositions of basic, recurring motifs, which encode critical information about a physical process of interest.
Signals of this type can be modeled as the convolution of a zero-padded short kernel a 0 P R p 0 (the motif) with a longer sparse signal x 0 P R m ( m " p 0 ) which encodes the locations of the motifs in the sample 1 :
[Equation]
We term this a short-and-sparse (SaS) model.
Since often only y is observed, short-and-sparse deconvolution (SaSD) is the problem of recovering both a 0 and x 0 from y .
Variants of SaSD arise in areas such as microscopy (Cheung et al., 2018), astronomy (Briers et al., 2013), and neuroscience (Song et al., 2018).
SaSD is a challenging inverse problem in both theory and practice.
Natural formulations are nonconvex, and very little algorithmic theory was available.
Moreover, practical instances are often ill-conditioned, due to the spectral decay of the kernel a 0 (Cheung et al., 2018).
This paper is motivated by recent theoretical advances in nonconvex optimization and, in particular, on the geometry of SaSD.
Zhang et al.
(2017) and Kuo et al.
(2019) study particular optimization formulations for SaSD and show that the landscape is largely driven by the problem symmetries of
SaSD.
They derive provable methods for idealized problem instances, which exactly recover p a 0 , x 0 q up to trivial ambiguities.
While inspiring, these methods are not practical and perform poorly on real problem instances.
Where the emphasis of Zhang et al.
(2017) and Kuo et al.
(2019) is on theoretical guarantees, here we focus on practical computation.
We show how to combine ideas from this theory with heuristics that better address the properties of practical deconvolution problems, to build a novel method that performs well on data arising in a range of application areas.
A critical issue in moving from theory to practice is the poor conditioning of naturally-occurring deconvolution problems: we show how to address this with a combination of ideas from sparse optimization, such as momentum, continuation, and reweighting.
The end result is a general purpose method, which we demonstrate on data from neural spike sorting, calcium imaging and ﬂuorescence microscopy.
Notation.
The zero-padding operator is denoted by ι : R p Ñ R m .
Projection of a vector v P R p onto the sphere is denoted by P S p ´ 1 p v q .
“ v { } v } 2 , and P z p v q .
“ v ´ x v , z y z denotes projection onto the tangent space of z P S p ´ 1 .
The Riemannian gradient of a function f : S p ´ 1 Ñ R at point z on the sphere is given by grad f p z q .
“ P z p ∇ f p z qq .
S YMMETRY AND G EOMETRY IN S A SD
In this section, we begin by describing two intrinsic properties for SaSD.
Later, we show how these play an important role in the geometry of optimization and the design of efﬁcient methods.
An important observation of the SaSD problem is that it admits multiple equivalent solutions.
This is purely due to the cyclic convolution between a 0 and x 0 , which exhibits the trivial ambiguity
[Equation]
for any nonzero scalar α and cyclic shift s (cid:96) r¨s .
These scale and shift symmetries create several acceptable candidates for a 0 and x 0 , and in the absence of further information we only expect to recover a 0 and x 0 up to symmetry.
Furthermore, they largely drive the behavior of certain nonconvex optimization problems formulated for SaSD.
Since the success of SaSD requires distinguishing between overlapping copies of a 0 , its difﬁculty also depends highly on the “similarity” of the a 0 to its shifts.
Here we capture this notion using the shift-coherence of a 0 ,
[Equation]
Intuitively, the shifts of a 0 become closer together as µ p a 0 q increases (Figure 10), making objective landscapes for optimization less favorable for recovering any speciﬁc shift of a 0 .
2. L ANDSCAPE GEOMETRY UNDER SHIFT - INCOHERENCE
A natural approach to solving SaSD is to formulate it as a suitable optimization problem.
In this paper we will focus on the Bilinear Lasso (BL) problem, which minimizes the squared error between the observation y and its reconstruction a f x , plus a (cid:96) 1 -norm sparsity penalty on x ,
[Equation]
Later in this section, we will see that the kernel length p should be set slightly larger than p 0 .
The Bilinear Lasso is a nonconvex optimization problem, as the shift symmetries of SaSD create discrete local minimizers in the objective landscape.
The regularization created by problem symmetries in nonconvex inverse problems are a fairly general phenomenon (Sun et al., 2015) and, as Kuo et al.
(2019) shows, its inﬂuence in SaSD extends beyond the neighborhoods of these local minimizers.
Kuo et al. analyzed an Approximate Bilinear Lasso (ABL) objective 3 Ψ ABL , which satisﬁes
[Equation]
This non-practical objective serves as a valid simpliﬁcation of the Bilinear Lasso for analysis when the true kernel is itself incoherent, i.e. µ p a 0 q »
(Figures 1d and 1e).
Under its marginalization
[Equation]
certain crucial properties regarding its curvature can be characterized for generic choices of x .
The reason we choose to partial minimize x instead of a is because (i) the problem (4) is convex w.r.t.
x , and (ii) the dimension of the subspace of a is signiﬁcantly smaller than that of x (i.e., p ! m ), which is the place that the measure concentrates.
[Figure]
Curvature in the span of a few shifts.
Suppose we set p ą p 0 , which ensures that we can ﬁnd an a » α 1 s (cid:96) 1 r a
s ` α 2 s (cid:96) 2 r a 0 s P S p ´ 1 that lies near the span of two shifts of a 0 .
If α 1 » ˘(or α 2 » 0 ) then, under suitable conditions on a 0 and x 0 , Kuo et al.
(2019) asserts that a lies in a strongly convex region of ϕ ABL , containing a single minimizer near s (cid:96)
r a
s (Figure 1a); the converse is also true.
A saddle-point exists nearby when α 1 » α 2 is balanced, characterized by large negative curvature along the two shifts and positive curvature in orthogonal directions (Figure 1b).
Interpolating between these two cases, large negative gradients point towards individual shifts.
The behavior of ϕ ABL between two shifts of a 0 — strong convexity near single shifts, and saddlepoints near balanced points — extends to regions of the sphere spanned by several shifts (Figure 1c); we elaborate on this further in Appendix A.1.
This regional landscape guarantees that a 0 can be efﬁciently recovered up to a signed shift using methods for ﬁrst and second-order descent, as soon as a can be brought sufﬁciently close to the span of a few shifts.
Optimization over the sphere.
For both the Bilinear Lasso and ABL, a unit-norm constraint on a is enforced to break the scaling symmetry between a 0 and x 0 .
Choosing the (cid:96) 2 -norm, however, has surprisingly strong implications for optimization.
The ABL objective, for example, is piecewise concave whenever a is sufﬁciently far away from any shift of a 0 , but the sphere induces positive curvature near individual shifts to create strong convexity.
These two properties combine to ensure recoverability of a 0 .
In contrast, enforcing (cid:96) 1 -norm constraints often leads to spurious minimizers for deconvolution problems (Levin et al., 2011; Benichoux et al., 2013; Zhang et al., 2017).
Initializing near a few shifts.
The landscape of ϕ ABL makes single shifts of a 0 easy to locate if a is initialized near a span of a few shifts.
Fortunately, this is a relatively simple matter in SaSD, as y is itself a sparse superposition of shifts .
Therefore, one initialization strategy is to randomly choose a length- p 0 window r y i .
“ r y
[Equation]
from the observation and set
[Equation]
This brings a p 0 q suitably close to the sum of a few shifts of a 0 (Appendix A.2); any truncation effects are absorbed by padding the ends of r y i , which also sets the length for a to be p “ 3 p 0 ´ 2 .
Implications for practical computation.
The (regionally) benign optimization landscape of ϕ ABL guarantees that efﬁcient recovery is possible for SaSD when a 0 is incoherent.
Applications of sparse deconvolution, however, are often motivated by sharpening or resolution tasks (Huang et al., 2009;
Candès & Fernandez-Granda, 2014; Campisi & Egiazarian, 2016) where the motif a 0 is smooth and coherent (i.e. µ p a 0 q is large).
The ABL objective is a poor approximation of the Bilinear Lasso in such cases and fails to yield practical algorithms, so we should optimize the Bilinear Lasso directly.
From Figures 1d and 1e, we can see that low-dimensional subspheres spanned by shifts of a 0 are empirically similar when a 0 is incoherent.
Although this breaks down in the coherent case, as we illustrate in Appendix A.3, the symmetry breaking properties of ϕ BL remain present.
This allows us to apply the geometric intuition discussed here to create an optimization method that, with the help of a number of computational heuristics, performs well in for SaSD even in general problem instances.
Input:
Initializations a p 0 q P S p ´ 1 , x P R m ; observation y P R m ; penalty λ ě 0 ; momentum α P r 0 , 1 q .
Output: p a p k q , x p k q q , a local minimizer of Ψ BL .
Initialize a p 1 q “ a p 0 q , x p 1 q “ x p 0 q . for k “ 1 , 2 , . . .
until converged do Update x with accelerated proximal gradient step:
[Figure]
where soft λ p v q .
“ sign p v q d max p| v ´ λ | , 0 q denotes the soft-thresholding operator.
Update a with accelerated Riemannian gradient step:
[Figure]
end for (
[Figure]
Gradient descent
[Figure]
D ESIGNING A PRACTICAL S A SD ALGORITHM
Several algorithms for SaSD-type problems have been developed for speciﬁc applications, such as image deblurring (Levin et al., 2011; Briers et al., 2013; Campisi & Egiazarian, 2016), neuroscience (Rey et al., 2015; Friedrich et al., 2017; Song et al., 2018), and image super-resolution (Baker &
Kanade, 2002; Shtengel et al., 2009; Yang et al., 2010), or are augmented with additional structure (Wipf & Zhang, 2014; Ling & Strohmer, 2017; Walk et al., 2017).
Here, we instead leverage the theory from Section 2 to build an algorithm for general practical settings.
In addition to applying an appropriate initialization scheme (5) and optimizing on the sphere, we minimize the Bilinear Lasso (3) instead of the ABL (4) to more accurately account for interactions between shifts of a 0 in highly shift-coherent settings.
Furthermore, we also address the negative effects of large coherence using a number of heuristics, leading to an efﬁcient algorithm for SaSD.
Momentum acceleration.
In shift-coherent settings, the Hessian of Ψ BL becomes ill-conditionednear shifts of a 0 , a situation known to cause slow convergence for ﬁrst-order methods (Nesterov, 2013).
A remedy is to add momentum (Polyak, 1964; Beck & Teboulle, 2009) to ﬁrst-order iterations, for instance, by augmenting gradient descent on some smooth f p z q with stepsize τ with the term w ,
[Equation]
Here, α controls the momentum added 6 .
As illustrated in Figure 2, this additional term improves convergence by reducing oscillations of the iterates for ill-conditioned problems.
Momentum has been shown to improve convergence for nonconvex and nonsmooth problems (Pock & Sabach, 2016;
Jin et al., 2018).
Here we provide an inertial alternating descent method (iADM) for ﬁnding local minimizers of Ψ BL (Algorithm 1), which modiﬁes iPALM (Pock & Sabach, 2016) to perform updates on a via retraction on the sphere (Absil et al., 2009) 7 .
Input:
Observation y P R m , motif size p 0 ; momentum α P r 0 , 1 q ; initial λ p 1 q ﬁnal λ ‹ , penalty decrease η P p 0 , 1 q ; precision factor δ P p 0 , 1 q .
Output:
Solution path (cid:32) p ˆ a p n q , ˆ x p n q ; λ p n q q ( for SaSD.
Set number of iterations N Ð X log p λ ‹ { λ p 1 q q { log η \ .
Initialize ˆ a p 0 q P R 3 p 0 ´ 2 using (5), ˆ x p 0 q “ 0 P R m .
for n “ 1 , . . .
, N do
Minimize Ψ λ p n q to precision δλ p n q with Algorithm 1: `
[Figure]
Update λ p n
[Figure]
q Ð ηλ p n q .
end for (
[Figure]
Homotopy continuation.
It is also possible to improve optimization by modifying the objective Ψ BL directly through the sparsity penalty λ .
Variations of this idea appear in both Zhang et al.
(2017) and Kuo et al.
, and can also help to mitigate the effects of large shift-coherence.
When solving (3) in the noise-free case, it is clear that larger choices of λ encourage sparser solutions for x .
Conversely, smaller choices of λ place local minimizers of the marginal objective ϕ
BL p a q .
“ min x Ψ BL p a , x q closer to signed-shifts of a 0 by emphasizing reconstruction quality.
When µ p a 0 q is large, however, ϕ BL becomes ill-conditioned as λ Ñ 0 due to the poor spectral conditioning of a 0 , leading to severe ﬂatness near local minimizers and the creation spurious local minimizers when noise is present (Figure 3).
Conversely, larger values of λ limit x to a small set of support patterns and simplify the landscape of ϕ BL , at the expense of precision.
It is therefore important both for fast convergence and accurate recovery for λ to be chosen appropriately.
When problem parameters — such as noise level, p 0 , or θ — are not known a priori, a homotopy continuation method (Hale et al., 2008; Wright et al., 2009; Xiao & Zhang, 2013) can be used to obtain a range of solutions for SaSD.
Using initialization (5), a rough estimate p ˆ a p 1 q , ˆ x p 1 q q is obtained by solving (3) with iADM using a large choice for λ p 1 q .
This estimate is reﬁned via a solution path (cid:32) p ˆ a p n q , ˆ x p n q ; λ p n q q ( by gradually decreasing λ p n q .
By ensuring that x remains sparse along the solution path, the objective Ψ BL enjoys restricted strong convexity w.r.t.
both a and x throughout optimization (Agarwal et al., 2010).
As a result, homotopy achieves linear convergence for SaSD where sublinear convergence is expected otherwise (Figures 4c and 4d).
We provide a complete algorithm for SaSD combining Bilinear Lasso and homotopy continuation in Algorithm 2.
E XPERIMENTS 4. S YNTHETIC EXPERIMENTS
Here we perform SaSD in simulations on both coherent and incoherent settings.
Coherent kernels are discretized from the Gaussian window function a 0 “ g p 0 , 0 . 5 , where g p,σ .
“ P S p ´`“ exp ` ´ p 2 i ´ p ´ 1 q 2 σ 2 p p ´ 1 q 2 ˘‰ p i “ 1 ˘ .
Incoherent kernels a 0 „ Unif p S
p 0 ´ 1 q are sampled uniformly on the sphere.
Recovery performance.
We test recovery probability for varying kernel lengths p 0 and sparsity rates θ .
To ensure the problem size is sufﬁciently large, we set m “ 100 p 0 .
For each p
and θ ,
we
[Figure] [Figure]
randomly generate 8 x „ i.i.d.
BR p θ q for both coherent and incoherent a 0 .
We solve ten trials of (3) on clean observation data a 0 f x 0 using iADM with λ “ 10 ´ 2 ?
p 0 θ .
The probability of recovering a signed shift of a 0 is shown in Figure 4.
Recovery is likely when sparsity is low compared to the kernel length.
The coherent problem setting has a smaller success region compared to the incoherent setting
Momentum and homotopy.
Next, we test the performance of Algorithm 1 with momentum ( α k “ k ´ 1 k ` 2 ; see Pock & Sabach (2016)) and without ( α “ 0 ).
This is done by minimizing Ψ BL with initialization (5), using clean observations with p 0 “ 10 2 , m “ 10 4 , and θ “ p ´ 3 { 4for coherent and incoherent a 0 .
We also apply homotopy (Algorithm 2) with λ p 1 q “ max (cid:96) | x s (cid:96) r a p 0 q s , y y | — see Xiao & Zhang (2013), λ ‹ “ 0 . 3 ?
p 0 λ , η “ 0 . 8 , and δ “ 0 . 1 .
The ﬁnal solve of (3) uses precision ε ‹ “ 10 ´ 6 , regardless of method.
Figures 4c and 4d show the comparison results on coherent problem settings.
Comparison to existing methods.
Finally, we compare iADM, and iADM with homotopy, against a number of existing methods for minimizing ϕ BL .
The ﬁrst is alternating minimization (Kuo et al., 2019), which at each iteration k minimizes a p k q with x p k q ﬁxed using accelerated (Riemannian) gradient descent with backtracking, and vice versa.
The next method is the popular alternating direction method of multipliers (Boyd et al., 2011).
Finally, we compare against iPALM (Pock &
Sabach, 2016) with backtracking, using the unit ball constraint on a 0 instead of the unit sphere.
For each method, we deconvolve signals with p 0 “ 50 , m “ 100 p 0 , and θ “ p ´ 3 { 4for both coherent and incoherent a 0 .
For both iADM, iADM with homotopy, and iPALM we set α “ 0 . 3 .
For homotopy, we set λ p 1 q “ max (cid:96) | x s (cid:96) r a p 0 q s , y y | , λ ‹ “ 0 . 3 ?
p 0 λ , and δ “ 0 . 5 .
Furthermore we set η “ 0 .or η “ 0 . 8 and for ADMM, we set the slack parameter to ρ “ 0 . 7 or ρ “ 0 . 5 for incoherent and coherent a 0 respectively.
From Figure 6, we can see that ADMM performs better than iADM in the incoherent case, but becomes less reliable in the coherent case.
In both cases, iADM with homotopy is the best performer.
Finally, we observe roughly equal performance between iPALM and iADM.
[Figure]
4. I MAGING APPLICATIONS
Here we demonstrate the performance and generality of the proposed method.
We begin with calcium ﬂuorescence imaging, a popular modality for studying spiking activity in large neuronal populations (Grienberger & Konnerth, 2012), followed by stochastic optical reconstruction microscopy (STORM)
(Rust et al., 2006; Huang et al., 2008; 2010), a superresolution technique for in vivo microscopy 9 .
Sparse deconvolution of calcium signals.
Neural spike trains created by action potentials, each inducing a transient response in the calcium concentration of the surrounding environment.
The aggregate signal can be modeled as a convolution between the transient a 0 and the spike train x 0 .
Whilst a 0 and x 0 both encode valuable information, neither are perfectly known ahead of time.
Here, we ﬁrst test our method on synthetic data generated using an AR2 model for a 0 , a shiftcoherent kernel that is challenging for deconvolution, see e.g. Friedrich et al.
We set x 0 „ i.i.d.
Bernoulli p p ´ 4 { 5q P R 10 4 with additive noise n „ i.i.d.
N p 0 , 5 ¨ 10 ´ 2 q .
Figures 5a and 5b demonstrate accurate recovery of a 0 and x 0 in this synthetic setting.
Next, we test our method on real data 10 ; Figures 5c and 5d demonstrate recovery of spike locations.
Although iADM provides decent performance, in the presence of large noise estimation quality can be improved by stronger sparsiﬁcation methods, such as the reweighting technique by Candes et al.
(2008), which we elaborate on in Appendix B. Additionally, Figure 6c shows that the proposed method converges to higher precision in comparison with state-of-the-art methods.
[Figure]
Super-resolution for ﬂuorescence microscopy.
Fluorescence microscopy is often spatially limited by the diffraction of light; its wavelength (several hundred nanometers) is often larger than typical molecular length-scales in cells, preventing a detailed characterization of subcellular structures.
The
STORM technique overcomes this resolution limit by using photoswitchable ﬂuorescent probes to multiplex the image into multiple frames, each containing a subset of the molecules present
[Figure]
(Figure 7). If the location of these molecules can be precisely determined for each frame, synthesizing all deconvolved frames will produce a super-resolution microscopy image with nanoscale resolution.
For each image frame, the localization task can be formulated via the SaS model
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
where f denotes 2D convolution.
Here we will solve this task on the single-molecule localization microscopy (SMLM) benchmarking dataset 11 via SaSD, recovering both the PSF A
and the point source maps X 0 ,t simultaneously.
We apply iADM with reweighting (Appendix B) on frames of size 128 ˆ 128 from the video sequence “Tubulin”; each pixel is of 100nm 2 resolution 12 , the ﬂuorescence wavelength is 690nm , and the framerate is f “ 25Hz .
Figure 7 shows examples of recovered activation maps, and the aggregated super-resolution image from all 500 frames, accurately predicting the PSF (see Appendix D) and the activation map for each video frame to produce higher resolution microscopy images.
Localization in calcium images.
Our methods are easily extended to handle superpositions of multiple SaS signals.
In calcium imaging, this can potentially be used to track the neurons in video sequences, a challenging task due to (non-) rigid motion, overlapping sources, and irregular background noise Pnevmatikakis et al.
; Giovannucci et al.
We consider frames video obtained via the two-photon calcium microscopy dataset from the Allen Institute for Brain Science 13 , shown in Figure 8.
Each frame contains the cross section of several neurons and dendrites, which have distinct sizes.
We model this as the SaS signal Y t “ ι A 1 f X 1 ,t ` ι A 2 f X 2 ,t , where each summand consists of neurons or dendrites exclusively.
By extending Algorithm 2 to recover each of the kernels A k and maps X k , we can solve this convolutional dictionary learning (SaS-CDL; see
Appendix C) problem which allows us to separate the dendritic and neuronal components from this image for localization of ﬁring activity, etc.
As a result, the application of SaS-CDL as a denoising or analysis tool for calcium imaging videos provides a very promising direction for future research.
D ISCUSSION
Many nonconvex inverse problems, such as SaSD, are strongly regulated by their problem symmetries.
Understanding this regularity and when or how it breaks down is important for developing effective algorithms.
We illustrate this by combining geometric intuition with practical heuristics, motivated by common challenges in real deconvolution, to produce an efﬁcient and general purpose method that performs well on data arising from a range of application areas.
Our approach, therefore, can serve as a general baseline for studying and developing extensions to SaSD, such as SaS-CDL (Bristow Lucey, 2014; Chun & Fessler, 2017; Garcia-Cardona & Wohlberg, 2018), Bayesian approaches (Babacan et al., 2008; Wipf & Zhang, 2014), and hierarchical SaS models (Chen et al., 2013).
A A PPROXIMATE B ILINEAR L ASSO O BJECTIVE
Recall from Section 2.2 of the main text that SaSD can be formulated as the Bilinear Lasso problem
[Equation]
Unfortunately, this objective is challenging for analysis.
A major culprit is that its marginalization
[Equation]
generally does not admit closed form solutions due the convolution with a in the squared error term
This motivates Kuo et al.
(2019) to study the nonconvex formulation
[Equation]
We refer to (11) as the Approximate Bilinear Lasso formulation and it is quite easy to see that Ψ ABL p a , x q « Ψ BL p a , x q when } a f x } 2 « } x } 2 , i.e. if a is shift-incoherent, or µ p a q « 0 .
The marginalized objective function ϕ BL p a q .
“ min x Ψ DQ p a , x q now has the closed form expression
[Equation]
Here soft denotes the elementwise soft-thresholding operator soft t p x i q “ sign p
x
i q ¨ max p| x i | ´ t, 0 q , and q a denotes the adjoint kernel of a , i.e. the kernel s.t.
x
ι a f u , v y “ x u , q a f v y @
u , v P R m .
A. L ANDSCAPE G EOMETRY
The rest of Section 2.2 discusses the regional characterization of ϕ ABL in the span of a small number of shifts from a 0 .
This language is made precise in the form of the subsphere
[Equation]
spanned by a small set of cyclic shifts of ι a 0 .
Although we will not discuss the explicit distance function here, the characterization by Kuo et al.
(2019) holds whenever a is close enough to such a subsphere with | I | ď 4 θp 0 , where θ is the probability that any individual entry of x 0 is nonzero.
Suppose we have a « ř (cid:96) P I α (cid:96) s (cid:96)
r ι a 0 s for some appropriate index set I .
Note that if µ s a 0 « 0 , then µ s a « 0 , @ a P S I .
Now let α p 1 q and α p 2 q be the ﬁrst and second largest coordinates of the shifts participating in a , and let s p 1 q r a 0 s and s p 2 q r a
s be the corresponding shifts.
Then
• If ˇˇˇ α p 2 q α p 1 q ˇˇˇ « 0 , then a is in a strongly convex region of ϕ ABL , containing a single local minimizer corresponding to s p 1 q r a 0 s .
• If ˇˇˇ α p 2 q α p 1 q ˇˇˇ « 1 , then a is near a saddle-point, with negative curvature pointing towards s p 1 q r a 0 s
and s p 2 q r a 0 s .
If ˇˇˇ α p 3 q α p 2 q ˇˇˇ « 0 , i.e. s p 1 q r a 0 s and s p 2 q r a
s
are the only two participating shifts, then ϕ ABL is also characterized by positive curvature in all orthogonal directions.
Otherwise, x´ grad ϕ ABL p a q , z ´ a y takes on a large positive value , for either u “ s p 1 q r a 0 s or u “ s
p 2 q r a 0 s , i.e. the negative Riemannian gradient is large and points towards one of the participating shifts.
This is an example of a ridable saddle property (Jin et al., 2017) that allows many ﬁrst and secondorder methods to locate local minimizers.
Since all local minimizers of ϕ ABL near S I must correspond to signed-shifts of a 0 , this guarantees that the Approximate Bilinear Lasso formulation can be efﬁciently solved to recover a 0 (and subsequently x 0 ) for incoherent a 0 , as long as a is initialized near some appropriate subsphere and the sparsity coherence tradeoff p 0 θ Æ p µ s p a 0 qq ´ 1 { 2 is satisﬁed.
We note that this is a poor tradeoff rate, which reﬂects that the Approximate Bilinear Lasso formulation is non-practical and cannot handle SaSD problems involving kernels with high shift-coherence.
A. D ATA - DRIVEN INITIALIZATION
For the SaS-BD problem, we usually initialize x by x p 0 q “ 0 , so that our initialization is sparse.
For the optimization variable a P R n , recall from Section 2.2 in the main text that it is desirable to obtain an initialization a 0 which is close to the intersection of S p ´ 1 and a subsphere S
I spanned by a few shifts of a 0 .
When x 0 is sparse, our measurement y is a linear combination of a few shifts of a 0 .
Therefore, an arbitrary consecutive p 0 -length window r y i .
“ r y i y i ` 1 . . .
y i
` p 0 ´ 1 s T of the data y should be not far away from such a subspace S I .
As illustrated in Figure 9, one step of the generalized power method (Kuo et al., 2019)
[Equation] [Equation]
¯¯ (15) produces a reﬁned initialization that is very close to a subspace S I spanned by a few shifts of a 0 with | I | « θp 0 .
However, (15) is a relatively complicated for a simple idea.
In practice, we ﬁnd that the simple initialization a p 0 q “ r a p 0 q from (14) works suitably well for solving SaSD with (9).
A. C OMPARISON TO THE B ILINEAR L ASSO
Although it is easy to see that Ψ ABL p a q and Ψ
BL p a q are similar as long as µ p a q « 0 , it is also clear that these two quantities can be very different when µ p a q is large.
This is especially signiﬁcant when µ p a 0 q is itself large, as the desired solutions for a are then also coherent.
From Figure 10, we can see that these changes are reﬂected in the low-dimensional subspheres (13) spanned by adjacent shifts of a 0 .
Compared to the incoherent case, ϕ BL also takes on small values in
[Figure] [Figure]
regions between adjacent shifts, creating a “global valley” on the subsphere.
Theoretically, this makes it difﬁcult to ensure exact recovery of up to symmetry when a 0 is coherent, and the objective function becomes much more complicated.
This is not a signiﬁcant issue in terms of practical computation, however, since adjacent shifts of a 0 become indistinguishable as µ p a 0 q Ñ 1 , meaning that one only needs to ensure that a lands in the “global valley” to achieve good estimates of a 0 up to symmetry
B R EWEIGHTED SPARSE PENALIZATION
When a 0 is shift-coherent, minimization of the objective Ψ BL with respect to x becomes sensitive to perturbations, creating “smudging” effects on the recovered map x .
These resolution issues can be remedied with stronger concave regularizers.
A simple way of facilitating this with the Bilinear
Lasso is to use a reweighting technique (Candes et al., 2008).
The basic idea is to adaptively adjust the penalty by considering a weighted variant of the original Bilinear Lasso problem from (9),
[Equation]
(16) where w P R m ` and d denotes the Hadamard product.
Here we will set the weights w to be roughly inverse to the magnitude of the true signal x 0 , i.e.,
[Equation]
[Equation]
In addition to choosing λ ą 0 , here ε ą 0 trades off between sparsiﬁcation strength (small ε ) and algorithmic stability (large ε ).
Let | x | p i q denote the i -th largest entry of | x | .
For experiments in the
Input:
[Equation]
Output:
[Equation]
Initialize w p 1 q “ 1 m , j Ð 1 . while not converged do
Using the initialization `
[Equation]
and weight
[Equation]
, solve (16) — e.g. with iADM — to obtain solution `
[Equation]
Set ε with (19) and update the weights as
[Equation]
Update (cid:96) Ð (cid:96) ` 1 .
end while
[Figure]
main text, we set
[Equation]
Starting with the initial weights w p 0 q “ 1 m , Algorithm 3 successively solves (16), updating the weights using (17) at each outer loop iteration j .
As j Ñ 8 , this method becomes equivalent to replacing the (cid:96) 1 -norm in (9) with the nonconvex penalty ř i log p| x i | ` ε q (Candes et al., 2008).
We can easily adopt our iADM algorithm to solve this subproblem, by taking the proximal gradient on x with a different penalty
λ
i for each entry x i .
Figure 11, as well as calcium imaging experiments in Section 4.2, Figure 5 of the main text, demonstrate improved estimation as a result of this method.
C E XTENSION FOR CONVOLUTIONAL DICTIONARY LEARNING
The optimization methods we introduced for SaSD here can be naturally extended for sparse blind deconvolution problems with multiple kernels/motifs (a.k.a.
convolutional dictionary learning; see
Garcia-Cardona & Wohlberg (2018)), which have broad applications in microscopy data analysis
[Equation]
[Figure]
[Equation]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
(Yellin et al., 2017; Zhou et al., 2014; Cheung et al., 2018) and neural spike sorting (Ekanadham et al., 2011; Rey et al., 2015; Song et al., 2018).
As illustrated in Figure 12, the new observation y is the sum of N convolutions between short kernels t a 0 ,k u Nk “ 1 and sparse maps t x 0 ,k u Nk “ 1 ,
[Equation]
The natural extension of SaSD, then, is to recover t a 0 ,
k u Nk “ 1 and t x 0 ,k u Nk “ 1 up to signed, shift, and permutation ambiguities, leading to the SaS convolutional dictionary learning (SaS-CDL) problem.
The SaSD problem can be seen as a special case of SaS-CDL with N “ 1 .
Based on the Bilinear
Lasso formulation in (9) for solving SaSD, we constrain all kernels a 0 ,k over the sphere, and consider the following nonconvex objective:
[Equation]
Similar to the idea of solving the Bilinear Lasso in (9), we optimize (21) via iADM, by taking alternating descent steps on t a k u Nk “ 1 and t x k u Nk “ 1 with the other variable ﬁxed.
D S UPER - RESOLUTION WITH STORM IMAGING
For point source localization in STORM frames, recall that we use the SaS model from Section 4.2.2,
[Equation]
[Figure]
[Equation]
We then apply our SaSD method to recover both A 0 and X 0 ,t from Y t .
We show our recovery of
X 0 ,t as well as the super-resolved image using all available frames in Figure 6 of the main text.
Since the main objective of STORM imaging is to recover the point sources, we have deferred the recovered
PSF A 0 to Figure 13 here.