S TAR -C ONVEXITY IN N ON -N EGATIVE M ATRIX F ACTORIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Non-negative matrix factorization (NMF) is a highly celebrated algorithm for matrix decomposition that guarantees non-negative factors.
The underlying optimization problem is computationally intractable, yet in practice gradient descent based solvers often ﬁnd good solutions.
This gap between computational hardness and practical success mirrors recent observations in deep learning, where it has been the focus of extensive discussion and analysis.
In this paper we revisit the NMF optimization problem and analyze its loss landscape in non-worst-case settings.
It has recently been observed that gradients in deep networks tend to point towards the ﬁnal minimizer throughout the optimization.
We show that a similar property holds (with high probability) for NMF, provably in a non-worst case model with a planted solution, and empirically across an extensive suite of real-world NMF problems.
Our analysis predicts that this property becomes more likely with growing number of parameters, and experiments suggest that a similar trend might also hold for deep neural networks — turning increasing data sets and models into a blessing from an optimization perspective.
I NTRODUCTION
Non-negative matrix factorization (NMF) is a ubiquitous technique for data analysis where one attempts to factorize a measurement matrix X into the product of non-negative matrices U , V (Lee and Seung, 1999).
This simple problem has applications in recommender systems (Luo et al., 2014), scientiﬁc analysis (Berne et al., 2007; Trindade et al., 2017), computer vision (Gillis, 2012), internet distance prediction (Mao et al., 2006), audio processing (Schmidt et al., 2007) and many more domains.
Often, the non-negativity is crucial for interpretability, for example, in the context of crystallography, the light sources, which are represented as matrix factors, have non-negative intensity (Suram et al., 2016).
Like many other non-convex optimization problems, ﬁnding the exact solution to NMF is NP-hard (Pardalos and Vavasis, 1991; Vavasis, 2009).
NMF’s tremendous practical success is however at odds with such worst-case analysis, and simple algorithms based upon gradient descent are known to ﬁnd good solutions in real-world settings (Lee and Seung, 2001).
At the time when NMF was proposed, most analyses of optimization problems within machine learning focused on convex formulations such as SVMs (Cortes and Vapnik, 1995), but owing to the success of neural networks, non-convex optimization has experienced a resurgence in interest.
Here, we revisit NMF from a fresh perspective, utilizing recent tools developed in the context of optimization in deep learning.
Speciﬁcally, our main inspiration is the recent work of Kleinberg et al.
(2018) and Zhou et al.
(2019) that empirically demonstrate that gradients typically point towards the ﬁnal minimizer for neural networks trained on real-world datasets and analyze the implications of such convexity properties for efﬁcient optimization.
In this paper, we show theoretically and empirically that a similar property called star-convexity holds in NMF.
From a theoretical perspective, we consider an NMF instance with planted solution, inspired by the stochastic block model for social networks (Holland et al., 1983; Decelle et al., 2011) and the planted clique problem studied in sum-of-squares literature (Barak et al., 2016).
We prove that between two points the loss is convex with high probability, and conclude that the loss surface is star-convex in the typical case — even if the loss is computed over unobserved data.
From an empirical perspective, we verify that our theoretical results hold for an extensive collection
[Table]
[Figure]
of real-world datasets spanning collaborative ﬁltering (Zhou et al., {+
 ;+} Kula, 2017; Harper and
Konstan, 2016), signal decomposition (Zhu, 2016; Li and Ngom, 2013; Li et al., 2001; Erichson et al., 2018) and audio processing (Flenner and Hunter, 2017; canto Foundation), and demonstrate that the star-convex behavior results in efﬁcient optimization.
Finally, we show that star-convex behavior becomes more likely with growing number of parameters, suggesting that a similar result may hold as neural networks become wider.
We provide supporting empirical evidence for this hypothesis on modern network architectures.
NMF AND STAR - CONVEXITY
The aim of NMF is to decompose some large measurement matrix X P R n ˆ m into two non-negative matrices U P R n ˆ r ` and V P R r ˆ m ` such that X « UV .
The canonical formulation of NMF is
[Equation]
[Equation] [Equation] p U , V q , where (cid:96)
[Equation]
NMF is commonly used in recommender systems where entries p
i, j q of X for example correspond to the rating user i has given to movie j (Luo et al., 2014).
In such settings, data might be missing as all users have not rated all movies.
In those cases, it is common to only consider the loss over observed data (Zhang et al., 2006; Candès and Recht, 2009).
We let ˆ1 p i,j q be an indicator variable that is 1 if entry p i, j q is "observed" and 0 otherwise.
The loss function is then
[Equation]
NMF is similar to PCA which admits spectral strategies; however, the non-negative constraints in
NMF prevent such solutions and result in NP-hardness (Vavasis, 2009).
Work on the computational complexity of NMF has shown that the problem is tractable for small constant dimensions r via algebraic methods (Arora et al., 2012).
In practice, however, these algorithms are not used, and simple variants of gradient descent, possibly via multiplicative updates (Lee and Seung, 2001), are popular and are known to work reliably (Koren et al., 2009).
This gap between theoretical hardness and practical performance is also found in deep learning.
Optimizing neural networks is NP-hard in general (Blum and Rivest, 1989), but in practice they can be optimized with simple stochastic gradient descent algorithms to outmatch humans in tasks such as face veriﬁcation (Lu and Tang, 2015) and playing Atari-games (Mnih et al., 2015).
Recent work on understanding the geometry of neural network loss surfaces has promoted the idea of convexity properties.
The work of Izmailov et al.
(2018) shows that the loss surface is convex around the local optimum, while Zhou et al.
and Kleinberg et al.
(2018) show that the gradients during optimization typically point towards the local minima the network eventually converges to.
Of central importance in this line of work is star-convexity , which is a property of a function f that guarantees that it is convex along straight paths towards the optima x ˚ .
See Figure 2 for an example.
Formally, it is deﬁned as follows.
Deﬁnition 1.
A function f : R n Ñ R is star-convex towards x ˚ if for all λ P r 0 , 1 s and x P R n , we have f
[Equation]
Optimizing star-convex functions can be done in polynomial time (Lee and Valiant, 2016), in Kleinberg et al.
it is shown how the function only needs to be star-convex under a natural noise model.
NMF is not star-convex in general as it is
NP-hard, however, it is natural to conjecture that NMF is starconvex in the typical case.
Such a property could explain the practical success of NMF on real-world datasets, which is not worst-case.
This will be the working hypothesis of this paper, where the typical case is formalized in Theorem 1.
Indeed, one can verify numerically that NMF is typically star-convex for natural distributions and realistically sized matrices, see Figure 1 where we consider a rank 10 decomposition of p 100 , 100 qmatrices with iid half-normal entries and a planted solution, sampled as per Assumption 1 given in the next section.
Following sections will be dedicated to proving that NMF is starconvex with high probability in a planted model, and to conﬁrm that this phenomenon generalizes to datasets from the real world, which are far from worst-case.
P ROVING TYPICAL - CASE STAR - CONVEXITY
Our aim now is to prove that the NMF loss-function is typically star-convex for natural non-worstcase distribution of NMF instances.
We will consider a slighly weaker notation of star-convexity where f
[Equation]
holds not for all x , but for random x with high probability.
This is in fact the best achievable, an NMF instance with u 1 “ 1 , u ˚ “ 0 and v 1 “ 0 , v ˚ “ 1 is not star-convex.
Our results hold with high probability in high dimensions, similar to Dvoretzky’s theorem in convex geometry (Dvoredsky, 1961; Davis).
Inspired by the stochastic block model of social networks (Holland et al., 1983; Decelle et al., 2011) and the planted clique problem (Barak et al., 2016), we will focus on a setting with a planted random solution.
In section 4 we verify conclusions drawn from this model transfers to real-world datasets.
We will assume that there is a planted optimal solution p U ˚ , V ˚ q , where entries of these matrices are sampled iid from a class of distributions with good concentration properties that include the halfnormal distribution and bounded distributions.
As is standard in random matrix theory (Vershynin, 2010), we will develop non-asymptotic results that hold with a probability that grows as the matrices of shape p n, r q and p r, m q become large.
For this reason, we will need to specify how r and m depend on n .
Assumption 1.
For p U , V q P R n ˆ r ˆ R r ˆ m we assume that the entries of the matrices U , V are sampled iid from a continuous distribution with non-negative support that either p i q is bounded or p ii q can be expressed as a 1 -Lipschitz function of a Gaussian distribution.
As n Ñ 8 , we assume that r grows as n γ up to a constant factor for γ P r 1 { 2 , 1 s , and m as n up to a constant factor.
We are now ready to state our main results, that the loss function equation 1 is convex on a straight line between points samples as per Assumption 1, and thus satisfy our slightly weaker notion of star-convexity, with high probability.
The probability increases as the size of the problem increases, suggesting a surprising beneﬁt of high dimensionality.
We also show similar results for the loss function of equation 2 with unobserved data under the assumption that the event that any entry is observed occurs independently with constant probability p .
Below we sketch the proof idea and key ingredients, the formal proof is given in Appendix D.
Theorem 1.
( Main ) Let matrices U 1 , V 1 , U 2 , V 2 , U ˚ , V ˚ be sampled according to Assumption 1.
Then there exists positive constants c 1 , c 2 , such that with probability ě 1 ´ c 1 exp p´ c 2 n 1 { 3 q , the loss function (cid:96) p U , V q in equation 1 is convex on the straight line p U 1 , V 1 q Ñ p U 2 , V 2 q .
The same holds along the line p
U 1 , V 1 q Ñ p
U ˚ , V ˚ q .
It also holds if any entry p i, j q is observed independently with constant probability p , but with probability ě 1 ´ c 1 exp p´ c 2 r 1 { 3 q .
3. P ROOF STRATEGY
Let us parametrize the NMF solution one gets along the line p U 2 , V 2 q Ñ p
U 1 , V 1 q as
[Equation]
Proving Theorem 1 amounts to showing that the loss function (cid:96) p λ q “ 12 } ˆ X p λ q ´ X } 2 F is convex in λ with high probability, our strategy is to show that its second-derivate is non-negative.
For ﬁxed matrices U 1 , U 2 , U ˚ , V 1 , V 2 , V ˚ , the function (cid:96) p λ q is a fourth-degree polynomial in λ , so its second derivate w.r.t.
λ will be a second-degree polynomial in λ .
For a general second-degree polynomial p p x q “ ax 2 ` bx ` c we have p p x q “ 1 a “` ax ` b˘ 2 ` ` ac ´ b 2 4 ˘‰ .
If a ą 0 , which in the case here (see Appendix D), showing that it’s positive could be done by showing ac ě b 2 4 .
This is equivalent to showing that
[Equation]
Where the matrices W 0 , W 1 , W 2 are given as W
“ U 2 V 2 ´ U ˚ V ˚ , W 1 “
[Equation]
U
[Equation]
˘
[Equation]
V
[Equation]
V˘ .
By slight abuse of notation, we have used x A , B y to denote Tr p AB T q for matrices A , B of the same shape.
By exchanging terms in equation 3 by their means one gets
[Equation]
Here, σ 2 is the variance of the distribution of the entries in the matrices, while µ is the mean.
By just counting terms of order p rmnσ 4 q 2 , we see that the LHS has 64 such terms while the RHS has only 48.
Thus, if all matrices W 0 , W 1 and W 2 would exactly be equal to their mean, inequality equation 3 would hold.
In proving that it holds in general, we will use concentration of measure results from random matrix theory to show that the terms will be concentrated around their means and that large deviations are exponentially unlikely.
Concentration of measure Consider the matrix W
[Equation]
Given that all matrices are iid we can center the variables such that W
[Equation]
¯
[Equation]
[Equation]
¯ V 2 ˘ , where the bar denotes the centered matrices.
The term } W 2 } 2 F can then be written as Tr
[Equation]
˘ .
Given that all matrix entries are independent as per Assumption 1, we would expect some concentration of measure to hold.
Bernstein-type inequalities turns out to be too weak for our purposes, but the ﬁeld of random matrix theory offer stronger results for matrices with independent sub-Gaussian entries (Ahlswede and Winter, 2002;
Tropp, 2012; Guionnet et al., 2000; Meckes and Szarek, 2012).
Via such results one can achieve the following concentration result, see Appendix D.
[Equation]
where c 3 , c 4 are positive constants.
In some expression we will however not be able to center all variables, and for such expression one gets similar but slightly weaker concentration results where the exponent scales as n 1 { 3 instead of n , see Appendix D.
Proof sketch Given that E “ } W 2 } 2 F “ 4 rmnσ 4 , equation 5 says that the probability that the term deviates from its mean by a relative factor (cid:15) is less than c 3 exp ` ´ c 5 (cid:15) 2 n ˘ for some small (cid:15) .
By applying similar arguments to terms x W 0 , W 2 y and x W 1 , W 2 y , one can show that the probability that they will deviate by a relative factor (cid:15) is less than c 6 exp p´ c 7 (cid:15) 2 n 1 { 3 q .
A problematic term is } W 1 } 2 F which contains a term of the type Tr
[Equation]
which has weak concentration properties.
Matrices of type A T A are psd, and psd matrices have non-negative trace.
Hence, this term will be non-negative, and since it occurs on the LHS of equation 3, we can simply omit it to lower bound the convexity.
Using union bound, we bound the probability that at least on term deviate with a relative factor (cid:15) by c 1 exp p´ c 8 (cid:15) 2 n 1 { 3 q for positive constants c 1 , c 8 .
Now, set (cid:15) “ 0 .
If neither variable deviates by a factor of more than 0 .
01 , then equation 10 still holds since 0 .
99 2 ˚ 64 ě 1 .
01 2 ˚ 48 .
Thus, inequality is violated with probability at most c 1 exp ` ´ c 2 n 1 { 3 ˘ for positive c 1 , c 2 .
(cid:4)
[Table]
/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ==</latexit> metabolic satellite extragalactic netﬂix movielens goodbooks
ORL
[Figure]
[Table]
[Table]
and
[Table]
[Table] (Zhu, 2016)
[Table]
50 q
[Table]
(Kula, 2017)
[Table]
, 36 , 3 q (Li and Ngom, 2013) [Table] p 3953 ,
[Table]
, 20 q [Table] (Harper and Konstan, 2016)
[Table]
p 47928 , 8963 , 20 q 0 .(Zhou
et al., 2008)
[Table]
p 400 , 10304 , 49 q (Li et al., 2001).
[Table]
p 162 , 94249 , 4 q (Erichson et al., 2018).
Proof sketch for unobserved data If the entries in equation 2 are "observed" independently with probability p , for ﬁxed matrices U 1 , U 2 , U ˚ , V 1 , V 2 , V ˚ such that Theorem 1 hold, we have
E r (cid:96)
[Equation]
Thus, the expectation of (cid:96) 2 p λ q is convex.
To show that it is convex with high probability, one ﬁrst observes that with high probability, no entry p i, j q in (cid:96) 2 p λ q is particularly large.
Assuming this holds via union bound, for ﬁx matrices U 1 , U 2 , U ˚ , V 1 , V 2 , V ˚ with elements that are "observed" independently with probability p , one gets that (cid:96) 2 p λ q is concentrated around its convex mean via
Hoeffding bounds (Hoeffding, 1994).
(cid:4)
E XPERIMENTS 4. V ERIFYING THEORETICAL PREDICTIONS
To verify that the conclusions from our planted model hold more broadly, we now consider realworld datasets previously studied in NMF literature.
Some have ranks outside the scope of our theoretical model but still display star-convexity properties, indicating that it might be a general phenomenon.
We focus on a handful of representative datasets spanning image analysis, scientiﬁc applications and collaborative ﬁltering.
In Table 1, we list these datasets together with references and sparsity, the decomposition rank we use is based upon values previously used in the literature.
We perform a non-negative matrix factorization via gradient descent, starting with randomly initialized data.
To enable comparison between datasets, we scale all data matrices so that the variance of
[Table]
/pxoFk3xWGWSGp5RN6Ij3LVU05sbPF+vOyIVVQhIl2j6FZKH+nshpbMw0Dmwypjg2q95c/M/rZxjd+LlQaYZcseVHUSYJJmR+OwmF5gzl1BLKtLC7EjammjK0DVVsCd7qyeukc1X33Lr3cF1r3hZ1lOEMzuESPGhAE+6hBW1gMIFneIU3J3VenHfnYxktOcXMKfyB8/kDPMWPfQ==</latexit> metabolic satellite
[Figure]
observed entries is 1 and divide the loss function by the number of (observed) entries.
The initialization is performed with half-normal distribution scaled so that the means match with the dataset.
For simplicity we use the same gradient descent parameters for all datasets, a learning rate of 1e ´ 7 with 1e4 gradient steps which gives convergence for all datasets.
For the collaborative ﬁltering datasets (movielens, netﬂix, and goodbooks) we have unobserved ratings, as is standard in NMF we only compute the loss over observed ratings (Zhang et al., 2006).
In Figure 3, we plot the loss function between two random points drawn from the initialization distribution and see that the loss is convex.
In Figure 4 we plot the loss function from an initialization point to an independent local optima.
These results agree with our planted model; the NMF loss-surface of real-world datasets seem to be largely convex along straight paths.
Gradient descent of course only gives local optima but these still display nice star-convexity properties, however, ﬁnding the true global optima remains a challenge.
4. A BLATION EXPERIMENTS
Theorem 1 suggests that as the matrices become larger, NMF is increasingly likely to be star-convex.
We are interested to see if this is the case for our real-world datasets, and to this end, we perform ablation experiments varying the dimensions of the matrices.
We decrease the number of data points n by subsampling rows and columns uniformly randomly.
Our measure of curvature at a point x , given some optimal solution x ˚ , is
[Equation]
Note that α ě 0 implies star-convexity.
In practice, x and x ˚ are obtained by random initialization and gradient descent as per the earlier section.
For each dataset and subsample rate, we ﬁndoptima and evaluate the curvature from 50 random points, thus giving 5000 samples of α .
Figure 5 show how the relative deviation, σµ , of α decrease as the dataset becomes larger.
As shown in
Appendix C, the curvature is always positive, thus the curvature becomes increasingly concentrated around its positive mean for larger matrices.
We are also interested in investigating whether the loss surface is star-convex during training.
In Figure 6, we show that the cosine similarity between the negative gradients ´ ∇ (cid:96) p U , V q and the straight line from x 0 to x ˚ is always positive, and how the loss surface is always star-convex during training.
As per (Lee and Valiant, 2016), star-convexity implies efﬁcient optimization.
In Figure 7, we illustrate the spectrum of singular values for U ˚ , found for the birdsong dataset, and a random matrix of the same shape with iid entries from a halfnormal distribution.
The spectra are similar, and while U ˚ is not random, it seems to share structural qualities with the random matrices used in our proofs.
See Appendix C for ﬁgures on more datasets.
[Figure]
step step step step c
o s i n e c
o s i n e c u r v a t u r e c
u r v a t u r e a) b) c) d)
4. I MPLICATIONS FOR NEURAL NETWORKS
We have seen how increasing the number of parameters makes NMF problems more likely to be starconvex and Figure 5 shows how more parameters make the curvature tend towards its positive mean.
Theorem 1 suggest that this is a result of concentration of measure and it is natural to believe that similar phenomenon would persist in the context of neural networks.
It has previously been observed how neural networks are locally convex (Izmailov et al., 2018; Kleinberg et al., 2018) and also how overparameterization is important in deep learning (Arora et al., 2018; Frankle and Carbin, 2018).
Based upon our observations in NMF we hypothesise that a major beneﬁt of overparameterization is in making the loss surface more convex via concentration of measure w.r.t. the weights.
[Figure]
[Equation]
[Table]
> <latexit sha1_base64="M4Mf/l4lQutxfxP2NbHDLfNjVhE=">AAACDHicbVDLSgMxFM34rPVVdekmWARXZUYEXRbduKxgH9Ap5U6amYYmkyHJiGWYD3Djr7hxoYhbP8Cdf2PajqCtBwKHc87l5p4g4Uwb1/1ylpZXVtfWSxvlza3tnd3K3n5Ly1QR2iSSS9UJQFPOYto0zHDaSRQFEXDaDkZXE799R5VmMr4144T2BEQxCxkBY6V+pepzGWE/VEAyX7NIQJ/lPyzzBdznuU25NXcKvEi8glRRgUa/8ukPJEkFjQ3hoHXXcxPTy0AZRjjNy36qaQJkBBHtWhqDoLqXTY/J8bFVBjiUyr7Y4Kn6eyIDofVYBDYpwAz1vDcR//O6qQkvehmLk9TQmMwWhSnHRuJJM3jAFCWGjy0Bopj9KyZDsMUY21/ZluDNn7xIWqc1z615N2fV+mVRRwkdoiN0gjx0juroGjVQExH0gJ7QC3p1Hp1n5815n0WXnGLmAP2B8/ENG9WcSg==</latexit>
To verify this hypothesis, we consider image classiﬁcation on
CIFAR10 (Krizhevsky et al., 2014) with Resnet networks (He et al., 2016) trained with standard parameters (see Appendix
B).
Networks are typically only locally convex, a property we quantify as the length of subsets of random "lines" in parameter space, along which the loss is convex.
We sample a random direction r
and then consider a subspace of length one along this direction centered around the current parameters w .
We then deﬁne the convexity length scale as the length of the maximal sub-interval r λ 1 , λ 2 s containing 0 on which (cid:96) p w ` λ r q is convex.
Directions are sampled from Gaussian distributions and then normalized for each ﬁlter f to have the same norm as the weights of f .
Table 2 show how this length-scale of local convexity varies with depth, width, and training, where width is varied by multiplying the number of channels by k .
Increased width indeed makes the landscape increasingly locally convex for all but the most shallow networks, results that support our hypothesis.
See Appendix B for extended tables.
[Table]
R ELATED WORK
Our work was initially motivated by ﬁndings in Kleinberg et al.
(2018) and Zhou et al.
(2019) regarding star-convexity in neural networks.
As the success of deep learning has become apparent, researchers have empirically investigated neural network trained on real-world datasets (Li and Yuan, 2017; Keskar et al., 2016).
In the context of sharp vs ﬂat local minima (Keskar et al., 2016), Li et al.
(2018) illustrate how width improved ﬂatness in a Resnet network, an observation Table 2 quantiﬁes.
In general, real-world datasets seem to be far more well-behaved than the worst case, given that training neural networks are NP-hard (Blum and Rivest, 1989).
There is extensive work on nonworst-case analysis of algorithms and machine learning models, and on what problem distributions can guarantee tractability which addresses such gaps (Bilu and Linial, 2012; Afshani et al., 2017).
On the positive side Arora et al.
(2012) has shown an exact algorithm for NMF that runs in polynomial time for small constant r , and there are positive results for ’separable’ NMF (Donoho and
Stodden, 2004).
Compressive sensing (Candes et al., 2004), smoothed analysis (Spielman and Teng, 2004) and problems with "planted" solutions (Barak et al., 2016)
(Holland et al., 1983) similarly makes assumptions on the input.
Researchers have also been interested in theoretical convergence properties of shallow and linear networks (Lu and Kawaguchi, 2017), where a common theme is that functions with only saddle points and global minima can be effectively optimized (Ge et al., 2015).
In analysis of neural networks, random matrix theory often plays a role, directly or indirectly (Choromanska et al., 2015; Glorot and Bengio, 2010; Pennington and Bahri, 2017; Xiao et al., 2018).
C ONCLUSIONS
This paper revisits NMF, a non-convex optimization problem in machine learning.
We have shown that NMF is typically star-convex, provably for a natural average-case model and empirically on an extensive set of real-world datasets.
Our results support the counter-intuitive observation that optimization might sometimes be easier in higher dimensions due to concentration of measure effects.
A D ATASETS
Here we describe the datasets we used to evaluate our results on and provide references for them.
Birdsong : time series for bird calls, see (
Flenner
and Hunter, 2017; canto
Foundation
Extragalactic : dataset for extragalactic light sources, see (Zhu, 2016).
Goodbooks : user ratings for books, see (Kula, 2017; Bourgin, 2018).
Metabolic :
Yest metabolic activity time series, see (Li and Ngom, 2013).
Movielens : user ratings for movies , see (Harper and Konstan, 2016;
Li et al., 2012; Zhang et al., 2006).
Netﬂix : User ratings for movies, see (Zhou et al., 2008; Ampazis, 2008; Li et al., 2012).
ORL faces : black and white facial images, see (Li et al., 2001; Hoyer, 2004).
Satellite : satellite urban spectral-image, see (Erichson et al., 2018; Gillis, 2014).
B DNN PARAMETERS
Table 3 presents the hyper-parameters used for training the neural networks.
Data augmentation consists of randomized cropping with 4 -padding and randomized horizontal ﬂipping.
The learning rate is decreased by a factor of 10 after epochs p 150 , 225 , 275 q .
We use the standard Resnet architecture for CIFAR10 with three segments with p 16 , 32 , 64 q channels each, and use the standard blocks (i.e. not bottleneck blocks)
(He et al., 2016).
[Table]
C E
XTENDED NMF FIGURES
D
P ROOFS
D. O VERVIEW
We here provide the proof of Theorem 1.
In section D.2 we present the notation and derive the main inequality.
In Section D.3, we prove that the loss function is convex between any two random points with high probability.
Section D.4, we prove that this also holds towards the planted solution.
For the case with unobserved data, we prove that the loss function is convex between any two random points with high probability in Section D.5.
That this holds towards the planted solution in the case of unobserved data follows from previous sections
[Table]
and the proof of this fact omitted.
Together, these results form Theorem 1 as stated in the main text.
Section D.6 presents the concentration
[Table] [Table]
birds
[Table]
metabolic
[Table]
c u r v a t u r e c u r v a t u r e
[Table]
https://www.kaggle.com/rtatman/british-birdsong-dataset/data extragalactic https://s3.us-east-2.amazonaws.com/setcoverproblem/Extra ...
... galacticTest/Extragalatic_Archetype_testsample_spec.fits goodbooks https://www.kaggle.com/zygmunt/goodbooks-10k metabolic ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE3nnn /GSE3431/matrix/GSE3431_series_matrix.txt.gz movielens https://www.kaggle.com/prajitdatta/movielens-100k-dataset netﬂix https://www.kaggle.com/netflix-inc/netflix-prize-data
ORL
https://github.com/chibuta /2Layer_convolution
/blob/master/ORL_faces.npz.zip satellite http://www.escience.cn/system/file?fileId= results we need.
The proofs include considerable amounts of elementary algebraic calculations.
For completeness and ease of reference, we present these in Appendix E.
[Figure] [Figure]
D. N OTATION AND THE M AIN I NEQUALITY
The proof will involve various constants c 0 , c 1 , c 2 that do not depend on n .
For convenience, we will let exact values be context dependent and will reuse the symbols c 0 , c 1 , c 2 and so on.
The proofs will not depend on the exact values of these constants.
Constants in our main result can be improved slightly, but at the cost of a more complicated expression.
We present the non-optimized version.
We will let boldface capital letters denote matrices, boldface lower-case letter vectors and non-boldface letters denote scalars.
Let U 1 , U 2 , U ˚ be n -by- r matrices and V 1 , V 2 , V ˚ r -by- m matrices sampled as per Assumption 1.
Without loss of generality, we assume m “ nc m where
ă c m ď 1 .
We will let µ var be the expectation, and σ 2 be the variance of the entry-wise distributions.
We will denote the centered variables by ¯ U 1 , ¯ V 1 and so on, and the mean matrix of U and V resp as 1 n ˆ r , 1 r ˆ m .
Then U “ ¯ U ` 1 n ˆ r .
By slight abuse of notation, we will use the convention x X , Y y
“ Tr `
XY T ˘ for matrices X , Y of the same shape.
The loss function we wish to minimize, subject to non-negativity constraints, is
[Table]
+sbm5Wt6vbO7t5+7eCwrZNMUebRRCSqGxLNBJfMM9wI1k0VI3EoWCcc3xZ+54kpzRP5YCYpC2IylDzilBgr+X5MzCiMcu/xfNqv1Z2GMwNeJm5J6lCi1a99+YOEZjGThgqidc91UhPkRBlOBZtW/UyzlNAxGbKepZLETAf5LPMUn1plgKNE2ScNnqm/N3ISaz2JQztZZNSLXiH+5/UyE10HOZdpZpik80NRJrBJcFEAHnDFqBETSwhV3GbFdEQUocbWVLUluItfXibti4brNNz7y3rzpqyjAsdwAmfgwhU04Q5a4AGFFJ7hFd5Qhl7QO/qYj66gcucI/gB9/gDoSZGV</latexit> birdsong extragalactic goodbooks metabolic movielens netﬂix
ORL satellite
[Figure]
[Table]
[Figure]
a random matrix of the same shape with entries drawn iid from a half-normal distribution.
For the datasets with larger ranks, the spectra typically agree signiﬁcantly.
[Equation]
When we interpolate the solution UV between any ﬁxed two points p U 1 , V 1 q and p U 2 , V 2 q , we write
[Equation]
The loss is then
[Equation]
Proving Theorem 1 in the case of two random points and completely observed data amounts to showing that the continuous function (cid:96) p λ q is convex in λ on the interval r 0 , 1 s with high probability.
We will show that the second-derivate is non-negative.
For ﬁxed matrices U 1 , U 2 , U ˚ , V 1 , V 2 , V ˚ , the loss (cid:96) p λ q is a fourth-degree polynomial in λ , so its second derivate w.r.t.
λ will be a seconddegree polynomial.
For a general second-degree polynomial p p x q “ ax 2 ` bx ` c we have p p x q “ 1 a “` ax ` b˘ 2 ` ` ac ´ b 2 4 ˘‰ .
By inspecting equation 8 one can conclude that a ą 0 .
Showing that the polynomial is non-negative can then be done by showing ac ě b 2 4 .
Let ˆ X ij p λ q be element
i, j of the matrix ˆ X p λ q .
The derivates of (cid:96) p λ q are then
[Equation]
Where we have
[Equation]
Let us deﬁne
[Equation]
By expanding (cid:96) 2 p λ q in a McLaurin series around 0 , we see that proving non-negativity of (cid:96) 2 p λ q can be done by proving 2 (cid:96) p 0 q 2 (cid:96) 4 p 0 q ě (cid:96) 3 p 0 q 2 .
Thus, proving Theorem 3 then amounts to showing that
[Equation]
D. P ROVING THE I NEQUALITY B ETWEEN R ANDOM P OINTS
Theorem 2.
Let p U 1 , V 1 q , p U 2 , V 2 q and the planted solution p U ˚ , V ˚ q be sampled as per Assumption 1.
The loss function equation 7 is convex on a straight line connecting p U 1 , V 1 q and p
U 2 , V 2 q with probability
ě 1 ´ c 1 exp `
[Equation]
for positive constants
[Equation]
Proof.
As per section D.2, we only need to prove that equation 9 holds with probability ě 1 ´
[Equation]
[Equation]
˘ .
Let us exhange all terms in equation 9 with their means, which are given in
Facts 7, 5, 6 and
[Equation]
Clearly, equation 9 would hold if one naively exchanges the terms for their means.
Just counting terms of order rmnσ 4 we have 64 on the LHS and 48 on the RHS.
We need to show that deviations sufﬁciently large to violate the inequality are unlikely.
Our strategy will be to show that the factors are exponentially unlikely to deviate substantially individually, and then use union bounds.
The concentration results and their derivations are detailed in Section D.6.
Consider the random variable } W 2 } 2 F .
As per equation 14, } W 2 } 2 F can be rewritten into polynomials of centered random matrices where we have to pad the matrices with rows/columns of all zeros.
We can then apply the concentration result of Fact 1 to conclude that the probability that } W 2 } 2 F deviates from its mean by a factor p 1 ` (cid:15) q is no more than c 1 exp p´ c 2 (cid:15) 2 n ˘ .
Now, consider the random variable } W 1 } 2 F .
As in the proof of Fact 7, } W 1 } 2 F can be rewritten as the sum of expression equation 16 plus terms of centered variables.
The latter expressions are polynomials in centered random matrices, after padding some columns/rows with zeros we can apply Fact 1 to conclude that the probability that it deviates from its mean by a factor p 1 ` (cid:15) q is no more than c 1 exp p´ c 2 (cid:15) 2 n ˘ .
Consider the expression equation 16.
Clearly the expressions
[Equation]
` ¯ U 1 ´ ¯ U 2 ˘ T ` ¯ U [-
-]1 ´ ¯ U 2 ˘ 1 r ˆ m and ` ¯ V [-
-]1 ´ ¯ V 2 ˘ T 1 Tn ˆ r 1 n ˆ r ` ¯ V 1 ´ ¯ V 2 ˘
[-´ ¯ V 2 ˘ T 
 Tn ˆ r 1 n ˆ r ` ¯ V 1 ´ ¯ V 2 ˘-] will be psd, and thus their trace will be nonnegative.
We can thus lower bound the LHS of equation 10 by omitting the terms in equation 16, and will thus only consider terms scaling as rn 2 σ 4 .
Now, consider the random variable x W 0 , W 2 y .
As per equation 19, the variable x W 0 , W 2 y can also be separated into two parts.
After padding with some zero rows/columns to obtain square matrices, the ﬁrst is a polynomial in centered matrices for which Fact 1 applies which again bounds the probability of deviations by a factor p 1 ` (cid:15) q by c 1 exp p´ c 2 (cid:15) 2 n ˘ .
The second one is a sum of the type 1 X 1 X 2 X 3 modulo permutations, under which the trace is invariant.
We can thus apply Factwhich bounds the probability for deviations of size (cid:15)
x W 0 , W 2 y by c 1 exp `
[Equation]
[Equation]
At last, consider the random variable x W 1 , W 2 y .
By equation 18 can, just like x W 0 , W 2 y , be written as the sum of a polynomial in centered random matrices, and sums of variables of the type 1 X 1 X 2 X
3 modulo permutations.
We can thus apply the same argument as for x W 0 , W 2 y to bounds the probability for deviations of relative size (cid:15) by c 1 exp ` ´
[Equation]
n 1 { [Equation] We can now apply union bound on the variables } W 1 } 2 F , } W 2 } 2 F , x W 0 , W 2 y and x W 1 , W 2 y , modulu term equation 16, deviating from their respective mean by a factor (cid:15) .
The probability that either of them does is no more than c 1 exp ` ´ c 2 (cid:15) 2 n 1 { 3 ˘ by union bounds.
Now, set (cid:15) “ 0 .
If neither variable deviates by a factor of more than 0 .
01 , then equation 10 still holds if we only count terms scaling as rmrσ 4 since 0 .
99 2 ˚ 64 ě 1 .
01 2 ˚ 48 .
Thus, inequality is violated with probability at most c 1 exp
[Equation]
D. P ROVING THE I NEQUALITY T OWARDS THE P LANTED S OLUTION
Theorem 3.
Let p
U 1 , V 1 q and the planted solution p U ˚ , V ˚ q be sampled as per Assumption 1.
The loss function of equation 7 is convex on a straight line connecting p U 1 , V 1 q and p U ˚ , V ˚ q with probability ě 1 ´ c 1 exp ` ´ c 2 n 1 { 3 ˘ for constants c 1 , c 2 ą 0 .
Proof.
To prove this, we can repeat the argument used in the proof of Theorem 3.
The terms } W 2 } 2 , } W 1 } 2 and x W 1 , W 2 y will have the same mean and concentration since U 1 and U ˚ are identically distributed.
The only difference will be the term x W 0 , W 2 y , which now will have mean 2 rmnσper Fact 10.
This will only make the LHS of equation 10, so the inequality will still hold.
(cid:4)
D. P ROVING THE I NEQUALITY B ETWEEN R ANDOM P OINTS FOR U NOBSERVED D ATA
As explained in the main text, we will assume that entries are observed with independent probability p .
We formalize this in the following assumption
Assumption 2.
For any n , let ˆ1 P t 0 , 1 u n ˆ m have entries iid drawn from a Bernoulli distribution with probability p , which is constant w.r.t n .
Given any set of observations ˆ1 , we deﬁne our loss function as
[Equation] [Equation]
Theorem 4.
Let p
U 1 , V 1 q , p U 2 , V 2 q and the planted solution p U ˚ , V ˚ q be sampled as per Assumption 1.
Let the set of observations ˆ1 be sampled as per Assumption 2.
The loss function equation 11 is convex on a straight line connecting p U 1 , V 1 q and p
U 2 , V 2 q with probability ě 1 ´ c
[Equation]
Proof.
If no entries are observed, we can claim trivially that the function is convex.
If some entries are observed, then a ą 0 since ˆ X 2 ij p λ q 2 ą 0 for any p i, j q with probability 1.
The second derivate of equation 11 can be written as
[Equation]
˙
Using Lemma 1, we can assume by union bound that no entry in ˆ X , ˆ X 1 , ˆ X 2 , X is larger than O p r 2 { 3 q with probability ď c 1 n 2 exp ` ´ c 2 r 1 { 3 ˘ .
Note that this estimate gives the exp ` ´ c 2 r 1 { 3 ˘ scaling of the proof.
Let assume that the entries are bounded like this, then no entry in ˆ X 1 2 ij ` ˆ X 2 ij p ˆ X
ij
´
X ij q will have magnitude more than O p r 4 { 3 q for any λ .
Standard Hoeffding bounds (Hoeffding, 1994) states that for variables t x i u bounded in r
A
i , B
i s we have
[Equation]
Let us deﬁne
[Equation]
Let us now consider ﬁxed matrices U 1 , U 2 , U ˚ , V 1 , V 2 , V ˚ satisfying Lemma 1, but let ˆ1 remain a random variable.
We can then consider the product of any two variables out of ˆ X ij , ˆ X 1 ij , ˆ X 2 ij , X ij as ﬁxed constants in the interval r´ c 1 r 4 { 3 , c 1 r 4 { 3 s for some constant c 1 .
We note that y p λ q is the sum of c m n 2 variables, each of size at most O p
r 4 { 3 q .
Now y p λ q is a sum of independent variables and Hoeffdinger applies with p b
i ´ a i
q 2 ď
O p r 8 { 3 q for all i .
Taking t “ cr 2 { 3 gives us
[Equation]
This will hold for any λ .
We can assume that this holds for let’s say 3 evenly spaced λ by union bound.
Using union bounds, we assume the following: 1) y p λ q ą c 1 rmn for all λ , which follows from the derivation of Theorem 2 with probability
ě 1 ´ c 1 exp p´ c 2 n 1 { 3 q . 2) | ˆ X ij | ď r 2 { 3 for all i, j via Lemma 1. 3) | Y p λ q ´ E r y p λ
qs| ď cr 2 { 3 n 2 for 3 evenly spaced λ
i by the above equation.
Now, since the (cid:96) 2 p λ q is a second-degree polynomial, if it’s close to its expectation at 3 places, it must be close it its expectation everywhere.
Formally, we can deﬁne the second-degree polynomial ˆ y p λ q “ y p λ q ´ E r y p λ qs .
Now, our union bounds states that | ˆ y p
λ
i q| ă cr 2 { 3 n 2 for three
evenly spaced λ i .
Now, let the three coefﬁcients of the second-degree polynomial be described by the vector p , its values at λ i by the vector v and let the matrix A describe the values of the monomials 1 , λ, λ 2 at λ i .
We then have Ap “ v .
Now, A will have elements bounded by 1 but will be full rank, this implies that if v is bounded by cr 2 { 3 n 2 so must p be.
The fact that the coefﬁcents in ˆ y have absolute value bounded by O p r 2 { 3 n 2 q , together with the fact that the mean scales as rn 2 gives us the following
[Equation]
Here c is positive as per our assumptions, whereas c 1 might be negative.
Now, for sufﬁciently large r , the above quality will be non-negative.
We can thus ﬁnd new constants, based upon c and c 1 , to complete the proof.
(cid:4)
D. C ONCENTRATION
This section contains the concentration results needed for the proof of Theorem 1.
We will use results from random matrix theory, which has stronger results for matrices with independent entries that are concentrated (Ahlswede and Winter, 2002; Tropp, 2012; Guionnet et al., 2000).
We will use the concentration results of Meckes and Szarek (2012) which relies on the notation of CCP – convex concentration property – which is a regularity condition.
Deﬁnition 2.
A random matrix X in a normed vector space satisﬁes CCP iff there exists positive constants c 1 , c 2 such that P
[Equation]
and all convex 1-Lipschitz functions f .
Here M is the median.
One can easily verify that independent Gaussian variables satisﬁes this (Ledoux, 2001), and it then follows that 1-Lipschitz functions of Gaussians also satisﬁes CCP.
One important class here is the half-normal distribution which is just the absolute value of a Gaussuan variable, and absolute value is a 1-Lipschitz function.
Any vector of independent bounded variables will also satisfy CCP (Meckes and Szarek, 2012).
Fact 1.
Let ˆ P be a polynomial of degree 4 in centered matrices
X i , X j , X k , X l otherwise sampled as per Assumption 1, where we allow i “ j , j “ l and any other arbitrary index relationship.
Let z “ Tr ˆ P p X i , X j , X k , X l q and let µ be then mean of z , then P ` | z ´ µ | ą t r n˘ ď c
[Equation]
` ´ c 2 min p t 2 , t 1 { 2 q n ˘ for positive constants c 1 , c 2 .
Proof.
Any vector CCP where some components are zero will still be CCP, this we can pad the matrices
X i , X j , X k , X l so that they are square.
We then invoke Theorem 1 of Meckes and Szarek (2012) which gives
[Equation]
Setting τ “ tr gives us
[Equation]
Using our assumption r “ O p n γ q for some γ P r 1 { 2 , 1 s gives
[Equation]
Fact 2.
Let z “ Tr ` 1 X i X j X k ˘ for centered matrices
X i , X j , X k otherwise sampled as per Assumption 1, and let µ be the mean of z . 1 is a matrix of all 1 s of the appropriate shape.
Then P
[Equation]
Proof.
As before, we can pad the matrices to be square while retaining the CCP property.
We note that the singular value of the matrix 1 is at most n .
This means that the shatten-norm } E 1 } k is at most n for all k .
Let us consider the matrix ˆ 1 “ 1 n 1 { 3 which then has shatten-norm } E ˆ 1 } d is at most n 2 { 3 .
For this matrix, Theorem 1 in Meckes and Szarek (2012) applies, so that we have
[Equation]
Let us take t “ t 0 rn ´ 1 { 3 so that tn 7 { 3 “ t 0 rn 2 .
We then have
[Equation]
By substituting t “ t 0 rn ´ 1 { 3 into equation 12 we then have
[Equation]
Recall our assumption r “ c 3 n γ for γ P r 1 { 2 , 1 s .
We then have min
[Equation]
[Equation]
[Equation]
Plugging this into equation 13 completes the proof (cid:4)
Lemma 1.
With probability ď c 1 n 2 exp [Equation] 1 { 3 ˘ , no entry in ˆ X ij , ˆ X 1 ij , ˆ X 2 ij , X ij has an absolute value larger than cr 2 { 3 , for some positive constant c .
Proof.
For a ﬁxed i, j consider the terms ˆ X ij , ˆ X 1 ij , ˆ X 2 ij , X ij .
Fact 9 says that each one can be expressed as a constant number of variables a with zero mean of the form
[Equation]
Since the variables themselves are sub-gaussian, by Lemma 2.7.7 in Vershynin (2018)
the product v p 1 q i v p 2 q
i is sub-exponential.
Thus, the variable a is a sum of iid sub-exponential variables, and
Theorem 2.8.1 in Vershynin (2018) states that
[Equation]
Here K is the orlicz norm } ¨ } Ψ 1 which is a constant for our ﬁxed distributions.
In the above expression, let us set t “ cr 2 { 3 .
This gives
[Equation]
We note that there is a polynomial number of entries a , so we can do a polynomial number (w.r.t. n ) of union bounds, which might change the constants, to show that no entry is further than c 1 r 2 {from its expectation for some positive c 1 .
(cid:4)
E A LGEBRAIC CALCULATIONS E. B ASIC FACTS
[Equation]
Proof.
We have
[Equation]
 Taking the mean, and using linearity of expectations, we only have nonzero mean if i “ k .
We get
[Equation]

Using linearity of expectation and independence, this becomes
[Equation]
For a single matrix entry, E “ ¯ U ji ¯ U ji ‰
[Equation]
σ 2 . i goes over the columns of U of which there are r , j goes over the rows of ¯ U of which there are n and l goes over the columns of ¯ V of which there are m , so we clearly get σ 4 rmn.
(cid:4)
Fact 4.
E
[Equation]
Proof.
We can can ﬁrst contract 1 r ˆ m
Tr ˆ m “ 1 mµ 2 var , where 1 is a r -by- r matrix of all ones.
We then want to ﬁnd
[Equation]
Now since the variables are centered, the expectations becomes zero unless i “ k .
The sum thus becomes
[Equation]
This gives us the result of rmnµ 2 var σ 2 .
(cid:4)
E.
E XPECTATION
Fact 5.
E r} W 2 } 2 s “ 4 mnrσ
Proof.
Let us rewrite E r} W 2 } 2 s as
[Equation]
˙
Trace and expectations are linear operators, so we can reorder them.
Let us consider the matrix
[Equation]
We can easily add and subtract the mean µ of all matrices which gives
[Equation]
Expanding the parantheses gives us 16 matrices, however they will have zero mean unless both the
U -matrices and V -matrices ar the same.
For calculating the mean, we can thus only consider four matrices of the type V T U T UV .
Permuting the indices cyclically and appealing to Fact 3 gives us
[Equation]
Fact 6.
E rx W 0 , W 2 ys “ rmnσ
Proof.
We can write E rx W 0 , W 2 ys as
[Equation]
˘˙
Again, we want to add and subtract the matrix means 1 n ˆ r , 1 r ˆ m to get
E Tr
[Equation]
As before, we can remove terms which are linear in any centered variable.
This removes matrices with index 1 , 3 and leaves
[Equation]
˙
Applying fact 3 to the only term that remain gives
[Equation]
Fact 7.
E } W 1 } 2 F “ 6 rmnσ 4 ` 4 rmnµ 2 var σ
Proof.
We rewrite E } W 1 } 2 F as
[Equation]
We will again want to center the variables
[Equation]
which gives
[Equation]
˘˙
Let us ﬁrst consider the terms involving constants.
We remove any terms linear in centered variables.
We are then left with
[Equation]
As before, any expressions linear in centered variables vanish when we take expectations.
Using this and Fact 4 we get that the above expression is equal to
[Equation]
We now return to the terms without constants These are
[Equation]
As before, we remove any expressions linear in centered variables.
As it turns out, we are left with 6 terms of the type ¯ V 2 T ¯ U 2 T
¯
U
¯ V 2 .
Applying Fact 3 gives us
[Equation]
Fact 8.
E x W 2 , W 1 y “ ´ 4 rmnσ
Proof.
Let us rewrite E x W 2 , W 1 y as
[Equation]
As per usual, we center the variables
[Equation]
Any terms involving the constant will be linear in a centered variable, and thus disappear under expectations.
Equating like terms and using Fact 3, we get
[Equation]
E. O THER ALGEBRAIC FACTS
Fact 9.
For any ﬁxed i, j the entries of ˆ X , ˆ X 1 and ˆ X 2 are zero mean.
Proof.
We have
[Equation]
The fact that U 1 and U 2 are iid implies that p U 1 ´ U 2 q has zero mean, and the similar holds for
V 1 and V 2 .
This implies that all terms of ˆ X 1 and ˆ X 2 are zero-mean.
Using the fact that U 1 , Uand U ˚ are iid implies that
[Equation]
This, in turn, implies that E r
ˆ X s “ 0 .
(cid:4)
Fact 10.
For U 1 “ U ˚ and V 1 “ V ˚ , we have E rx W 0 , W 2 ys “ 2 rmnσ
Proof.
We can write E rx W 0 , W 2 ys as
[Equation]
˘˙
Again, we want to add and subtract the matrix means 1 n ˆ r , 1 r ˆ m to get
E Tr
[Equation]
˘˙
As before, we can remove terms which are linear in any centered variable.
The only term that remain gives
[Equation]