Subspace Recovery from Heterogeneous Data with Non-isotropic Noise
Abstract
Recovering linear subspaces from data is a fundamental and important task in statistics and machine learning.
Motivated by heterogeneity in Federated Learning settings, we study a basic formulation of this problem: the principal component analysis (PCA), with a focus on dealing with irregular noise.
Our data come from n users with user i contributing data samples from a d -dimensional distribution with mean µ i .
Our goal is to recover the linear subspace shared by µ 1 , . . . , µ n using the data points from all users, where every data point from user i is formed by adding an independent mean-zero noise vector to µ i .
If we only have one data point from every user, subspace recovery is information-theoretically impossible when the covariance matrices of the noise vectors can be non-spherical, necessitating additional restrictive assumptions in previous work.
We avoid these assumptions by leveraging at least two data points from each user, which allows us to design an efficiently-computable estimator under non-spherical and user-dependent noise.
We prove an upper bound for the estimation error of our estimator in general scenarios where the number of data points and amount of noise can vary across users, and prove an information-theoretic error lower bound that not only matches the upper bound up to a constant factor, but also holds even for spherical Gaussian noise.
This implies that our estimator does not introduce additional estimation error (up to a constant factor) due to irregularity in the noise.
We show additional results for a linear regression problem in a similar setup.
Introduction
We study the problem of learning low-dimensional structure amongst data distributions, given multiple samples from each distribution.
This problem arises naturally in settings such as federated learning, where we want to learn from data coming from a set of individuals, each of which has samples from their own distribution.
These distributions however are related to each other and in this work, we consider the setting when these distributions have means lying in a low-dimensional subspace.
The goal is to learn this subspace, even when the distributions may have different (and potentially non-spherical) variances.
This heterogeneity can manifest itself in practice as differing number of samples per user, or the variance differing across individuals, possibly depending on their mean.
Recovery of the subspace containing the means can in turn help better estimate individual means.
In other words, this can allow for learning good estimator for all individual means, by leveraging information from all the individuals.
The irregularity of the noise makes this task challenging even when we have sufficiently many individual distributions.
For example, suppose we have n individuals and for every i = 1 , . . .
, n , an unknown µ i ∈ R d .
For simplicity, suppose that µ 1 , . . . , µ n are distributed independently as
N (0 , σ 2 uu T ) for σ ∈ R ≥ 0 and an unknown unit vector u ∈ R d .
In this setting, our goal is to recover the one-dimensional subspace, equivalently the vector u .
For every i , we have a data point x i = µ i
+ z i where z i ∈ R
d is a mean-zero noise vector.
If z i is drawn independently from a spherical
Gaussian N (0 , α 2 I ) , we can recover the unknown subspace with arbitrary accuracy as n grows to infinity because 1 n (cid:80) x
i x T i concentrates to E
[ x i x T i ]
= σ 2 uu T + α 2 I , whose top eigenvector is ± u .
However, if the noise z i is drawn from a non-spherical distribution, the top eigenvector of 1 n (cid:80) x i x T i can deviate from ± u significantly, and to make things worse, if the noise z i is drawn independently from a non-spherical Gaussian N (0 , σ 2 ( I − uu T )
+ α 2 I ) , then our data points x i = µ
i
+ z i distribute independently as N (0 , ( σ 2 + α 2 ) I ) , giving no information about the vector u .
The information-theoretic impossibility in this example however disappears as soon as one has at least two samples from each distribution.
Indeed, given two data points x i 1 = µ i + z i
and x i
= µ i + z ifrom user
i , as long as the noise z i 1 , z i 2 are independent and have zero mean, we always have E
[ x i 1 x T
i 2 ]
= σ 2 uu T regardless of the specific distributions of z i 1 and z i 2 .
This allows us to recover the subspace in this example, as long as we have sufficiently many users each contributing at least two examples.
As this is commonly the case in our motivating examples, we make this assumption of multiple data points per user, and show that this intuition extends well beyond this particular example.
We design efficiently computable estimators for this subspace recovery problem given samples from multiple heteroscedastic distributions (see Section 1.1 for details).
We prove upper bounds on the error of our estimator measured in the maximum principal angle (see Section 2 for definition).
We also prove an information-theoretic error lower bound, showing that our estimator achieves the optimal error up to a constant factor in general scenarios where the number of data points and the amount of noise can vary across users.
Somewhat surprisingly, our lower bound holds even when the noise distributes as spherical Gaussians.
Thus non-spherical noise in setting does not lead to increased error.
We then show that our techniques extend beyond the mean estimation problem to a linear regression setting where for each µ i , we get (at least two) samples ( x ij , x T ij µ i + z ij ) where z ij is zero-mean noise from some noise distribution that depends on i and x ij .
This turns out to be a model that was recently studied in the meta-learning literature under more restrictive assumptions (e.g. z ij is independent of x ij )
[Kong et al., 2020, Tripuraneni et al., 2021, Collins et al., 2021, Thekumparampil et al., 2021].
We show a simple estimator achieving an error upper bound matching the ones in prior work without making these restrictive assumptions.
1.1 Our contributions
PCA with heterogeneous and non-isotropic noise: Upper Bounds.
In the PCA setting, the data points from each user i are drawn from a user-specific distribution with mean µ i ∈ R d , and we assume that µ 1 , . . . , µ n lie in a shared k -dimensional subspace that we want to recover.
Specifically, we have m i data points x ij ∈ R d from user i for j = 1 , . . .
, m i , and each data point is determined by x ij = µ i + z ij where
z ij ∈ R
d is a noise vector drawn independently from a mean zero distribution.
We allow the distribution of z ij to be non-spherical and non-identical across different pairs ( i, j ) .
We use η i ∈ R ≥ 0 to quantify the amount of noise in user i ’s data points by assuming that z ij is an η i -sub-Gaussian random variable.
As mentioned earlier, if we only have a single data point from each user, it is information-theoretically impossible to recover the subspace.
Thus, we focus on the case where m i ≥ 2 for every i = 1 , . . . , n .
In this setting, for appropriate weights w 1 , . . .
, w n ∈ R ≥ 0 , we compute a matrix A :
[Equation]
where the inner summation is over all pairs j 1 , j 2 ∈ { 1 , . .
, m i } satisfying j 1 ̸ =
j 2 .
Our estimator is then defined by the subspace spanned by the top- k eigenvectors of A .
Although the inner summation is over m i ( m i − 1) terms, the time complexity for computing it need not grow quadratically with m i because of the following equation:
[Equation]
The flexibility in the weights w 1 , . . .
, w n allows us to deal with variations in m i and η i for different users i .
In the special case where η 1 = · · · = η n = η and m 1 = · · · = m n = m , we choose w 1 = · · · = w n = 1 /n
and we show that our estimator achieves the following error upper bound with success probability at least 1 − δ :
[Equation]
Here, θ is the maximum principal angle between our estimator and the true subspace shared by µ 1 , . . . , µ n , and we define σ ℓ ≥ 0 such that σ 2 ℓ is the ℓ -th largest eigenvalue of 1 n (cid:80) ni
=
µ i µ T i .
Our error upper bound for general m i , η i , w i is given in Theorem 3.1.
We instantiate our error upper bound to the case where µ 1 , . . . , µ n are drawn iid from a Gaussian distribution N (0 , σ 2 UU T ) , where the columns of U ∈ R d × k form an orthonormal basis of the subspace containing µ 1 , . . . , µ n .
By choosing the weights w 1 , . . .
, w n according to m 1 , . . .
, m n and η 1 , . .
η n , our estimator achieves the error upper bound
[Equation]
under a mild assumption (Assumption 3.2), where γ ′ i is defined in Definition 3.1 and often equals (cid:16)
[Equation]
PCA: Lower Bounds.
We show that the error upper bound (2) is optimal up to a constant factor by proving a matching information-theoretic lower bound (Theorem 3.7).
Our lower bound holds for general m i and η i that can vary among users i , and it holds even when the noise vectors z ij are drawn from spherical Gaussians, showing that our estimator essentially pays no additional cost in error or sample complexity due to non-isotropic noise.
We prove the lower bound using Fano’s method on a local packing over the Grassmannian manifold.
We carefully select a non-trivial hard distribution so that the strength of our lower bound is not affected by a group of fewer than k users each having a huge amount of data points with little noise.
Linear Models.
While the PCA setting is the main focus of our paper, we extend our research to a related linear models setting that has recently been well studied in the meta-learning and federated learning literature [Kong et al., 2020, Tripuraneni et al., 2021, Collins et al., 2021, Thekumparampil et al., 2021].
Here, the user-specific distribution of each user i is parameterized by β i ∈ R d , and we again assume that β 1 , . . .
, β n lie in a linear subspace that we want to recover.
From each user i we observe m i data points ( x ij , y ij )
∈ R
d × R for j = 1 , . . .
, m i drawn from the user-specific distribution satisfying y ij = x
T ij β
i + z ij for an O (1) -sub-Gaussian measurement vector x ij ∈ R d with zero mean and identity covariance and an η i -sub-Gaussian mean-zero noise term z ij ∈ R .
While it may seem that non-isotropic noise is less of a challenge in this setting since each noise term z ij is a scalar, our goal is to handle a challenging scenario where the variances of the noise terms z ij can depend on the realized measurements x ij , which is a more general and widely applicable setting compared to those in prior work.
Similarly to the PCA setting, our relaxed assumptions on the noise make it information-theoretically impossible to do subspace recovery if we only have one data point from each user (see Section 4), and thus we assume each user contributes at least two data points.
For appropriate weights w 1 , . . . , w n ∈ R ≥ 0 , we use the subspace spanned by the top- k eigenvectors of the following matrix A as our estimator:
[Equation]
In the special case where η 1 = · · · = η n = η, m 1 = · · · = m n = m , and ∥ β i ∥ 2 ≤ r for all i , our estimator achieves the following error upper bound:
[Equation]
where σ 2 k is the k -th largest eigenvalue of 1 n (cid:80)
ni
=1 β i β T i (Corollary L.2).
Our error upper bound extends smoothly to more general cases where η i and m i vary among users (Theorem L.1).
Moreover, our upper bound matches the ones in prior work [e.g. Tripuraneni et al., 2021, Theorem 3] despite requiring less restrictive assumptions.
1.2 Related Work
Principal component analysis under non-isotropic noise has been studied by Vaswani and Narayanamurthy [2017], Zhang et al.
[2018] and Narayanamurthy and Vaswani [2020].
When translated to our setting, these papers focus on having only one data point from each user and thus they require additional assumptions—either the level of non-isotropy is low, or the noise is coordinate-wise independent and the subspace is incoherent.
The estimation error guarantees in these papers depend crucially on how well these additional assumptions are satisfied.
Zhu et al.
[2019] and Cai et al.
[2021] study PCA with noise and missing data, and Chen et al.
[2021] and Cheng et al.
[2021] study eigenvalue and eigenvector estimation under heteroscedastic noise.
These four papers all assume that the noise is coordinate-wise independent and the subspace/eigenspace is incoherent.
The linear models setting we consider has recently been studied as a basic setting of meta-learning and federated learning by Kong et al.
[2020], Tripuraneni et al.
[2021], Collins et al.
[2021], and
Thekumparampil et al.
These papers all make the assumption that the noise terms z ij are independent of the measurements x ij , an assumption that we relax in this paper.
Collins et al.
[2021] and Thekumparampil et al.
[2021] make improvements in sample complexity and error guarantees compared to earlier work by Kong et al.
[2020] and Tripuraneni et al.
[2021], but Collins et al.
[2021] focus on the noiseless setting ( z ij = 0 ) and Thekumparampil et al.
[2021] require at least Ω( k 2 ) examples per user.
Tripuraneni et al.
[2021] and Thekumparampil et al.
[2021] assume that the measurements x ij are drawn from the standard (multivariate) Gaussian distribution, where as
Kong et al.
[2020], Collins et al.
and our work make the relaxed assumption that x ij are sub-Gaussian with identity covariance, which, in particular, allows the fourth-order moments of x ij to be non-isotropic.
There is a large body of prior work on meta-learning beyond the linear setting [see e.g. Maurer et al., 2016, Tripuraneni et al., 2020, Du et al., 2020].
When collecting data from users, it is often important to ensure that private information about users is not revealed through the release of the learned estimator.
Many recent works proposed and analyzed estimators that achieve user-level differential privacy in settings including mean estimation [Levy et al., 2021, Esfandiari et al., 2021], meta-learning [Jain et al., 2021] and PAC learning [Ghazi et al., 2021].
Recently, Cummings et al.
[2021] study one-dimensional mean estimation in a setting similar to ours, under a differential privacy constraint.
The matrix A we define in (1) is a weighted sum of A i :=
[Equation]
users i = 1 , . .
. , n , and each A i has the form of a U -statistic
[Halmos, 1946, Hoeffding, 1948].
U -statistics have been applied to many statistical tasks including tensor completion [Xia and Yuan, 2019] and various testing problems [Zhong and Chen, 2011, He et al., 2021, Schrab et al., 2022].
In our definition of A i , we do not make the assumption that the distributions of x i 1 , . . .
, x im i are identical although the assumption is commonly used in applications of U -statistics.
The matrix A in (3) is also a weighted sum of U -statistics where we again do not make the assumption of identical distribution.
1.3 Paper Organization
In Section 2, we formally define the maximum principal angle and other notions we use throughout the paper.
Our results in the PCA setting and the linear models setting are presented in Sectionsand 4, respectively.
We defer most technical proofs to the appendices.
Preliminaries
We use ∥ A ∥ to denote the spectral norm of a matrix A , and use ∥ u ∥ 2 to denote the ℓ 2 norm of a vector u .
For positive integers k ≤ d , we use O d,k to denote the set of matrices A ∈ R d × k satisfying
A T A = I k , where I k is the k × k identity matrix.
We use O d to denote O d,d , which is the set of d × d orthogonal matrices.
We use col ( A ) to denote the linear subspace spanned by the columns of a matrix A .
We use the base- e logarithm throughout the paper.
Maximum Principal Angle.
Let U, ˆ U ∈
O d be two orthogonal matrices.
Suppose the columns of U and ˆ U are partitioned as U
=
[ U 1 U 2 ] , ˆ U =
[ ˆ U 1 ˆ U 2 ] where U 1 , ˆ U 1 ∈
O d,k for an integer k satisfying 0 < k < d .
Let Γ (resp. ˆΓ ) be the k -dimensional linear subspace spanned by the columns of U 1 (resp.
ˆ U 1 ).
Originating from [Jordan, 1875], the maximum principal angle θ ∈ [0 , π/ 2] between Γ and ˆΓ , denoted by ∠ (Γ , ˆΓ) or ∠ ( U 1 , ˆ U 1 ) , is defined by sin θ = ∥
U 1 U T 1 − ˆ U
ˆ U T 1 ∥ =
∥ U T 1 ˆ U 2 ∥ =
∥ U T 2 ˆ U 1 ∥ .
It is not hard to see that the maximum principal angle depend only on the subspaces Γ , ˆΓ and not on the choices of U and ˆ U , and sin ∠ (Γ , ˆΓ) is a natural metric between k -dimensional subspaces (see Appendix A for more details where we discuss the definition of principal angles for any two subspaces with possibly different dimensions).
With the definition of the maximum principal angle, we can now state a variant of the Davis–Kahan sin θ theorem [Davis and Kahan, 1970] that will be useful in our analysis (see Appendix E for proof):
Theorem 2.1 (Variant of Davis–Kahan sin θ theorem) .
Let A, ˆ A ∈ R d
× d be symmetric matrices.
Let λ i denote the i -th largest eigenvalue of A .
For a positive integer k smaller than d , let θ denote the maximum principal angle between the subspaces spanned by the top- k eigenvectors of A and ˆ A .
Assuming λ k > λ k +1 , sin θ ≤ 2 ∥ A − ˆ A ∥
λ k − λ k +1 .
Sub-Gaussian and sub-exponential distributions.
We say a random variable x ∈ R with expectation E [ x ] ∈ R has sub-Gaussian constant b ∈ R ≥ 0 if E [ | x − E [ x ] | p ] 1 /p ≤ b √ p for every p ≥ 1 .
We say x has sub-exponential constant b ∈ R ≥
if E [ | x − E [ x ] | p ] 1 /p ≤ bp for every p ≥ 1 .
We say a random vector y ∈ R
d has sub-Gaussian (resp.
sub-exponential) constant b ∈ R ≥
if for every unit vector u ∈ R d (i.e., ∥ u ∥ 2 = 1 ), the random variable u T y ∈ R has sub-Gaussian (resp.
sub-exponential) constant b .
We say y is b -sub-Gaussian (resp.
b -sub-exponential ) if it has sub-Gaussian (resp.
sub-exponential) constant b .
Principal Component Analysis
In the principal component analysis (PCA) setting, our goal is to recover the k -dimensional subspace Γ spanned by the user-specific means µ 1 , . . . , µ n ∈ R d of the n users.
From each user i , we have m i ≥ 2 data points
[Equation]
We assume the noise z ij ∈ R d is drawn independently from a mean zero distribution with subGaussian constant η i .
We do not assume that the variance of z ij is the same along every direction, nor do we assume that the distribution of z ij is the same for different ( i, j ) .
We first show an error upper bound for our estimator when the user-specific means µ 1 , . . . , µ n are deterministic vectors (Section 3.1) and then apply this result to the case where µ 1 , . . . , µ n are drawn from a sub-Gaussian distribution (Section 3.2).
In Section 3.3 we prove an information-theoretic error lower bound matching our upper bound.
3.1 Fixed User-Specific Means
We first focus on the case where µ 1 , . .
µ n are deterministic vectors.
In this case, all the randomness in the data comes from the noise z ij .
Our estimator is the subspace ˆΓ spanned by the top- k eigenvectors of A defined in (1).
For ℓ = 1 , . . .
, d , we define σ ℓ ≥
such that σ 2 ℓ is the ℓ -th largest eigenvalue of (cid:80) ni =1 w i µ i µ T i .
Since µ 1 , . . . , µ n share a k -dimensional subspace, σ ℓ = 0 for ℓ > k .
We prove the following general theorem on the error guarantee of our estimator:
Theorem 3.1.
Define ξ 2 = ∥ (cid:80) ni
=1 w 2 i µ i µ T i η 2 i /m i ∥ and let θ denote the maximum principal angle between our estimator ˆΓ and the true subspace Γ spanned by µ 1 , . . . , µ n .
For any δ ∈ (0 , 1 / 2) , with probability at least 1 − δ ,
[Equation]
We can simplify the bound in Theorem 3.1 by considering special cases:
Corollary 3.2.
Assume max { η 1 / √ m 1 , . . .
, η n / √ m n } = t
and we choose w 1 = · · · = w n = 1 /n .
For any δ ∈ (0 , 1 / 2) , with probability at least 1 − δ ,
[Equation]
In particular, when η 1 = · · · = η n = η , and m 1 = · · · = m n = m , error bound (7) becomes
[Equation]
We defer the complete proof of Theorem 3.1 and Corollary 3.2 to Appendices F and G. Our proof is based on the Davis-Kahan sin θ theorem (Theorem 2.1).
Since σ 2 k +
=
0 , Theorem 2.1 implies
[Equation]
This reduces our goal to proving an upper bound on the spectral norm of A − (cid:80) ni
=
w i µ i µ T i .
Since for distinct j 1 and j 2 in { 1 , . . .
, m i } we have E [ x ij 1 x T ij 2 ]
= µ i µ T i , our construction of A in (1) guarantees E [ A ] = (cid:80) ni
=
w i µ i µ T i .
Therefore, our goal becomes controlling the deviation of A from its expectation, and we achieve this goal using techniques for matrix concentration inequalities.
3.2 Sub-Gaussian User-Specific Means
We apply our error upper bound in Theorem 3.1 to the case where µ 1 , . . . , µ n ∈ R d are drawn iid from N (0 , σ 2 UU T ) for an unknown U ∈
O d,k .
We still assume that each data point x ij ∈ R d is generated by adding a noise vector z
ij ∈ R d to the user-specific mean µ i as in (5).
We do not assume that the noise vectors ( z ij ) 1 ≤ i ≤ n, 1 ≤ j ≤ m i are independent of the user-specific means ( µ i ) 1 ≤ i ≤ n , but we assume that when conditioned on ( µ i )≤ i ≤ n , every noise vector z ij independently follows a distribution with mean zero and sub-Gaussian constant η i .
We use the same estimator ˆΓ as before: ˆΓ is the subspace spanned by the top- k eigenvectors of A defined in (1).
We determine the optimal weights w 1 , . . .
, w n in (1) as long as m 1 , . . .
, m n and η 1 , . .
, η n satisfy a mild assumption (Assumption 3.2), achieving an error upper bound in Theorem 3.4.
In the next subsection, we prove an error lower bound (Theorem 3.7) that matches our upper bound (Theorem 3.4) up to a constant factor, assuming d ≥ (1 + Ω(1))
k and δ = Θ(1) .
We prove our error upper bound in a slightly more general setting than µ 1 , . . . , µ n drawn iid from
N (0 , σ 2 UU T ) .
Specifically, we make the following assumption on the distribution of µ 1 , . . . , µ n :
Assumption 3.1.
The user-specific means µ 1 , . . . , µ n ∈ R d are mean-zero independent random vectors supported on an unknown k -dimensional subspace Γ .
Moreover, for a parameter σ > 0 , for every i = 1 , . . . , n , µ i has sub-Gaussian constant O ( σ ) , and the k -th largest eigenvalue of E [ µ i µ T i ] is at least σ 2 .
Under this assumption, we have the following lower bound on the σ 2 k in Theorem 3.1 (see Appendix H for proof):
Claim 3.3.
Under Assumption 3.1, let w 1 , . . .
, w n ∈ R ≥ 0 be user weights satisfying w 1 + · · · + w n = 1 and σ 2 k be the k -th largest eigenvalue of (cid:80) ni
=
w i µ i µ T i .
There exists an absolute constant C ∗ >such that for any δ ∈ (0 , 1 / 2) , as long as max 1 ≤ i ≤ n w i ≤ 1 /C ∗ ( k + log(1 /δ )) , then σ 2 k ≥ σ 2 / 2 with probability at least 1 − δ/ 2 .
The following definition is important for us to choose the weights w 1 , . . .
, w n in (1) optimally:
Definition 3.1.
Define γ i = (cid:16) η
i
σ 2 m i + η 4 i σ 4 m
i (cid:17) − 1 and assume w.l.o.g. that γ 1 ≥ · · · ≥ γ n .
Define γ ′ i
= γ i if i ≥ k , and γ ′ i = γ k if i < k .
Intuitively, we can view γ i as measuring the “amount of information” provided by the data points from user i .
This is consistent with the fact that γ i increases as the number m i of data points from user i increases, and γ i decreases as the noise magnitude η i from user i increases.
With the users sorted so that γ 1 ≥ · · · ≥ γ n , the quantity γ ′ i is then defined to be γ k for the k most “informative” users i = 1 , . .
k , and γ ′ i
= γ i for other users.
We make the following mild assumption on γ ′ i under which we achieve optimal estimation error:
Assumption
[Equation] [Equation]
for C ∗ defined in Claim 3.3.
By the definition of γ ′ i , it is easy to show that Assumption 3.2 is equivalent to (cid:80) n i =
k +1 γ i ≥
( C ∗ − 1)
k + C ∗ log(1 /δ ))
γ k .
Therefore, if we view γ i as the “amount of information” from user i ,
Assumption 3.2 intuitively requires that a significant contribution to the total “information” comes from outside the k most “informative” users.
This assumption allows us to avoid the case where we only have exactly n = k users: in that case, we would have σ 2 k ≈ σ 2 /k 2 for uniform weights w 1 = · · · = w n (see [Rudelson and Vershynin, 2008] and references therein), as opposed to the desired σ 2 k ≥ σ 2 / 2 in Claim 3.3.
Assumption 3.2 is a mild assumption.
For example, when γ k = · · · = γ n , Assumption 3.2 holds as long as n ≥ C ∗ ( k +log(1 /δ )) .
Also, since γ ′ 1 = · · · = γ ′ k ≥ γ ′ k +1 ≥ · · · ≥ γ ′ n ≥ 0 , it trivially holds that (cid:80) ni
=1 γ ′
i ≥ kγ ′ 1 .
Assumption 3.2 is relatively mild when compared to this trivial inequality.
Under Assumption 3.2, we show that it is optimal to choose the weights w 1 , . . .
, w n as
[Equation]
Specifically, if we plug (9) into Theorem 3.1 and bound ξ and σ k based on the distribution of µ 1 , . . . , µ n , we get the following error upper bound which matches our lower bound (Theorem 3.7) in Section 3.3.
We defer its proof to Appendix I.
Theorem 3.4.
Under Assumptions 3.1 and 3.2, if we choose w 1 , . . .
, w n as in (9) and define θ = ∠ (Γ , ˆΓ) , for δ ∈ (0 , 1 / 2) , with probability at least 1 − δ ,
[Equation]
For comparison, consider the setting when σ = η i = 1 for every i = 1 , . .
. , n .
The result then says that sin θ is bounded by approximately (cid:113) d (cid:80) ni
=
m i .
This is the same rate as we would get if we have (cid:80) ni
=
m i users each contributing a single independent data point with homogeneous spherical noise.
Thus as long as the data points are not too concentrated on fewer than k users, the heterogeneity comes at no additional cost.
3.3 Lower Bound
We prove a lower bound matching the upper bound in Theorem 3.4 up to constant in the setting where δ = Θ(1) , d ≥ (1 + Ω(1)) k .
For every positive integer d , there is a natural “uniform” distribution over O d given by Haar’s theorem [Haar, 1933]
(see e.g. [Diestel and Spalsbury, 2014] for a textbook).
We denote this distribution by
Haar ( O d ) .
A random matrix A drawn from Haar ( O d ) has the following invariance property: for any deterministic matrix B ∈ O d , the random matrices A, AB and BA all have the same distribution.
For an integer k ≤ d , we can construct a random matrix
A 1 ∈ O d,k by first drawing A ∈ R d × d from
Haar ( O d ) and then take the first k columns of A .
We denote the distribution of A 1 by Haar ( O d,k ) .
The invariance property of Haar ( O d ) immediately implies the following claims:
Claim 3.5.
Let A ∈ O d be a random matrix drawn from Haar ( O d ) and let B ∈ O d,k be a fixed matrix.
Then AB distributes as Haar ( O d,k ) .
Proof.
The matrix B can be written as the first k columns of a matrix C ∈ O d .
Now AB is the first k columns of AC , where AC distributes as Haar ( O d ) by the invariance property.
This implies that
AB distributes as Haar ( O d,k ) .
Claim 3.6.
Let B ∈ O d,k be a random matrix.
Assume for every fixed matrix A ∈ O d , the random matrices B and AB have the same distribution.
Then B ∼ Haar ( O d,k ) .
Proof.
If we draw A independently from Haar ( O d ) , the random matrices B and AB still have the same distribution.
By Claim 3.5, AB distributes as Haar ( O d,k ) , so B must also distribute as
Haar ( O d,k ) .
With the definition of Haar ( O d,k ) , we state our lower bound in the following theorem:
Theorem 3.7.
Let k, d, n be positive integers satisfying k < d and k ≤ n .
Let m 1 , . . .
, m n be positive integers and σ, η 1 , . . .
, η n be positive real numbers.
Suppose we draw U ∈
O d,k from Haar ( O d,k ) and then draw µ 1 , . .
. , µ n independently from N (0 , σ 2 UU T ) .
For every i = 1 , . . . , n , we draw m i data points x ij for j = 1 , . . .
, m i as x ij =
µ
i + z ij , where each z ij is drawn independently from the spherical Gaussian N (0 , η 2 i I ) .
Let ˆΓ be any estimator mapping ( x ij ) 1 ≤ i ≤ n, 1 ≤ j ≤ m i to a (possibly randomized) k -dimensional subspace of R d .
Let θ denote the maximum principal angle between ˆΓ(( x ij ) 1 ≤ i ≤ n, 1 ≤ j ≤ m i ) and the true subspace Γ = col ( U ) .
If real numbers t ≥ 0 and δ ∈ [0 , 1 / 2) satisfy Pr[sin θ ≤ t ] ≥ 1 − δ, then
[Equation]
where γ 1 , . . . , γ n are defined in Definition 3.1.
Note that γ ′ i
= γ i for i ≥ k , so our upper bound in (10) matches the lower bound (11) up to a constant factor assuming δ = Θ(1) and d ≥ (1 + Ω(1)) k .
We use the local Fano method to prove the lower bound using the technical lemmas in Appendix D.
In particular, we reduce our goal to proving an upper bound on the KL divergence between Gaussian distributions whose covariance matrices are defined based on matrices U, ˆ
U ∈
O d,k with ∥ UU T − ˆ
U ˆ U T ∥
F bounded.
We prove the following lemma in Appendix J that upper bounds the KL divergence using ∥ UU T − ˆ
U ˆ U T ∥ F :
Lemma 3.8.
For σ ∈ R ≥ 0 , η ∈ R > 0 , U, ˆ U ∈
O d,k , define Σ = σ 2 UU T + η 2 I and ˆΣ =
σ 2 ˆ U
ˆ U T + η
I .
Then,
[Equation]
Lemma 3.8 and the results in Appendix D allow us to prove a version of (11) in which the sum in the demoninator is over i = 1 , . . . , n .
This, however, is weaker and less useful than (11) in which the sum in the denominator is over i =
k, k + 1 , . .
. , n .
To prove Theorem 3.7, we extract a hard distribution in which the data points from users 1 , . .
, k − 1 are “useless” in terms of subspace recovery.
Let Γ 1 be the ( k − 1) -dimensional subspace spanned by µ 1 , . . .
, µ k − 1 .
We let v 1 , . .
, v k − 1 be a random orthonormal basis of Γ 1 , and we append another vector v k ∈ Γ to form an orthonormal basis v 1 , . . .
, v k of Γ .
We define V 1 =
[ v 1 · · · v k − 1 ] ∈ O d,k − 1 and V =
[ v 1 · · · v k ] ∈ O d,k .
In Figure 1 we show a graphical model demonstrating the dependency among the random objects we defined.
Let us focus on the joint distribution of ( V 1 , V, ( µ 1 , . . .
, µ k − 1 )) .
By the invariance property, for any matrices ˜ V 1 ∈ O d,k − 1 , ˜ V ∈ O d,k , measurable set S ⊆
( R d ) k − 1 , and orthogonal matrix G ∈
O d ,
Pr[( µ 1 , . . . , µ k − 1 ) ∈ S | V =
˜
V , V 1 = ˜ V 1 ] =
Pr[( µ 1 , . .
. , µ k − 1 ) ∈ S G | V = G ˜ V , V 1 = G ˜ V 1 ] ,
[Table]
[Figure]
where S G = { ( G ˜ µ 1 , . .
, G ˜ µ k − 1 ) : (˜ µ 1 , . . . , ˜ µ k − 1 ) ∈ S } .
For any ˜ V , ˜ V ′ ∈
O d,k whose first k − 1 columns are both ˜ V 1 , there exists G ∈ O d such that ˜ V
=
G ˜ V and thus ˜ V 1 =
G ˜ V 1 .
This implies that for any ˜ µ ∈ col ( ˜ V 1 ) , we have G ˜ µ = ˜ µ , and thus ( S ∩ col ( ˜ V 1 ) k − 1 ) G = S ∩ col ( ˜ V 1 ) k −for any measurable S ⊆
( R d ) k − 1 .
Here, col ( ˜ V 1 ) k − 1 = { (˜ µ 1 , . . .
, ˜ µ k − 1 ) :
˜ µ i ∈ col ( ˜ V 1 ) for i = 1 , . .
, k − 1 } ⊆
( R d ) k − 1 .
When conditioned on V 1 = ˜ V 1 , for every i = 1 , . .
, k − 1 we have µ i ∈ Γ 1 = col
( V 1 )
=
col ( ˜ V 1 ) , which implies that ( µ 1 , . .
. , µ k − 1 ) ∈ col
( ˜ V 1 ) k − 1 .
Therefore,
[Equation]
This implies that ( µ 1 , . . . , µ k − 1 ) and V are conditionally independent given V 1 .
Therefore, the joint distribution of ( V 1 , V, ( µ 1 , . .
. , µ k − 1 )) can be formed by first drawing V and V 1 , and then drawing µ 1 , . . . , µ k − 1 based only on V 1 and not on V .
Since µ k , . . .
, µ n are drawn iid from N (0 , σ 2 UU T ) =
N (0 , σ 2 V V T ) , we have the graphical model shown in Figure 2.
[Equation]
[Table]
=</latexit>
[Figure]
<latexit sha1_base64="DgSM/I3nSRZMDaEWpjQDuGrqfY4=
[Figure]
>
[Figure]
+h
[Figure]
+
[Figure]
By Claim 3.6, the marginal distribution of V is Haar ( O d,k ) .
By Claim 3.5, we can implement this distribution by first drawing W ∼ Haar ( O d ) and then drawing E independently from any distribution over O d,k and let V = WE .
We choose the distribution of E later, where we ensure that the first k − 1 columms of E is always
[Equation]
[Equation]
This guarantees that the first k − 1 columns of W and V are the same, and thus V 1 is exactly the first k − 1 columns of W , resulting in the graphical model shown in Figure 3.
[Equation]
[Table]
">AAACfXicbZHNbhMxEMedLR8lfPQDiQsXi6gShyhaJ4jkGKUHOLZA0krZKPJ6Z1Ortndle2lXlh+h1/IUvAUvwYVXAefj0KaMZOuvmd+MxzNpKbixcfy7Ee08evzk6e6z5vMXL1/t7R8cTkxRaQZjVohCn6fUgOAKxpZbAeelBipTAWfp5fEyfvYdtOGF+mbrEmaSLhTPOaM2uL5O5mS+34o78crwQ0E2ojV8c/qH/xz9OpkfNLIkK1glQVkmqDFTEpd25qi2nAnwzaQyUFJ2SRcwDVJRCWbmVr16fBQ8Gc4LHY6yeOW9m+GoNKaWaSAltRdmO7Z0/i82rWw+mDmuysqCYuuH8kpgW+Dlx3HGNTAr6iAo0zz0itkF1ZTZMJ5mM8kgDzNc9eNknYoKvPvyaeRd3MaEDML1ceC3MQ3Zhur2+m3c+xCwXqAUXLFCSqoyl1x755J7da/9FlLfRZY1a+/DXsj2Fh6KSbdDep3uKWkNR2htu+gteofeI4L6aIg+oxM0Rgwt0A26RT8af6OjqB111mjU2OS8Rvcs6v8DWZLGyQ==</latexit>
Note that in Figure 3 there is no directed path from E to ( µ 1 , . . . , µ k − 1 ) .
Intuitively, this means that knowing ( µ 1 , . . . , µ k − 1 ) gives us no information about E .
Now by choosing the distribution of E appropriately, we can prove (11) in which the denominator does not contain γ 1 , . . .
, γ k − 1 .
We defer the complete proof of Theorem 3.7 to Appendix K.
Linear Models
In the linear models setting, the data distribution of user i is parameterized by an unknown vector β i ∈ R d .
As before, we assume that the vectors β 1 , . . .
, β n from the n users lie in an unknown k -dimensional subspace Γ .
Our goal is to recover the subspace using the following data.
For every i = 1 , . . . , n , we have m i data points from user i : ( x i 1 , y i 1 ) , . . .
, ( x im i , y im i ) ∈ R d × R .
For every j = 1 , . . .
, m i , we assume the measurement x ij ∈ R d is a random vector drawn independently from an O (1) -sub
-Gaussian distribution with zero mean and identity covariance matrix.
The measurement outcome y ij is determined by y ij =
x T ij β
i + z ij , where the random noise z ij ∈ R can depend on the measurements x i 1 , . . .
, x im i .
When conditioned on x i 1 , . .
, x im i , we assume every z ij for j = 1 , . . .
, m i is independently drawn from an η i -sub-Gaussian distribution with zero mean, but we do not assume that the conditional distribution of z ij is the same for every j = 1 , . . .
, m i .
The (in)dependence among x ij and z ij for i = 1 , . .
. , n and j = 1 , . .
, m i can be summarized by the example graphical model in Figure 4.
[Equation]
[Table] [Table]
JacjX1CDZ38PJu4AlgVJwyUopqcpdeuWdS2/
[Table] [Table]
>
Since we allow the noise z ij to depend on the measurements x ij , it is information-theoretically impossible to recover the subspace if we only have one data point from every user.
Consider the scenario where every β i is drawn independently from N (0 , σ 2 uu T ) for an unknown unit vector u ∈ R d and every x ij is drawn independently and uniformly from {− 1 , 1 } d .
If we set z ij to be z ij = x T ij ν ij where ν ij is independently drawn from N (0 , σ 2 ( I − uu T )) , then every y ij satisfies y ij = x T ij ( β i + ν ij ) where β i + ν ij distributes as N (0 , σ 2 I ) independently from x ij .
This implies that the joint distribution of (( x i 1 , y i 1 ))
i
=1 ,...,n does not change with u , i.e., we get no information about u from one data point per user.
Thus, we assume m i ≥ 2 for every user i .
In this case, we achieve error upper bounds that match the ones in [Tripuraneni et al., 2021] despite our relaxed assumptions on the noise.
Our estimator is the subspace ˆΓ spanned by the top- k eigenvectors of A defined in (3).
We defer the analysis of our estimator to Appendix L.
Acknowledgments and Disclosure of Funding
Part of this work was performed while LH was interning at Apple.
LH is also supported by Omer
Reingold’s NSF Award IIS-1908774, Omer Reingold’s Simons Foundation Investigators Award and Moses Charikar’s Simons Foundation Investigators Award.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work aims to develop better subspace recovery methods for a diverse population of users with heterogeneous data distributions.
Researchers and practitioners working on transfer learning, federated learning, data analysis, and statistics will benefit from our work.
We are not aware of any negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]