P RIVATE AND E FFICIENT M ETA -L EARNING WITH L OW R ANK AND S PARSE DECOMPOSITION
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta-learning is critical for a variety of practical ML systems – like personalized recommendations systems – that are required to generalize to new tasks despite a small number of task-speciﬁc training points.
Existing meta-learning techniques use two complementary approaches of either learning a low-dimensional representation of points for all tasks, or task-speciﬁc ﬁne-tuning of a global model trained using all the tasks.
In this work, we propose a novel meta-learning framework that combines both the techniques to enable handling of a large number of data-starved tasks.
Our framework models network weights as a sum of low-rank and sparse matrices.
This allows us to capture information from multiple domains together in the low-rank part while still allowing task speciﬁc personalization using the sparse part.
We instantiate and study the framework in the linear setting, where the problem reduces to that of estimating the sum of a rank- r and a k -column sparse matrix using a small number of linear measurements.
We propose an alternating minimization method with hard thresholding – AMHT-LRS– to learn the low-rank and sparse part effectively and efﬁciently.
For the realizable, Gaussian data setting, we show that AMHT-LRS indeed solves the problem efﬁciently with nearly optimal samples.
We extend AMHT-LRS to ensure that it preserves privacy of each individual user in the dataset, while still ensuring strong generalization with nearly optimal number of samples.
Finally, on multiple datasets, we demonstrate that the framework allows personalized models to obtain superior performance in the data-scarce regime.
I NTRODUCTION
Typical real world settings – like multi user/enterprise personalization – have a long tail of tasks with a small amount of training data.
Meta-learning addresses the problem by learning a "learner" that extracts key information/representation from a large number of training tasks, and can be applied to new tasks despite limited number of task speciﬁc training data points.
Most existing meta-learning approaches can be categorized as: 1) Neighborhood Models : these methods learn a global model, which is then "ﬁne-tuned" to speciﬁc tasks (Guo et al., 2020; Howard &
Ruder, 2018; Zaken et al., 2021), 2) Representation Learning : these methods learn a low-dimensional representation of points which can be used to train task-speciﬁc linear learners (Javed & White, 2019; Raghu et al., 2019; Lee et al., 2019; Bertinetto et al., 2018;
Hu et al., 2021).
In particular, task-speciﬁc ﬁne-tuning has demonstrated exceptional results across many natural language tasks (Devlin et al., 2018; Liu et al., 2019;
Yang et al., 2019; Lan et al., 2019).
However, such ﬁne-tuned models update all parameters, so each ﬁne-tuned total parameter footprint is same as the original model.
This implies that ﬁne-tuning large models –like say a standard BERT model (Devlin et al., 2018) with about 110M parameters – for thousands or millions of tasks would be quite challenging even from storage point of view.
One potential approach to handle the large number of parameters is to ﬁne-tune only the last layer, but empirical ﬁndings suggest that such solutions can be signiﬁcantly less accurate than ﬁne-tuning the entire model (Chen et al., 2020a;
Salman et al., 2020).
Moreover, representation learning based approaches apply strict restrictions like each task’s parameters have to be in a low-dimensional subspace which tend to affect performance in general (Sec. 3).
In this work, we propose and study the LRS framework that combines both the above mentioned complementary approaches.
That is, LRS restricts the model parameters Θ ( i ) for the i th task as Θ ( i ) :
=
U · W ( i ) + B ( i ) , where ﬁrst term denotes applying a low-dimensional linear operator on
W ( i ) , while B ( i ) is restricted to be sparse.
That is the ﬁrst term is based on representing parameters of each task in a low dimensional subspace while the second term allows task-speciﬁc ﬁne-tuning but only of a few parameters.
Note that methods that only allows ﬁne-tuning of batch-norm statistics, or of the ﬁnal few layers can also be thought of as "sparse" ﬁne-tuning but with a ﬁxed set of parameters.
In contrast, we allow the tunable parameters to be selected from any part of the network.
The framework allows collaboration among different tasks so as to learn accurate model despite lack of data per task.
Similarly, presence of sparse part allows task-speciﬁc personalization of arbitrary but a small set of weights.
Finally, the framework allows tasks/users to either contribute to a central model using privacy preserving bill-board models (see Section 2.2) or also allows ﬂexibility where certain tasks only learns their parameters locally and do not contribute to the central model.
While the framework applies more generally, to make the exposition easy to follow, we instantiate it in the case of linear models.
In particular, suppose the goal is to learn a linear model for the i th task that is parameterized by θ ( i ) , i.e., expected prediction for the data point x ∈ R d is given by (cid:104) x
, θ ( i ) (cid:105) .
θ ( i ) is modeled as θ ( i ) :
= U (cid:63) w (cid:63) ( i ) + b (cid:63) ( i ) for all tasks 1 ≤ i ≤ t , where U (cid:63) ∈ R d × r (where r (cid:28) d ) is a tall orthonormal matrix that captures task representation and is shared across all the tasks.
Note that estimation of the low-rank and sparse part is similar to the robust-PCA (Netrapalli et al., 2014) that is widely studied in the structured matrix estimation literature.
However, in that setting, the matrices are fully observed and the goal is to separate out the low-rank and the sparse part.
In comparison, in the framework proposed in this work, we only get a few linear measurements of the underlying matrix due to which the estimation problem is signiﬁcantly more challenging.
We address this challenge of estimating U (cid:63) along with w (cid:63) ( i ) , b (cid:63) ( i ) (for each task) using a simple alternating minimization style iterative technique.
Our method – AMHT-LRS– alternatingly estimates the global parameters U (cid:63) as well as the task-speciﬁc parameters w (cid:63) ( i ) and b (cid:63) ( i ) independently for each task.
To ensure sparsity of b we use an iterative hard-thresholding style estimator (Jain et al., 2014).
In general, even estimating U (cid:63) is an NP-hard problem (Thekumparampil et al., 2021).
One of the main contributions of the paper is a novel analysis that shows that AMHT-LRS indeed efﬁciently converges to the optimal solution in the realizable setting assuming the data is generated from a
Gaussian distribution.
Formally, consider t , d -dimensional linear regression tasks (indexed by i ∈ [ t ] ) with m samples { ( x ( i ) j , y ( i ) j ) }
mj
=1 being provided to each of them such that
[Equation]
where z ( i ) j ∼ iid N (0 , σ 2 ) .
Below, we state our main result informally in the noiseless setting ( σ = 0 ):
Theorem (Informal, Noiseless setting) .
Suppose we are given m · t samples from t linear regression tasks of dimension d as in equation 1.
Goal is to learn a new regression task’s parameters using m samples, i.e., learn the shared rank- r parameter matrix U (cid:63) along with task-speciﬁc w (cid:63) , b (cid:63) .
Then,
AMHT-LRS with total m · t = (cid:101) Ω( kdr 4 ) samples and m
= (cid:101) Ω(max( k, r 3 ))
samples per task can recover all the parameters exactly and in time nearly linear in m · t .
That is, AMHT-LRS is able to learn the underlying model exactly as long as the total number of tasks is large enough, and per task samples scale only linearly in sparsity of b and cubically in the rank of U (cid:63) .
Assuming r, k (cid:28) d , additional parameter overhead per task is small, allowing efﬁcient deployment of such models in production.
Finally, using the billboard model of ( (cid:15), δ ) differential privacy (DP) (Jain et al., 2021; Chien et al., 2021; Kearns et al., 2014), we can extend AMHT-LRS to preserve privacy of each individual.
Furthermore, for similar sample complexity as in the above theorem albeit with slightly worse dependence on r , we can guarantee strong generalization error up to a standard error term due to privacy.
Summary of our Contributions:
• We propose a theoretical framework for combining the meta-learning approaches of representation learning and neighborhood model .
Our model non-trivially generalizes guarantees of Thekumparampil et al.
Tripuraneni et al.
; Boursier et al.
to the setting where the parameter matrix allows a low rank plus sparse decomposition.
• We propose an efﬁcient method AMHT-LRS for the above problem and provide rigorous total sample complexity and per-task sample complexity bounds that are nearly optimal (Theorem 1).
• We provide a DP variant of AMHT-LRS that guarantees user level privacy.
At a high-level we show that under ( (cid:15), δ ) -DP, one can obtain a generalization error as a non-private version but with an additional error due to privacy budget (see Theorem 3).
• We demonstrate experiments on synthetic data and Movielens dataset using linear models (Sec. 3) and toy neural nets (Appendix A); apart from showing the advantage of our framework, they also show limitations of only using representation learning or a single model among other baselines.
Technical Challenges (Linear Models): Denote ( W (cid:63) ) T =
[ w (cid:63) (1) , . .
, w (cid:63) ( t ) ]
∈ R r × t and B (cid:63) =
[ b (cid:63) (1) , . . .
, b (cid:63) ( t ) ]
∈ R d × t ; hence the matrix of optimal regressors is given by U (cid:63) ( W (cid:63) ) T + B (cid:63) .
There are several technically novel steps that are required for recovering the model parameters in (1) that combine both the representation and neighborhood models.
In the general setting, even the taskspeciﬁc r -dimensional regression coefﬁcients { w (cid:63) ( i ) } i ∈ [ t ] are unknown and therefore, we are faced with the additional challenge of learning { w (cid:63) ( i ) }
i ∈
[ t ] jointly along with the shared representation matrix U (cid:63) and the task-speciﬁc sparse parameter vectors { b (cid:63) ( i ) } i ∈ [ t ] .
Note that in the AM framework proposed in Thekumparampil et al.
(2021), the authors consider only the representation model and not the neighborhood model (which is a simpler case of LRS setting).
Similarly, in Netrapalli et al.
(2014), the authors design an AM algorithm for the problem of reconstructing the low rank and sparse components of a matrix if the matrix is provided as an input.
However, in our setting, we only observe linear measurements of the individual columns of the parameter matrix.
Therefore, informally speaking, our analysis is faced with the key challenge of combining both sets of complementary techniques in Thekumparampil et al.
; Netrapalli et al.
This leads to the analysis of several crucial steps in each iteration: 1) We track the incoherence of several intermediate matrices corresponding to the latest estimates W ( (cid:96) ) , U ( (cid:96) ) of W (cid:63) , U (cid:63) .
We also track the L 2 , ∞ norm of the matrix ( I − U (cid:63) ( U (cid:63) ) T ) U ( (cid:96) ) to make progress on learning B (cid:63) .
In particular, the second step is the most technically involved component of our analysis.
Organization:
In Sec. 2, we introduce the general low rank+sparse ( LRS ) framework.
In Sec. 2.1, we provide theoretical guarantees for the canonical linear model in the LRS framework and provide a differentially private version in Sec. 2.2.
We analyze a special setting in Appendix B as warm-up (Rmk. 3) while detailed proofs are delegated to Appendix C, D.
In Sec. 3 and Appendix A, we provide experimental results on synthetic and real datasets.
In Appendix E, we discuss how to obtain a good initialization for our methods in the realizable setting via Method of Moments.
LRS FRAMEWORK FOR M ETA - LEARNING /P ERSONALIZATION
Notations: [ m ] to denotes the set { 1 , 2 , . . .
, m } .
For a matrix A , A i denotes i th row of A .
For a vector x , x i denotes i th element of x .
We sometimes use x j to denote an indexed vector; in this case x j,i denotes the i th element of x j .
||·|| 2 denotes euclidean norm of a vector and the operator norm of a matrix.
||·|| ∞ , ||·|| 0 will denote the (cid:96) ∞ and (cid:96) 0 norms of a vector respectively.
||·|| 2 , ∞ , ||·|| F will be used to denote the L 2 , ∞ and Frobenius norm of a matrix respectively.
For a sparse vector v ∈ R d , we deﬁne the support supp ( v ) ⊆
[ d ] to be a set of indices such that v i (cid:54) =
for all i ∈ supp ( v ) and v i = 0 otherwise.
We use I to denote the identity matrix.
(cid:101) O ( · ) notation subsumes logarithmic factors.
X ( i ) ∈
R m × d denotes the matrix of covariates for the i th task such that X ( i ) j
= ( x ( i ) j ) T .
Similarly, we write y ( i ) , z ( i
) ∈
R m to denote the task-speciﬁc response vector and noise vector respectively.
Let x ∈ R
d be the input point, and let (cid:98) y =
f ( x ; Θ ) be the predicted label using a DNN f with parameters Θ .
Let there be t tasks/domains.
Then, the goal is to personalize Θ to Θ ( i ) for each task i ∈
[ t ] such that a) Θ ( i ) does not over-ﬁt despite a small number of data-points labelled as task i , b) { Θ ( i ) , 1 ≤ i ≤ t } can be stored and inferred fast even for large t , say for t ≥ 1 M , c) the framework allows enough ﬂexibility to ensure that different tasks/domains/users can contribute their data at different levels of privacy risks.
Our method LRS attempts to address all the three requirements using a simple low-rank+sparse approach: we model each Θ ( i ) as Θ ( i ) :
= U · W ( i ) + B ( i ) , where U W ( i ) denotes a linear operation on W ( i ) in a r -dimensional basis speciﬁed by U (can be very large for e.g. standard BERT with ∼ 110 M parameters), and B ( i ) is a k -sparse matrix.
That is, we represent Θ ( i ) as a combination of a small number of parameter matrices represented by U , along with a sparse set of weights that can be ﬁne-tuned arbitrarily for a given task.
Note that due to low-dimensional and sparse representation, LRS should require relatively small number of points per task.
In the next section, we formally prove this claim for a simple linear setting with Gaussian data.
Furthermore, for each task the additional number of parameters is relatively small – assuming r and k are small – which implies that memory cost can be controlled.
The latency cost also remains same as the baseline model, assuming we can explicitly compute Θ ( i ) on the ﬂy.
Finally, the model allows tasks/domains/users to contribute data to learn U with differential privacy (see Section 2.2), but it also admits domains/tasks who are not inclined to share data, and can just ﬁne-tune their model privately by learning W ( i ) and B ( i ) in isolation.
was proposed by Hu et al. (2021) for meta-learning with large number of tasks at scale.
Although the authors demonstrate promising experimental results, LORA only allows a central model (in a low dimensional manifold) and does not incorporate sparse ﬁne-tuning.
Hence, LORA becomes ineffective when the output dimension is small (say 1 ).
The said limitation of LORA has been demonstrated in detailed experiments on the MovieLens 1M dataset (a standard recommendation dataset) in Sec. 3.
Moreover, the low rank ﬁne-tuning (as proposed in LORA) is limited when output dimension is small.
Finally, LORA does not have any theoretical guarantees even in simple settings.
Comparison with representation learning style models : A recent line of work (Thekumparampil et al., 2021; Du et al., 2020; Tripuraneni et al., 2021; Boursier et al., 2022; Jain et al., 2021) proposes a similar model to LRS but is speciﬁc to representation learning.
These papers only consider a low-rank representation of the task parameters but no sparse ﬁne-tuning.
Moreover, the proposed algorithms in these papers have not been explored at scale and mostly been applied to vanilla linear models (without considering privacy constraints or extension to more complex models).
Even from a theoretical point of view, these methods do not apply in our case due to the additional non-convex sparsity constraint.
Comparison with Prompt-based and Batch-norm Fine-tuning : Another popular approach for personalization is to use prompt-based or batch-norm based ﬁne-tuning (Wang et al., 2022; Liu et al., 2021; Lester et al., 2021); this usually involves a task-based feature embedding concatenated with the covariate.
Note that in a linear model, such an approach will only lead to an additional scalar bias which can be easily modeled in our framework; thus our framework is richer and more expressive with a smaller number of parameters.
We have compared against such techniques in our experiments and demonstrated their limitations in both real and synthetic datasets (see Sec. 3).
Private Meta-learning: Model-personalization is a key application of meta-learning, where we wish to have a personalized model for each user i.e. each user represents a task.
Due to sensitivity of user-data, we would want to preserve privacy of each user for which we use user-level ( (cid:15), δ ) -DP as the privacy notion (see Deﬁnition 1).
In this setting, each user i ∈ [ t ] holds a set of data samples
D ( i ) =
{ x ( i ) j , 1 ≤ j ≤ m } .
Furthermore, users interact via a central algorithm that maintains the common representation matrix U which is guaranteed to be DP w.r.t.
all the data samples of any single user.
The central algorithm publishes the current U to all the users (a.k.a.
on a billboard)
and obtains further updates from the users.
It has been shown in prior works (Jain et al., 2021;
Chien et al., 2021; Thakkar et al., 2019) that such a billboard mechanism allows for signiﬁcantly more accurate privacy preserving methods while ensuring user-level privacy.
In particular, it allows learning of U effectively, while each user can keep a part of the model which is personal to them, e.g., the W (cid:63) ( i ) , B (cid:63) ( i ) ’s in our context.
See (Jain et al., 2021, Section 3) for more details about billboard model in the personalization setting.
Traditionally, such model of private computation is typically called the billboard model of DP, which in turn is a subclass of joint DP (Kearns et al., 2014).
Deﬁnition 1.
Differential Privacy Dwork et al.
(2006b;a); Bun & Steinke (2016)
A randomized algorithm A is ( ε, δ ) -differentially private if for any pair of data sets D and D (cid:48) that differ in one user (i.e., | D (cid:52) D (cid:48)
| = 1 ), and for all S in the output range of A , we have
[Equation]
where probability is over the randomness of A .
Similarly, an algorithm A is ρ -zero Concentrated DP (zCDP) if D α
( A ( D ) ||A ( D (cid:48) )) ≤ αρ , where D α is the Rényi divergence of order α .
In Deﬁnition 1, when we deﬁne the notion of neighborhood, we deﬁne it w.r.t.
the addition (removal) of a single user (i.e., additional removal of all the data samples D i for any user i ∈ [ t ] ).
In the literature Dwork & Roth (2014), the deﬁnition is referred to as user-level DP.
2. L INEAR LRS : A LGORITHM AND A NALYSIS
In this section, we describe our LRS framework for the linear setting, provide an efﬁcient algorithm for parameter estimation, and provide rigorous analysis under realizable setting with Gaussian data.
We then extend our framework, algorithm and analysis to allow user-level differential privacy.
Consider the linear LRS model introduced in Section 1 where we have t d -dimensional linear regression tasks (indexed by i ∈ [ t ] ) with m samples { ( x ( i ) j , y ( i ) j ) } mj =1 being provided to each of them such that the i th sample for the j th task ( x ( i ) j ,
y ( i ) j ) is generated independently according to eq.
So the problem reduces to that of designing statistically and computationally efﬁcient algorithms to estimate the common representation learning parameter U (cid:63) as well as task-speciﬁc parameters { w
[Equation]
( i ) } i ∈ [ t ] , { b (cid:63) ( i ) } i ∈ [ t ] .
The ERM for this model assuming squared loss is given by:
[Equation]
minimize L ( U , W , B ) = (cid:88)
[Equation]
(2) where U ∈ R d × r , W =
[ w (1) w (2) . . .
w ( t ) ] T ∈ R t × r stores the task-speciﬁc coefﬁcients, and
B =
[ b (1) b (2) . . .
b ( t ) ] ∈ R d × t stores the task-speciﬁc sparse vectors for ﬁne-tuning .
Note that
LRS is non-convex due to: a) bilinearity of U , W , b) non-convexity of (cid:96) 0 norm constraint.
We propose AMHT-LRS that handles the non-convexity in the objective and the constrained set by carefully combining alternating minimization for U , w and b with hard thresholding to ensure sparsity of b .
Let HT : R d × R → R d be a hard thresholding function that takes a vector v ∈ R d and a parameter ∆ as input and returns a vector v (cid:48)
∈ R d such that v (cid:48)
i =
v i if | v i | > ∆ andotherwise.
Let U +( (cid:96) − 1) , { w ( i,(cid:96) − 1) } i ∈ [ t ] and { b ( i,(cid:96) − 1) } i ∈
[ t ] be the latest iterates at the beginning of the (cid:96) th iteration.
First, for each task i ∈ [ t ] , given estimates U +( (cid:96) − 1) , w ( i,(cid:96) − 1) , we can update b ( i,(cid:96) ) by solving the following problem:
[Equation]
While the problem is non-convex, we can still apply a projected gradient descent algorithm which reduces to iterative hard thresholding.
In particular, we use Algorithm 2 for the i th task where in each iteration, we run a gradient descent step on the parameter vector estimate b (of b (cid:63) ( i ) ) and subsequently apply HT ( · , ∆) function where ∆ > 0 is set appropriately.
Next, given estimates
U +( (cid:96) − 1) , b ( i,(cid:96) ) , we can update w ( i,(cid:96) ) by solving the following task-speciﬁc optimization problem
[Equation]
∈ [ t ]
[Equation]
Subsequently, given the updated estimates of the task-speciﬁc parameters { w ( i,(cid:96) ) } i ∈ [ t ] and { b ( i,(cid:96) ) } i ∈ [ t ] , we update U +( (cid:96) )
(estimate of shared representation matrix) using:
[Equation]
followed by a QR decomposition of the solution.
Note that the above two problems ((4) and (5)) can be solved using standard least squares regression methods.
Finally, we must ensure independence of the estimates (which are random variables themselves) from the data that is used in a particular update.
We can ensure such independence by using a fresh batch of samples in every iteration.
Analysis: As in prior works, we are interested in the few-shot learning regime when there are only a few samples per task.
From information theoretic viewpoint, we expect the number of samples per task to scale linearly with the sparsity k and rank r and logarithmically with the dimension d .
On the other hand, U (cid:63) has dr parameters and therefore, it is expected that the total number of samples across all tasks scales linearly with dr which implies we would want the number of tasks t to scale linearly with dimension d .
Note that if the sparse vectors { b (cid:63) ( i ) } i ∈ [ t ] have the same support (or a high overlap between the supports), then the model parameters might not be uniquely identiﬁable.
This is because, in that case, the matrix B (cid:63) can be represented as a low-rank matrix.
To establish identiﬁability of U (cid:63) and sparse vectors b (cid:63) , we make the following assumption:
Require: Data { ( x ( i ) j ∈ R d , y ( i ) j ∈ R ) } mj =1 for all i ∈
[ t ] , column sparsity k of B , (cid:12)(cid:12)(cid:12)(cid:12) ∆( U +(0) , U (cid:63) ) (cid:12)(cid:12)(cid:12)(cid:12) F ≤ B , max i (cid:107) b ( i, 0) − b (cid:63) ( i ) (cid:107) ∞
≤ γ (0) , Parameter (cid:15) > 0 .
1: for (cid:96) = 1 , 2 , . . .
do 2:
Set T ( (cid:96) )
=
Ω (cid:16) (cid:96) log (cid:16) γ ( (cid:96) − 1) (cid:15) (cid:17)(cid:17) 3: for i = 1 , 2 , . .
. , t do 4: b ( i,(cid:96) )
←
OptimizeSparseVector (( X ( i ) , y
( i ) ) , v =
U +( (cid:96) − 1) w ( i,(cid:96) − 1) , α =
O (cid:16) c (cid:96) −
B √ k (cid:17) , β
= O ( c (cid:96) −
B ) , γ = γ ( (cid:96) − 1) , T = T ( (cid:96) ) ) for suitable constants
c 4 , c 5 > 0 .
5: w ( i,(cid:96) )
= (cid:16) ( X ( i ) U
[Table]
[Table] [Table]
U
[Table] [Table]
) (cid:17)
[Table]
(cid:16) (
[Table]
U
[Table]
[Table] [Table]
− X (
[Table] [Table]
,(cid:96) for 7: i
Assumption 1 (A1) .
Consider the matrix B (cid:63) ∈ R d × t whose i th column is the vector b (cid:63) ( i ) .
Then each row of B (cid:63) is ζ -sparse i.e. (cid:107) B (cid:63)i (cid:107)
≤ ζ for all i ∈ [ d ] , and each column is k -sparse.
Note that the orthonormal matrix U (cid:63) cannot have extremely sparse columns otherwise it would be information theoretically impossible to separate columns of U (cid:63) from b (cid:63) .
Moreover, similar to
Tripuraneni et al.
(2021), we need to ensure that each task contributes to learning the underlying representation U (cid:63) .
These properties can be ensured by the standard incoherence assumptions
Tripuraneni et al.
; Collins et al.
; Netrapalli et al.
and therefore, we have
Assumption 2 (A2) .
Let λ (cid:63) 1 and λ (cid:63)r be the largest and smallest eigenvalues of the task diversity matrix ( r/t )( W (cid:63) )
T W (cid:63) ∈ R r × r .
We assume that W (cid:63) ∈ R t × r and the representation matrix U (cid:63) ∈ R d × r are µ (cid:63) -incoherent i.e. || W (cid:63) || 2 , ∞ ≤ (cid:112) µ (cid:63) λ
(cid:63)r and || U (cid:63) || 2 , ∞ ≤ (cid:113) µ (cid:63) rd .
Theorem 1.
Consider the LRS problem equation 2 with t linear regression tasks and samples obtained by equation 1.
Let model parameters satisfy assumptions A1, A2.
Let the row sparsity of B (cid:63) satisfy ζ
= O (cid:16) t ( r 2 µ (cid:63) )
− 1 (cid:113) λ
(cid:63)r λ (cid:63) 1 (cid:17) , and let k = O (cid:16) d · ( λ (cid:63)r λ (cid:63) 1 ) 2 (cid:17) .
Suppose Algorithm 1 is initialized with
U +(0) such that (cid:12)(cid:12)(cid:12)(cid:12) ( I − U (cid:63) ( U (cid:63) ) T ) U +(0) (cid:12)(cid:12)(cid:12)(cid:12) F
= O (cid:16)(cid:113) λ
(cid:63)r λ (cid:63) 1 (cid:17) and (cid:12)(cid:12)(cid:12)(cid:12) U
+(0) (cid:12)(cid:12)(cid:12)(cid:12) 2 , ∞
=
O ( (cid:112) µ (cid:63) r/d ) , and is run for L = (cid:101)
O
(1) iterations.
Then, with high probability, the outputs U +( L ) , { b ( i, L ) } i ∈ [ t ] satisfy:
[Equation]
, i
[Equation]
[ t ] , (6) where S = (cid:16) µ (cid:63) (cid:113) r 3 dmt + (cid:113) r 3 mλ (cid:63)r + (cid:113) km (cid:17) provided the total number of samples satisﬁes:
[Equation]
λ
[Equation]
For a new task, modiﬁed AMHT-LRS (Alg. 6 in Appendix D) has the following generalization bound:
[Equation]
Note that the per-task sample complexity of our method roughly scales as m =
( r 3 + k ) , which is information theoretically optimal in k and is roughly r 2 factor larger.
Total sample complexity scales as mt = kdr 4 , which is roughly kr 3 multiplicative factor larger than the information theoretic bound.
Note that typically r and k are considered to be small, so the additional factors are small, but we leave further investigation into obtaining tighter bounds for future work.
Finally, the generalization error scales as σ 2 ( r + k ) /m which is nearly optimal.
Note that, ignoring meta-learning, and directly optimizing the single-task error would lead to signiﬁcantly larger error of σ 2 d/m .
Remark 1 (Runtime and Memory) .
The run-time of Algorithm 1 is dominated by the update for
U +( (cid:96) ) .
For each iteration (cid:96) , Step 8 has a time complexity of O (( dr ) 3 + ( mt )( dr ) 2 ) ; however in practice, a gradient descent step for the update of U ( (cid:96) ) can bring down the time complexity to
O ( mtdr ) .
Moreover, the memory usage of Algorithm 1 is O (( dr ) 2 + tr 2 ) .
Remark 2 (Initialization) .
Note that Algorithm 1 has local convergence properties as described in
Theorem 1.
In practice, typically we use random initialization for U +(0) .
However, similar to the representation learning framework in Tripuraneni et al.
(2021), we can use the Method of Moments to obtain a good initialization.
See Appendix E for more details.
Remark 3 (Special Settings) .
In the setting where for each task, we just need to learn a single central model for all tasks and sparse ﬁne-tune the weights for each task i.e. w (cid:63) ( i ) =
1 for all i ∈
[ t ] is ﬁxed, AMHT-LRS obtains global convergence guarantees (Theorem 4 in Appendix B).
Moreover, if the central model U (cid:63) is also frozen, then the task-based sparse ﬁne-tuning reduces to standard compressed sensing.
In the realizable setting, our framework recovers the standard generalization error of σ 2 k/m in compressed sensing (Jain & Kar, 2017)[Chapter 7].
Remark 4 (Sample complexity comparison) .
Note that the theoretical guarantees in representation models studied in Tripuraneni et al.
; Thekumparampil et al.
; Collins et al.
(2021) cannot capture the sparse ﬁne-tuning in each task (with potentially arbitrary magnitude).
However, since our framework combines neighborhood and representation models, our sample complexity guarantees are sub-optimal by only a factor of r 2 (generalization error is optimal) when restricted to the special case of representation model (Thekumparampil et al., 2021).
2. P RIVATE L INEAR LRS : P RIVACY P RESERVING M ETA - LEARNING
Algorithm
Require: Data ( X , y ) ∈
R m × d × R m where we minimize || y − X
( v (cid:63) + b (cid:63) )
||
2 such that || b (cid:63) || 0 ≤ k .
Estimate v (of v (cid:63) ) and initialization b (of b (cid:63) ).
Iterations T , parameters α, β, γ > 0 and suitable constants c > 0 , 0 < c 1 < 1 / 2 .
1: for j = 1,2,..., T do 2: c ← b − 1 m · ( X ( i ) )
T ( X ( i ) b + X ( i ) v − y
( i ) )
3: ∆ ← α + c 1 (cid:16) γ + β √ k (cid:17) and b ← HT ( c , ∆) 4: γ ← 2 c 1 γ + 2( α + c 1 √ k β ) 5: end for 6: Return vector b .
In this section, we provide a user level
DP variant of Algorithm 1 in the billboard model.
We obtain DP for the computation of each U ( (cid:96) ) by perturbing the covariance matrix A and the linear term V in the algorithm with Gaussian noise to ensure that the contribution of any single user is protected.
We start by introducing the function clip : R × R
→ R that takes as input a scalar x , parameter ρ and returns clip ( x, ρ ) = x · min (cid:110) 1 , ρx (cid:111) .
We can extend the deﬁnition of clip to vectors and matrices by using clip ( v , ρ ) = v · min (cid:110) 1 , ρ (cid:107) v (cid:107) 2 (cid:111) for a vector v and clip ( A , ρ )
=
A · min (cid:110) 1 , ρ (cid:107) A (cid:107) F (cid:111) for a matrix A .
In order to ensure that
Algorithm 1 is private, for input parameters
A 1 , A 2 , A 3 , A w , we ﬁrst clip the covariates and responses: for all i ∈ [ t ] , j ∈ [ m ] , we will have (cid:100) x ( i ) j ← clip (cid:16) x
( i ) j , A 1 (cid:17) , (cid:100) y
( i ) j ← clip (cid:16) y
( i ) j , A 2 (cid:17) , (cid:92) ( x ( i ) j ) T b ( i,(cid:96) )
← clip (cid:16) ( x ( i ) j ) T b ( i,(cid:96) ) , A 3 (cid:17) and (cid:92) w ( i,(cid:96) )
← clip (cid:0) w ( i,(cid:96) ) , A w (cid:1) .
Now, we can modify Line 7 in Algorithm 1 as follows (let L be the number of iterations of Alg. 1):
[Equation]
[Figure]
where, for some σ DP > 0 , each entry of N (1) is independently generated from
N (cid:0) 0 , m 2 · A 41 · A 4 w · L · σ 2 DP (cid:1) ; similarly, each entry of N (2) is independently generated from
N (cid:0) 0 , m 2 · A 21 ( A 2 + A 3 ) 2 A 2 w · L · σ 2 DP (cid:1) .
We are now ready to state our main result:
Theorem 2.
Algorithm 1 (with modiﬁcations mentioned in equation 7 and equation 8) satisﬁes σ − 2 DP − zCDP and correspondingly satisﬁes ( ε, δ ) -differential privacy in the billboard model, when we set the noise multiplier σ DP ≥ 2 ε − 1 (cid:112) (log(1 /δ ) + ε ) .
Furthermore, if ε ≤ log(1 /δ ) , then σ DP ≥ ε − 1 (cid:112) 8 log(1 /δ )
sufﬁces to ensure ( ε, δ ) -differential privacy.
Next, we characterize the generalization properties of modiﬁed AMHT-LRS:
Theorem 3.
Consider the LRS problem equation 2 with all parameters m, t, ζ obeying the bounds stated in Theorem 1 and furthermore, t = (cid:101) Ω( ( rd ) 3 /
[Equation]
µ
[Equation]
Suppose we run AMHT-LRS (Step 7 in Alg.
1 replaced with 7 and 8) for L = (cid:101) O (1) iterations with A 1 = (cid:101) O ( √ d ) , A 2 = (cid:101)
O ( (cid:112) µ (cid:63)
λ (cid:63)r
+
[Equation]
max i (cid:107) b (cid:63) ( i ) (cid:107) 2 )) ,
[Equation]
(cid:101) O (cid:16) λ
(cid:63)r (cid:113) µ (cid:63) λ (cid:63) 1 (cid:17) , A w = (cid:101) O
( (cid:112) µ (cid:63)
λ (cid:63)r ) .
Then, with high probability, generalization error for a new task satisﬁes:
[Equation]
where S = (cid:16) µ (cid:63)
[Equation]
r
[Equation] [Equation]
=
[Equation]
λ
[Equation]
σ DP
[Equation]
and κ = 1 + (cid:113)
[Equation]
max
[Equation]
√
[Equation]
λ (cid:63)r
[Equation]
Note that the modiﬁed AMHT-LRS ensures ( (cid:15), δ ) − differential privacy without any assumptions.
However Thm.
3 still has good generalization properties; moreover, the per-task sample complexity guarantee m still only needs to scale polylogarithmically with the dimension d .
In other words, our algorithm can ensure good generalization along with privacy in data-starved settings as long as the number of tasks is large - scales as ∼ d 3 / 2 /(cid:15) .
Similarly, generalization error for a new task has two terms: the ﬁrst has a standard dependence on noise σ 2 and the second has a scaling of d 3 ( (cid:15)t ) −which is standard in private linear regression and private meta-learning (Smith et al., 2017; Jain et al., 2021).
Detailed proofs of our main results namely Theorems 1,3 are delegated to Appendix C, D.
E MPIRICAL RESULTS
In this section, we conduct an empirical study of AMHT-LRS with the following two goals: a) demonstrate that personalization with AMHT-LRS indeed improves accuracy for tasks with a small number of points, b) for a ﬁxed budget of parameters, AMHT-LRS is signiﬁcantly more accurate than existing baselines.
For simplicity, we ﬁx the model class to be linear and consider the following baselines:
Single Model ( u central ) : learns a single model for all tasks, 2) Full Fine-tuning ( u indv ) separate model for each task aka standard ﬁne-tuning, 3) Representation Learning or Rep.
Learning
( uw
: only low
rank model (Thekumparampil et al., 2021) and 4) Prompt Learning u
T (x
|| c)
Modiﬁed covariate
by
concatenation with a task-embedding vector.
Note that the models considered in Hu et al.
; Chua et al.
; Denevi et al.
all reduce to Full ﬁne-tuning models (that have a high memory footprint) in the experimental settings that we consider.
Also, the above approaches are not just restricted to linear models and can be extended to complex model classes such as Neural Networks (see Appendix A for extension to 3 layer Neural Net architectures).
We conduct experiments on two datasets: a. Synthetic dataset : here, for each task i ∈ [ t ] , we generate m = 100 samples { ( x ( i ) j , y ( i ) j ) } j ∈ [ m ] where x ( i ) j ∼ N ( 0 , I d × d ) , y ( i ) j = (cid:104) x
( i ) j , u (cid:63) w (cid:63) ( i ) + b (cid:63) ( i ) (cid:105) .
We select d = 150 , set k, ζ , the column and row sparsity level of { b (cid:63) ( i ) } i ∈
[ t ] to be 10 and 5 , respectively.
We sample u (cid:63) uniformly from the unit sphere; non-zero elements of { b (cid:63) ( i ) } i ∈ [ t ] and w (cid:63) ( i ) are sampled i.i.d.
from N (0 , 1) with the indices of zeros selected randomly.
b. MovieLens Data : The MovieLens 1M dataset comprises of 1M ratings of 6K users for 4K movies.
Each user is associated with some demographic data namely gender, age group, and occupation in the
MovieLens dataset.
We partition the users into 241 disjoint clusters where each cluster represents a unique combination of the demographic data.
Each user group thus represents a "task" in the language of our paper.
We partition the data into training and validation in the following way: for each task, we randomly choose 20% movies rated by at least one user from that task and put all ratings made by users from that task for the chosen movie into the validation set.
The remaining ratings belong to the training set.
Based on the ratings in the training set, we ﬁt a matrix of rank 50 onto the ratings matrix and obtain a 50 dimensional embedding of each movie.
Thus we ensure that there is no data leakage while creating the embeddings.
For each task, the samples consist of (movie embedding, average rating) tuples; the response is the average rating of the movie given by users in that task.
The number of samples per task varies from 22 to 3070 - clearly many clusters are data starved.
We use the training data to learn the different models (with some hyper-parameter tuning) mentioned earlier and use them to predict the ratings in the validation data.
Empirical Observations on Synthetic Data : Figure 2 shows that not only having a single model can lead to poor performance, but a fully ﬁne-tuned model per task can also be highly inaccurate as scarcity of data per task can leading to over-ﬁtting.
Finally, low-rank representation learning as well as prompt tuning based techniques do not perform well due to lack of modeling power.
In contrast, our method is able to exactly recover the underlying parameters – as also predicted by Theorem 1 – and provides 5 orders of magnitude better RMSE.
Empirical Observations on MovieLens :
The overall average validation RMSE for
AMHT-LRS and the different baselines that we consider is shown in Fig.
1a against percentage of ﬁne-tunable parameters used by the model.
With respect to the single model as reference, in the linear rank- 1 case, the representation learning and the prompt learning based baselines have 1 and 50 additional parameters per task respectively; they are unable to personalize well.
In contrast, with only 10%(= 5) additional parameters per task, AMHT-LRS has smaller
RMSE than fully ﬁne-tuned model, which require 241x more parameters.
However, for data-starved clusters/tasks (samples < 100 ), we observe that fully ﬁne-tuning approach start to overﬁt.
In contrast, our method outperforms other baselines for both data-starved and data-surplus tasks.
C ONCLUSIONS
We presented a powerful theoretical framework to study meta-learning, and develop novel algorithms.
In particular, our framework combines representation learning and neighborhood ﬁne-tuning based approaches for meta-learning.
We proposed AMHT-LRS method, that combines alternative minimization – popular in representation learning – with hard thresholding based methods.
We rigorously proved that AMHT-LRS is statistically and computationally efﬁcient, and is able to generalize to new tasks with only O ( r + k ) samples, where r is the representation learning dimension and k is the number of ﬁne-tuning weights.
Finally, we extended our result to ensure that privacy of each task is preserved despite sharing information across tasks.
Extending our framework to non-realizable setting and adversarial settings are critical future directions.