Homomorphic Matrix Completion
Abstract
In recommendation systems, global positioning, system identiﬁcation, and mobile social networks, it is a fundamental routine that a server completes a low-rank matrix from an observed subset of its entries.
However, sending data to a cloud server raises up the data privacy concern due to eavesdropping attacks and the singlepoint failure problem, e.g., the Netﬂix prize contest was canceled after a privacy lawsuit.
In this paper, we propose a homomorphic matrix completion algorithm for privacy-preserving purpose.
First, we formulate a homomorphic matrix completion problem where a server performs matrix completion on cyphertexts, and propose an encryption scheme that is fast and easy to implement.
Secondly, we prove that the proposed scheme satisﬁes the homomorphism property that decrypting the recovered matrix on cyphertexts will obtain the target matrix (on plaintexts).
Thirdly, we prove that the proposed scheme satisﬁes an ( ✏ ,  ) -differential privacy property.
While with similar level of privacy guarantee, we reduce the best-known error bound O (p n 31 n 2 ) to EXACT recovery at a price of more samples.
Finally, on synthetic data and real-world data, we show that both homomorphic nuclear-norm minimization and alternating minimization algorithms achieve accurate recoveries on cyphertexts, verifying the homomorphism property.
Introduction
The recurring low-rank matrix completion problem [4, 6, 22, 30, 13, 29, 46] concerns completing a low-rank matrix from a randomly observed subset of entries.
It has wide applications in recommendation systems (collaborative ﬁltering) [1, 47, 26], computer vision [2, 16, 27], global positioning [48, 32], sensory data analysis in Internet of Things [25, 34, 33], system identiﬁcation, network data analysis [51, 11], mobile social networks [23, 38], etc.
Existing works [6, 9] have demonstrated a remarkable fact: if an n ⇥ n matrix with rank r ⌧ n satisﬁes a certain incoherence property, then with high probability, it is possible to exactly recover the matrix from O ( nr poly log n ) ⌧ n 2 entries using polynomial-time algorithms.
Intuitively, one needs roughly (2 nr  r 2 ) parameters [6] (this is the dimension of the tangent space to the manifold of rank- r matrices) to ﬁx an n ⇥ n matrix of rank r , and the sampling randomness introduces a log n factor due to a coupon collector’s effect.
The information theoretical lower bound is ⌦ ( nr log n ) [6], while the tightest known upper bound is
O ( nr log 2 n ) [9] with another log n factor from the Golﬁng scheme used by the recovery algorithms.
The low-rank matrix completion problem usually deals with large-scale matrices that involve extensive computations, while in mobile computing, smart devices usually outsource such a huge computation task to a cloud server.
However, sending data to a server or publishing anonymized data raises up privacy concerns [23, 44, 42], e.g., the recommendation contest Netﬂix prize was canceled after privacy lawsuit [35].
There are two major obstructive factors: anonymization in data publishing is still vulnerable, and storing sensitive data on a cloud server may encounter the single-point of failure
(SPOF) problem, say hackers.
Existing works [20, 19, 10] address the privacy concern in various ways, e.g., a popular approach is to add noise to the data [20], therefore making a tradeoff between the recovery accuracy and the level of privacy.
In cloud computing and distributed systems, the homomorphism property [14, 45] allows computations to be carried out on cyphertexts, generating an encrypted result which, when decrypted, matches the result of operations performed on the corresponding plaintexts.
In this manner, homomorphic encryption securely chains together different services without sacriﬁcing recovery accuracy, but may at a price of extensive computation .
There are several partially homomorphic crypto-systems, and also a number of fully homomorphic crypto-systems [14, 45].
In addition, the homomorphic property can also be used to create many other secure systems, for example secure voting systems, collision-resistant hash functions, private information retrieval schemes [43], etc.
In this paper, we integrate the large-scale distributed matrix completion task with a homomorphic encryption-decryption scheme, which guarantees the EXACT recovery and differential privacy at a price of more samples.
First, we deﬁne the homomorphic matrix completion problem that ensures data privacy by preserving a homomorphism property between plaintexts and cyphertexts.
Speciﬁcally, we propose a homomorphic encryption-decryption scheme, in which each node performs local encryption and decryption, and uploads an encrypted incomplete vector to a server that carries out the matrix completion computation.
Then, we theoretically prove that the proposed scheme satisﬁes the homomorphism and differential privacy properties — reducing the best-known error bound O (p n 31 n 2 ) [20] to EXACT recovery.
Finally, based on synthetic and real-world data, we show that the homomorphic nuclear-norm minimization and alternating minimization algorithms achieve accurate recoveries on both cyphertexts and plaintexts, verifying the homomorphism property.
Homomorphic Matrix Completion Problem
First, we formally deﬁne the homomorphic matrix completion problem.
Then, we introduce a notation of privacy by adapting the join ( ✏ ,  ) differential privacy, which is a subspace-aware variant.
2.1 Notations and Preliminaries
Notations : Let e i denote the i -th standard basis, I k denote the k ⇥ k identity matrix, and I denote the identity linear operator.
For matrix X , the ( i, j ) -th element is X ij or X ( i, j ) , the j -th column is X j , and the transpose is X > .
The concatenation of two matrices A 2 R n 1 ⇥ n 2 and B 2 R n 1 ⇥ n 3 with the same number of rows is denoted by [ A , B ] 2 R n 1 ⇥ ( n 2 + n 3 ) .
By with high probability (w.h.p.)
we mean that with probability at least 1  c 1 n  c 2 for some positive constants c 1 , c 2 .
Let N (0 ,  2 ) denote a Gaussian distribution with mean 0 and standard deviation  .
We use an overline to represent the encrypted version of a variable: variables before encryption are called plaintexts , e.g., X , while the encrypted variables are called cyphertexts , e.g., X .
Let M ⌦ 2 R n 1 ⇥ n 2 denote the observed entries of a data matrix M 2 R n 1 ⇥ n 2 , where ⌦ ✓ { (1 , 1) , (1 , 2) , ..., ( n 1 , n 2 ) } indicates the observed entries.
We deﬁne a linear operator
P ⌦ : R n 1 ⇥ n 2 ! R n 1 ⇥ n 2 to represent the partial observation model as follows
[Equation]
Assuming the true data matrix M is low-rank, i.e., rank ( M )
=
r ⌧ min( n 1 , n 2 ) .
The (compact) singular value decomposition (SVD) is M = U ⌃ V > , where U 2 R n 1 ⇥ r represents r left singular vectors (a basis of the column subspace), V 2 R n 2 ⇥ r represents r right singular vectors (a basis of the row subspace), and ⌃ = diag ([  1 ,  1 , · · · ,  r ]) 2 R r ⇥ r with singular values  1   2  · · ·   r  0 .
The ` 2 -norm of a vector is || x
|| 2 , while the Frobenius norm and nuclear norm of M are || M || F = qP i,j | M ij | 2 and || M ||
⇤ =
P ri =1  i , respectively.
The operator norm (spectral norm) of a matrix and a linear operator L are deﬁned as follows
[Equation]
Deﬁnition 1.
( Column subspace and null space [28]) Let A 2 R n 1 ⇥ n 2 .
The set S ( A ) = { b R n 1 | b =
Ax , x 2 R n 2 } is the column space or range of A , and the set Ker ( A )
=
{ x R n 2 | Ax = 0 } is the kernel or (right) null space of A .
The null space (kernel space) of operator P ⌦ is Ker ( P ⌦ ) =
{ Z 2 R n 1 ⇥ n 2 | P ⌦ ( Z ) = 0 } , which is denoted as ⌦ ?
Let ⌦ ⇠ Uni ( m ) denote a set with m entries, which is sampled uniformly from all sets of m entries, and ⌦ ⇠ Ber ( p ) denote a set with E | ⌦ | = m entries, where each entry is sampled independently according to a Bernoulli model with p = m/ ( n 1 n 2 ) .
Let P U and P V denote the orthogonal projections onto the column and row space of M , respectively,
[Equation]
Deﬁne an orthogonal decomposition R n 1 ⇥ n 2 = T  T ? , where T is the linear space spanned by matrices with the same column space or row space as M , and T ? is its orthogonal complement that consists of matrices with row-space orthogonal to the row-space V and column-space orthogonal to the column-space U .
T can be expressed as follows
[Equation]
The orthogonal projection P T onto T and the orthogonal projection onto T ? are as follows
[Equation]
2.2 Problem Formulation for Homomorphic Matrix Completion
We are interested in completing large-scale matrices and want to outsource this compute-intensive task from mobile devices to a cloud server.
Here we aim to preserve the matrix entries from leakage, which is the key concern for recommendation systems as in Netﬂix’s privacy lawsuit [35].
Distributed matrix completion problem on plaintexts .
Assume that there are n 2 nodes with limited computing power and a cloud server with superior computing power.
The j -th node has an attribute vector M j 2 R n 1 , j = 1 , ..., n 2 , however, it is incomplete and the observed entries are indexed by a set ⌦ j ✓ { (1 , j ) , (2 , j ) , ..., ( n 1 , j ) } .
We assume that the true values of these n 2 vectors form a low-rank matrix M 2 R n 1 ⇥ n 2 with rank r ⌧ min( n 1 , n 2 ) , the ` 2 -norms of the attribute vectors is bounded by L , i.e., max 1  j  n 2 || M j || 2  L , and the observation set ⌦ =
S j =1 ,...,n 2 ⌦ j .
Nodes upload their incomplete vectors to a cloud server to carry out a matrix completion task
Find a matrix X [Equation]
[Equation]
where ⌦ ⇠ Uni ( m ) and r may be unknown.
Without loss of generality, we assume that n 1  nfrom now on.
Note that our formulation also includes the case [37] where a matrix is distributed into blocks and then is completed in parallel.
Homomorphic matrix completion problem on cyphertexts .
In cloud computing, the homomorphism property allows computations to be carried out on cyphertexts, generating an encrypted result which, when decrypted, matches the result of operations performed on the plaintext.
Following this paradigm, we deﬁne a homomorphic matrix completion problem that ensures data privacy.
As shown in Fig.
1, this novel framework consists of three main steps:
1) each node locally encrypts as P ⌦ j ( M j ) =
P ⌦ j ( g ( M j )) with its private keys, j = 1 , ..., n 2 , and uploads P ⌦ j ( M j ) to a cloud server that later forms an incomplete matrix P ⌦ ( M ) 2 R n 1 ⇥ n 2 ; 2) the cloud server solves the following matrix completion problem given P ⌦ ( M ) and sends back the recovered vector
[Equation]
where r = rank ( M ) may be slightly bigger than r due to by the encryption scheme g ( · ) .
3) each node locally decrypts its own vector using private keys, i.e., [Equation]
2.3 Notions of Privacy
We introduce a new variant of differential privacy for low-rank matrices.
2.3.1 Differential Privacy
Let D = { d 1 , ..., d n } be a dataset of n entries and T be a ﬁxed domain, where each entry d j 2 T encodes potentially sensitive information about node j . Let
A : T n ! O n be an algorithm that operates on dataset D and produces n outputs, one for each node j and from a set of possible output
O .
Let D  j denote the dataset D without the entry of the j -th node, and similarly A  j ( D ) denote the set of outputs without the output for the j -th node.
Let ( d j ; D  j ) denote the dataset obtained by adding a data entry
d j to the dataset D  j .
The ( ✏ ,  ) -differential privacy and joint ( ✏ ,  ) -differential privacy [21] are given in the following.
Deﬁnition 2.
( ( ✏ ,  ) -differential privacy [12]).
An algorithm A satisﬁes ( ✏ ,  ) -differential privacy if for any node j , any two possible values of data entry d j , d 0 j 2 T for node j , any tuple of data entries for all other nodes D  j 2 T n  1 , and any output set O ✓ O n , we have
[Equation]
Deﬁnition 3.
(Joint ( ✏ ,  ) -differential privacy [21]).
An algorithm A satisﬁes ( ✏ ,  ) -joint differential privacy if for any node j , any two possible values of data entry d j , d 0 j 2 T for node j , any tuple of data entries for all other nodes D  j 2 T n  1 , and any output set O ✓ O n  1 , we have
[Equation]
Intuitively, an algorithm A satisﬁes ( ✏ ,  ) -differential privacy if for any node j and dataset D , A ( D ) and D  j do not reveal “much" information about d j .
For low-rank matrices, [20] used a relaxed notion joint ( ✏ ,  ) -differential privacy : an algorithm A satisﬁes joint ( ✏ ,  ) -differential privacy if for any node j and dataset D , A  j ( D ) (the output for the other n  1 nodes) and D  j (data entries of the other n  1 nodes) do not reveal “much" information about d j .
Relaxing ( ✏ ,  ) -differential privacy to joint ( ✏ ,  ) -differential privacy is reasonable for the matrix completion problem since the j -th column for the j -th node can reveal a lot of information about d j .
2.3.2 Differential Privacy for Low-rank Matrix Completion
We would like to point out that joint ( ✏ ,  ) -differential privacy in Def.
3 can be further reﬁned.
For a low-rank matrix M , its column subspace S ( M ) is global information , which is shared across all n 2 nodes and can be easily inferred from A  j ( D ) and D  j .
Note that the differential privacy notion aims to protect individual information, rather than global information.
We adapt it to low-rank matrices by excluding the shared column subspace.
Low-rank matrices have linearly dependent columns, and this dependency is reﬂected in the fact that they share a common column subspace.
Formally, a rank- r matrix M = U ⌃ V > can be expressed as M = UC where U 2 R n 1 ⇥ r and C = ⌃ V > 2 R r ⇥ n 2 ; alternatively, a column can be expressed as M j = UC j , for j = 1 , ..., n 2 , where C j is the coefﬁcient vector (individual information) of the j -th node in the column subspace with basis U (global information).
The following subspace-aware joint ( ✏ ,  ) -differential privacy considers the coefﬁcient vectors C j for j = 1 , ..., n 2 , i.e., D in Def.
3 corresponds to the coefﬁcient matrix C 2 R r ⇥ n 2 .
Deﬁnition 4.
(Subspace-aware joint ( ✏ ,  ) -differential privacy).
Assume n 2 nodes’ data vector form a rank- r matrix M 2 R n 1 ⇥ n 2 with M = USV > = UC where U 2 R n 1 ⇥ r and C = SV >
R r ⇥ n 2 .
A matrix completion algorithm
A satisﬁes subspace-aware ( ✏ ,  ) -joint differential privacy if for any node j , any two possible coefﬁcient vectors C j , C {+
+} j 2 R r for node j , any tuple of coefﬁcient vectors for all other nodes C  j 2 R r ⇥ ( n 2  1) , and any output set O ✓ R r ⇥ n 2 that consists of estimated coefﬁcient vectors in a column subspace with basis U , we have
[Equation]
Novel Homomorphic Framework for Matrix Completion
We propose a homomorphic encryption-decryption scheme: a node performs local encryption and decryption, and uploads an encrypted vector to a server to perform the matrix completion computation.
3.1 Our Idea: Hiding a Low-rank Data Matrix in a Larger Subspace
To preserve the privacy of a low-rank data matrix M 2 R n 1 ⇥ n 2 with rank r , our idea is to hide
M (lies in an r -dimensional subspace) into a larger subspace of dimension r , such that r  r and r, r ⌧ n 1 .
A sound approach would be enlarging the original subspace of the data matrix (i.e., the plaintext) as follows: a cloud server generates a random matrix K 2 R n 1 ⇥ k as public keys, k ⌧ n 1 , and broadcasts K to all n 2 nodes; then, node j generates k random numbers as private keys R j 2 R k , and encrypts its vector M j 2 R n 1 as follows (a version with missing entries will be given in (12))
[Equation]
In the encryption scheme (11), M is added up with KR , resulting in a matrix M with rank r  r + k .
Since r ⌧ n 1 , M is also low-rank, it is possible to recover M from a subset of entries.
3.2 Proposed Homomorphic Encryption-Decryption Scheme
We propose a homomorphic encryption-decryption scheme that consists of the following steps, while the pseudocodes are summarzied in Alg. 1 and Alg. 2.
• First, in line 1 of Alg. 1, the cloud server generates a random matrix K 2 R n 1 ⇥ k as public keys, then broadcasts K to all n 2 nodes.
Second, in lines 1-5 of Alg. 2, after receiving K 2 R n 1 ⇥ k from the server (line 1 in Alg. 1), the j -th node locally carries out an encryption with k private keys (i.e., R j 2 R k ).
As shown in Fig.
2, the j -th node locally encrypts its incomplete vector P ⌦ j ( M j ) as follows
[Equation]
where R j i.i.d ⇠ N ( 0 ,  2 I k ) , P ⌦ j ( KR j ) means keeping the entries in ⌦ j and setting the entries in the complement set of ⌦ j to be zeros,
thus P ⌦ j
( M j ) has the same set of missing entries as P ⌦ j
( M j ) .
Note that R j is stored locally, which are private keys that will NOT be shared with any other node.
Then, each node uploads its encrypted vector P ⌦ j ( M j ) to the cloud server.
Third, in lines 2-5 of Alg. 1, after receiving all n 2 encrypted vectors P ⌦ j ( M j ) , j = 1 , ..., n 2 , the server forms an incomplete matrix M ⌦ with ⌦ =
S j =1 ,...,n 2 ⌦ j .
Then, the server carries out a matrix completion task in (7), and sends the recovered vector c [Equation]
j back to the j -th node, j = 1 , ..., n 2 .
• Finally, in lines 11-13 of Alg. 2,
using the locally stored private keys R j , and the public keys K , the j -th node decrypts its own vector, i.e.,
[Equation]
Homomorphism Property Holds at Price of More Samples
We prove that the homomorphism property holds for the proposed scheme, which guarantees exact recovery on cyphertexts at a cost of more samples.
The detailed proofs are given in Appx.
C.
Overview : Starting from a necessary and sufﬁcient condition in Lemma 1, we relax to a sufﬁcient condition in Lemma 3 for the homomorphism property to hold.
Then, we provide a homomorphic version of Rudelson Selection Estimation Theorem in Theorem 2 that guarantees Lemma 3 with high probability.
Therefore, we obtain a sample complexity for EXACT recovery in Theorem 3, where our interesting ﬁnding is that the homomorphism property holds at price of more samples .
4.1 Sufﬁcient Condition for Low-rank Matrix Completion
We start from a necessary and sufﬁcient condition for low-rank matrix completion.
Note that a similar necessary and sufﬁcient condition for sparse vector recovery is discussed in compressive sensing [3, 8, 46].
Here, we apply a similar argument to obtain Lemma 1.
We deﬁne a set of matrices with rank at most r and a rank-descent cone as follows
[Equation]
where M is the closure of the manifold of rank- r matrices.
Accordingly, for M , we have
[Equation]
Lemma 1.
( Necessary and sufﬁcient condition for low-rank matrix completion )
M is the unique optimal solution to (6) if and only if ⌦ ? \ D M ( M ) = { 0 } , where ⌦ ? denotes Ker ( P ⌦ ) .
Geometric interpretation : M is the unique optimal solution to problem (6) if and only if starting from M , the rank of M + D increases for all directions D 2 ⌦ ? , where D is nonzero.
Therefore, the homomorphism property of low-rank matrix completion in problem (7) holds if
[Equation]
Lemma 2.
([15, 7] (Theorem 6 . 1 ))
Let M = U ⌃ V > be the compact SVD of matrix M .
The tangent cone T M ( M ) of the set M at M is a linear subspace given by
[Equation]
and its complementary space is denoted by T ? .
Since the rank-descent cone is a subset of the tangent cone deﬁned in (16) ([17], Theorem 4 . 8 ),
D M ( M ) ✓ T, and D M ( M ) ✓ T , we relax (15) to the following sufﬁcient condition.
Lemma 3.
A sufﬁcient condition for the homomorphic property of matrix completion under the proposed scheme in Alg. 1 and Alg. 2 is ⌦ ? \ T = { 0 } .
Interpretation : if ⌦ ? \ T = { 0 } holds, then we know that M = M + KR is the unique optimal solution to problem (7) and M is the unique optimal solution to problem (6).
Since M = M + KR is a one-to-one mapping , a decryption scheme M  KR will return the desired true matrix M .
4.2 Homomorphic Version of Rudelson Selection Estimation Theorem
The Rudelson selection estimation theorem [39] investigates the number of random points needed to bring a convex body into a nearly isotropic position.
Such an approximate isometry property is fundamentally useful to characterize the number of entries needed to complete a low-rank matrix.
Deﬁnition 5.
(Coherence) Let U 2 R n ⇥ r be the r left singular vectors of M (corresponds to the column subspace S ( M ) ) and P U be the orthogonal projection onto U .
Then the coherence of U (or
S ( M ) , respectively) is deﬁned as
[Equation]
17) since ( U > U )  1 =
I and U is orthonormal.
The concept “coherence" measures the relationship between a low-dimensional space and the observation operator P ⌦ , namely the cosine (with a scaling factor nr ) of the principal angle between the low-dimensional space and a standard basis.
M is said to satisfy the standard incoherence condition with parameter µ 0 if
[Equation]
A small µ 0 ensures that the information of the row/column spaces of M is not too concentrated on a small number of rows/columns.
It characterizes the contribution of an entry in recovering M : a small µ 0 means that each entry provides approximated the same amount of information.
Theorem 1.
( Rudelson selection estimation theorem [3]) Assume that ⌦ ⇠ Ber ( p ) with p = ⇥ ( mn 1 n 2 ) , and M obeys the standard incoherence condition (18) with parameter µ 0 .
There is a constant C R such that for  > 1 , with probability  1  3 n   2 , we have
[Equation]
We derive the following homomorphic variant of the Rudelson selection estimation theorem [39] and will use it to guarantee Lemma 3.
Our new contribution here is to derive the conditions when the approximate isometry property will hold simultaneously for both cyphertexts and plaintexts.
Theorem 2.
( Homomorphic version of Rudelson selection estimation theorem ) Assume that ⌦ ⇠ Ber ( p ) with p = ⇥ ( mn 1 n 2 ) , M and M satisfy the standard incoherence condition (18) with parameter µ 0 and µ 0 , respectively.
Under the proposed scheme in Alg. 1 and Alg. 2, there are constants C R , C {+
+} R such that for  > 1 , with probability  1  3 n   2 , we have
[Equation]
which implies
[Equation]
Note that || p  1 P T P ⌦ P T  P T || < 1 implies that the sufﬁcient condition ⌦ ? \ T = { 0 } holds.
4.3 Sample Complexity for EXACT Recovery
Then, we prove Theorem 3 that the homormophism property holds for the proposed scheme in Alg.and Alg. 2, provided that there are sufﬁcient number of observations ( | ⌦ | is large enough).
Theorem 3.
For Alg. 1 and Alg. 2, with probability  1  3 n   2 , the homomorphism property holds if p  C 0 µ 0 r (  log n [Equation] n, where C 0 is a positive constant.
Next, we characterize the coherence change of µ 0 and provide the sample complexity for the EXACT recovery in Alg. 1 and Alg. 2.
Lemma 4.
The new coherence under the proposed scheme in Alg. 1 and Alg.
2 satisﬁes
[Equation]
) , with
[Equation]
Combining Theorem 3 and Lemma 4, we characterize the required number of entries.
Corollary 1.
For Alg. 1 and Alg. 2, with probability  1  6 n   2  cn  3 2 log n 2 , the homomorphism property holds if p  C 0 ( rµ 0 + C max( k, log n 2 ))
(  log n 2 ) n, where c, C 0 , C are positive constants.
Differential Privacy Property Holds
In this section, we show that the differential privacy holds for the proposed scheme in Alg. 1 and Alg.
It is well-known that one can achieve ( ✏ ,  ) -differential privacy by adding Gaussian noise.
Deﬁnition 6.
( Privacy loss as a random variable [12])
Considering a mechanism A on a pair of databases D, D 0 .
For an outcome o 2 O , the privacy loss on o is deﬁned as the logarithmic ratio between the probability to observe o on input D compared to that on input D 0 :
[Equation]
where P ( A ( D )
= o ) is a probability density over a continuous set O .
Two potential issues of the proposed scheme in Alg. 1 and Alg. 2 is the projection recovery and the rank value r may be unknown.
Namely, for a single-round encryption case, one can do a corresponding projection to obtain the real data.
Therefore, we execute the proposed scheme twice and introduce two parameters  1 and  2 :
• First-round encryption: the server randomly generats a matrix K 1 2 R n 1 ⇥ k and each node generates k random numbers R {+
+} j i.i.d ⇠ N ( 0 ,  2 1 I k ) .
Then, we have P ⌦ ( M ) =
P ⌦ ( M ) + P ⌦ ( K 1 R 1 ) .
• Second-round encryption: the server obtains the column space of M as K 2 2 R n 1 ⇥ r with r = r + k
and then each node generates r + k random numbers R {+
+}
j i.i.d ⇠ N ( 0 ,  22 I r + k ) .
Then we have P ⌦ ( M ) =
P ⌦ ( M ) + P ⌦ ( K 2 R 2 ) .
Theorem 4 states that the proposed scheme satisﬁes the subspace-aware joint ( ✏ ,  ) -differential privacy in Section 2.3.2.
The detailed proofs are given in Appx.
D, where the key is to quantify  under which the random variable privacy loss in (22) is bounded by ✏ , with probability at least 1   .
Theorem 4.
Let ✏ 2 (0 , 1) and c 2 > 2 ln(1 .
25 /  ) .
Assume the true data matrix M 2 R n 1 ⇥ nhas rank r and each column has bounded ` 2 -norm, i.e.,  = max 1  j  n 2 || M j || 2  L .
Let R 1 j ⇠
N ( 0 ,  2 1 I k ) with  1  2 cL p 2 ln(2 /  ) / ✏ and R 2 j ⇠ N ( 0 ,  2 2 I k + r ) with  2  2 cL p 2 ln(2 /  ) / ✏ , then the encryption and decryption scheme in Alg. 1 and Alg. 2, satisﬁes the subspace-aware joint ( ✏ ,  ) -differential privacy property.
A substantial improvement is: for the same level of privacy (the same parameters ✏ ,  in the above joint ( ✏ ,  ) -DP property), our algorithms are able to achieve EXACT recovery.
Note that by proving the homomorphism property and characterising the sample complexity, we reduce the error bound
O (p n 31 n 2 ) from [20] to ZERO since we have EXACT recovery.
Performance Evaluation
We evaluate the proposed scheme on synthetic data and real-world datasets using two matrix completion algorithms [41, 18], verifying the homomorphism property.
6.1 Experimental Settings
Datasets .
We experiment with synthetic data and real-world datasets.
The synthetic data is generated randomly according to the low-rank 1 , 000 ⇥ 1 , 000 matrix model and serves as well-controlled inputs for veriﬁcation.
The real-world datasets include two benchmark datasets for recommendation systems, namely the MovieLens10M (Top 400) 3 and Netﬂix (Top 400) datasets.
The MovieLens dataset contains ratings of 400 most rated movies made by approximately 7 , 000 users, and the Netﬂix dataset contains ratings of 400 most rated movies made by approximately 480 thousand users.
Matrix completion algorithms .
For the matrix completion on the server, we use nuclear-norm minimization (NN) and alternating minimization (AM) algorithms.
In Section 6.2, we compare both algorithms with their homomorphic versions.
In Section 6.3, on the real-world datasets, we also include the private Frank-Wolf (FW) algorithm [20] for comparison.
Performance metric .
We measure the recovery error via the relative square root error RSE =
|| c M  M || F || M || F .
All experiments are executed for ten times and we report the average results.
6.2 Results on Synthetic Data
We experiment with randomly generated low-rank matrices on NN and AM algorithms and their homomorphic versions HNN and HAM.
We vary the rank r of the generated matrix and the percentage of observed entries from 1 , 5 , to 95 .
As shown in Fig.
6.2, we observe two trends: 1) for a certain rank r , the success rate increases as the percentage of observed entries increases; and 2) for a certain percentage of observed entries, the success rate decreases as the rank r increases.
On the other hand, we ﬁnd that the HNN and HAM need slightly more observed entries to reach the success threshold, which veriﬁes Theorem 3 that the scheme guarantees exact recovery at a cost of more samples.
As an interpretation, the homomorphic version is to hide the plaintext matrix into a larger space, namely from rank r to rank r + k .
In this case, given that we set k = 10 for the experiments, we ﬁnd that the results of HNN and HAM can be obtained by shifting the results of their counterparts left one grid.
6.3 Results on MovieLens10M and Netﬂix Datasets
Fig.
4 shows the results on MovieLens10M and Neﬂix datasets.
For the newly introduced compared algorithm FW, we set the privacy parameter ✏ = 2 log(1 /  ) and  = 10  6 .
For the NN and AM algorithms, the setting is the same in Section 6.2.
First of all, we observe that the homomorphic algorithms can achieve signiﬁcantly lower recovery errors than the error of FW algorithm.
This points out the difference between the proposed scheme and existing strategies, in which we do not sacriﬁce the recovery error to improve the privacy.
On the other hand, we ﬁnd that the homomorphic algorithms can reach the same level of recovery error as the vanilla algorithms on plaintexts, but need more samples.
Such a performance is consistent with our theoretical proofs and our observations in Section 6.2.
Moreover, we analyze the impact of increasing the percentage of observed entries on three types of algorithms, as shown in Fig.
For
AM and FW algorithms, the recovery error decreases smoothly as the percentage increases (note that the y-axis decreasing in log ).
However, the NN algorithm demonstrates a signiﬁcant error drop as we increase the percentage of observed entries.
Conclusion and Future Work
This work studied the problem of privacy-preserving data completion in a distributed manner.
To address the privacy concern, we deﬁne the homomorphic matrix completion problem and propose a homomorphic encryption-decryption scheme.
Unlike existing works that preserve privacy by sacriﬁcing recovery accuracy, our work guarantees the EXACT recovery while making a tradeoff between privacy and the number of samples.
Then, we theoretically prove that the proposed scheme satisﬁes the homomorphism and differential privacy properties.
Experimentally, we show that the proposed scheme is compatible with two matrix completion algorithms, namely the nuclear norm minimization and alternating minimization, and verify the homomorphism property.
In the future, it would be interesting to extend this homomophic framework to the tensor completion problem [31, 32].
It would also be practically interesting to study federated learning application [24] and develop high-performance implementations for high-dimensional data analysis [50, 49, 36, 50].
Acknowledgement
Xiao-Yang Liu would like to thank Prof. John Wright (Department of Electrical Engineering at
Columbia University) for his insightful sharing.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] Not aware of foresee negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-=
-]