Homomorphic Matrix Completion
Abstract
In recommendation systems, global positioning, system identification, and mobile social networks, it is a fundamental routine that a server completes a low-rank matrix from an observed subset of its entries.
However, sending data to a cloud server raises up the data privacy concern due to eavesdropping attacks and the singlepoint failure problem, e.g., the Netflix prize contest was canceled after a privacy lawsuit.
In this paper, we propose a homomorphic matrix completion algorithm for privacy-preserving purpose.
First, we formulate a homomorphic matrix completion problem where a server performs matrix completion on cyphertexts, and propose an encryption scheme that is fast and easy to implement.
Secondly, we prove that the proposed scheme satisfies the homomorphism property that decrypting the recovered matrix on cyphertexts will obtain the target matrix (on plaintexts).
Thirdly, we prove that the proposed scheme satisfies an ( ϵ, δ ) -differential privacy property.
While with similar level of privacy guarantee, we reduce the best-known error bound O ( 10 (cid:112) n 31 n 2 ) to EXACT recovery at a price of more samples.
Finally, on synthetic data and real-world data, we show that both homomorphic nuclear-norm minimization and alternating minimization algorithms achieve accurate recoveries on cyphertexts, verifying the homomorphism property.
Introduction
The recurring low-rank matrix completion problem [4, 6, 22, 30, 13, 29, 46] concerns completing a low-rank matrix from a randomly observed subset of entries.
It has wide applications in recommendation systems (collaborative filtering)
[1, 47, 26], computer vision [2, 16, 27], global positioning [48, 32], sensory data analysis in Internet of Things [25, 34, 33], system identification, network data analysis [51, 11], mobile social networks [23, 38] etc.
Existing works [6, 9] have demonstrated a remarkable fact: if an n × n matrix with rank r ≪ n satisfies a certain incoherence property, then with high probability, it is possible to exactly recover the matrix from O ( nr poly log n ) ≪ n 2 entries using polynomial-time algorithms.
Intuitively, one needs roughly (2 nr − r 2 ) parameters [6] (this is the dimension of the tangent space to the manifold of rank- r matrices) to fix an n × n matrix of rank r , and the sampling randomness introduces a log n factor due to a coupon collector’s effect.
The information theoretical lower bound is Ω( nr log n ) [6], while the tightest known upper bound is
O ( nr log 2 n ) [9] with another log n factor from the Golfing scheme used by the recovery algorithms.
The low-rank matrix completion problem usually deals with large-scale matrices that involve extensive computations, while in mobile computing, smart devices usually outsource such a huge computation task to a cloud server.
However, revealing data to a server or releasing anonymized data raises up privacy concerns [23, 44, 42], e.g., the recommendation contest Netflix prize was canceled after privacy lawsuit [35].
There are two major obstructive factors: anonymization in data publishing is still vulnerable, and storing sensitive data on a cloud server may encounter the single-point of failure
(SPOF) problem, say hackers.
Existing works [20, 19, 10] address the privacy concern in various ways, e.g., a popular approach is to add noise to the data [20], therefore making a tradeoff between the recovery accuracy and the level of privacy.
In cloud computing and distributed systems, the homomorphism property [14, 45] allows computations to be carried out on cyphertexts, generating an encrypted result which, when decrypted, matches the result of operations performed on the corresponding plaintexts.
In this manner, homomorphic encryption securely chains together different services without sacrificing recovery accuracy, but may at a price of extensive computation .
There are several partially homomorphic crypto-systems, and also a number of fully homomorphic crypto-systems [14, 45].
In addition, the homomorphic property can also be used to create many other secure systems, for example secure voting systems, collision-resistant hash functions, private information retrieval schemes [43], etc.
In this paper, we integrate the large-scale distributed matrix completion task with a homomorphic encryption-decryption scheme, which guarantees the EXACT recovery and differential privacy at a price of more samples.
First, we define the homomorphic matrix completion problem that ensures data privacy by preserving a similarly homomorphism property between plaintexts and cyphertexts.
Specifically, we propose a homomorphic encryption-decryption scheme, in which each node performs local encryption and decryption, and uploads an encrypted incomplete vector to a server that carries out the matrix completion computation.
Then, we theoretically prove that the proposed scheme satisfies the homomorphism and differential privacy properties — reducing the best-known error bound O ( 10 (cid:112) n 31 n 2 ) [20] to EXACT recovery.
Finally, based on synthetic and real-world data, we show that the homomorphic nuclear-norm minimization and alternating minimization algorithms achieve accurate recoveries on both cyphertexts and plaintexts, verifying the homomorphism property.
Homomorphic Matrix Completion Problem
First, we formally define the homomorphic matrix completion problem.
Then, we introduce an notation of privacy by adapting the join ( ϵ, δ ) differential privacy, which is a subspace-aware variant.
2.1 Notations and Preliminaries
Notations : Let e i denote the i -th standard basis, I k denote the k × k identity matrix, and I denote the identity linear operator.
For matrix X , the ( i, j ) -th element is X ij or X ( i, j ) , the j -th column is X j , and the transpose is X ⊤ .
The concatenation of two matrices A ∈ R n 1 × n 2 and B ∈ R n 1 × n 3 with the same number of rows is denoted by [ A , B ] ∈ R n 1 × ( n 2 + n 3 ) .
By with high probability (w.h.p.)
we mean that with probability at least 1 − c 1 n − c 2 for some positive constants c 1 , c 2 .
Let N (0 , σ 2 ) denote a Gaussian distribution with mean 0 and standard deviation σ .
We use an overline to represent the encrypted version of a variable: variables before encryption are called plaintexts , e.g., X , while the encrypted variables are called cyphertexts , e.g., X .
Let M Ω ∈ R n 1 × n 2 denote the observed entries of a data matrix M ∈ R n 1 × n 2 , where Ω ⊆ { (1 , 1) , (1 , 2) , ..., ( n 1 , n 2 ) } indicates the observed entries.
We define a linear operator
P Ω : R n 1 × n 2 → R n 1 × n 2 to represent the observation model as follows
[Equation]
Assuming the true data matrix M is low-rank, i.e., rank ( M )
=
r ≪ min( n 1 , n 2 ) .
The (compact) singular value decomposition (SVD) is M = U Σ V ⊤ , where U ∈ R n 1 × r represents r left singular vectors (a basis of the column subspace), V ∈ R n 2 × r represents r right singular vectors (a basis of the row subspace), and Σ = diag ([ σ 1 , σ 1 , · · · , σ r ]) ∈ R r × r with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ r ≥ 0 .
The ℓ 2 -norm of a vector is || x
|| 2 , while the Frobenius norm and nuclear norm of M are || M || F = (cid:113)(cid:80) i,j | M ij | 2 and || M || ∗
=
(cid:80) ri =1 σ i , respectively.
The operator norm (spectral norm) of a matrix and a linear operator L are defined as follows
[Equation]
Definition 1.
( Column subspace and null space [28]) Let A ∈ R n 1 × n 2 .
The set S ( A ) = { b ∈ R n| b
=
Ax , x ∈ R n 2 } is the column space or range of A , and the set Ker ( A ) = { x ∈ R n| Ax = 0 } is the kernel or (right) null space of A .
The null space (kernel space) of operator P Ω is Ker ( P Ω ) =
{ Z ∈ R n 1 × n 2 | P Ω ( Z ) = 0 } , which is denoted as Ω ⊥ 2 .
Let Ω ∼ Uni ( m ) denote a set with m entries, which is sampled uniformly from all sets of m entries, and Ω ∼ Ber ( p ) denote a set with E | Ω | = m entries, where each entry is sampled independently according to a Bernoulli model with p = m/ ( n 1 n 2 ) .
Let P U and P V denote the orthogonal projections onto the column and row space of M , respectively
[Equation]
Define an orthogonal decomposition R n 1 × n 2 = T ⊕ T ⊥ , where T is the linear space spanned by matrices with the same column space or row space as M , and T ⊥ is its orthogonal complement that consists of matrices with row-space orthogonal to the row-space V and column-space orthogonal to the column-space U .
T can be expressed as follows
[Equation]
The orthogonal projection P T onto T and the orthogonal projection onto T ⊥ are as follows
[Equation]
2.2 Problem Formulation for Homomorphic Matrix Completion
We are interested in completing large-scale matrices and want to outsource this compute-intensive task from mobile devices to a cloud server.
Here we aim to preserve the matrix entries from leakage, which is the key concern for recommendation systems as in Netflix’s privacy lawsuit [35].
Distributed matrix completion problem on plaintexts .
Assume that there are n 2 nodes with limited computing power and a cloud server with superior computing power.
The j -th node has an attribute vector M j ∈ R n 1 , j = 1 , ..., n 2 , however, it is incomplete and the observed entries are indexed by a set Ω j ⊆ { (1 , j ) , (2 , j ) , ..., ( n 1 , j ) } .
We assume that the true values of these n 2 vectors form a low-rank matrix M ∈ R n 1 × n 2 with rank r ≪ min( n 1 , n 2 ) , the ℓ 2 -norms of the attribute vectors is bounded by L , i.e., max 1 ≤ j ≤ n 2 || M j || 2 ≤ L , and the observation set Ω
= (cid:83) j =1 ,...,n 2 Ω j .
Nodes upload their incomplete vectors to a cloud server to carry out a matrix completion task
Find a matrix X ∈ R n 1 × n 2 , s.t.
P Ω ( X ) =
P Ω ( M ) , rank ( X ) ≤ r,
[Equation]
6) where Ω ∼ Uni ( m ) and r may be unknown.
Without loss of generality, we assume that n 1 ≤ nfrom now on.
Note that our formulation also includes the case [37] where a matrix is distributed into blocks and then is completed in parallel.
Homomorphic matrix completion problem on cyphertexts .
In cloud computing, the homomorphism property allows computations to be carried out on cyphertexts, generating an encrypted result which, when decrypted, matches the result of operations performed on the plaintext.
Following this paradigm, we define a homomorphic matrix completion problem that ensures data privacy.
As shown in Fig.
1, this novel framework consists of three main steps:
1) each node locally encrypts as P Ω j ( M j ) =
P Ω j ( g ( M j )) with its private keys, j = 1 , ..., n 2 , and uploads P Ω j ( M j ) to a cloud server that later forms an incomplete matrix P Ω ( M ) ∈ R n 1 × n 2 ; 2) the cloud server solves the following matrix completion problem given P Ω ( M ) and sends back the recovered vector (cid:99) M j to the j -th node, j = 1 , ..., n 2 ,
Find a matrix
[Equation]
where r = rank ( M ) may be slightly bigger than r due to by the encryption scheme g ( · ) .
3) each node locally decrypts its own vector using private keys, i.e., (cid:99) M j = g − 1 ( (cid:99) M j ) , j = 1 , ..., n 2 .
2.3 Notions of Privacy
We introduce a new variant of differential privacy for low-rank matrices.
2.3.1 Differential Privacy
Let D = { d 1 , ..., d n } be a dataset of n entries and T be a fixed domain, where each entry d j ∈ T encodes potentially sensitive information about node j .
Let A : T n → O n be an algorithm that operates on dataset D and produces n outputs, one for each node j and from a set of possible output
O .
Let D − j denote the dataset D without the entry of the j -th node, and similarly A − j
( D ) denote the set of outputs without the output for the j -th node.
Let ( d j ; D − j ) denote the dataset obtained by adding a data entry
d j to the dataset D − j .
The ( ϵ, δ ) -differential privacy and joint ( ϵ, δ ) -differential privacy [21] are given in the following.
Definition 2.
( ( ϵ, δ ) -differential privacy [12]).
An algorithm A satisfies ( ϵ, δ ) -differential privacy if for any node j , any two possible values of data entry d j , d ′ j ∈ T for node j , any tuple of data entries for all other nodes D − j ∈ T n − 1 , and any output set O ⊆
O n , we have
[Equation]
Definition 3.
(Joint ( ϵ, δ ) -differential privacy [21]).
An algorithm A satisfies ( ϵ, δ ) -joint differential privacy if for any node j , any two possible values of data entry d j , d ′ j ∈ T for node j , any tuple of data entries for all other nodes D − j ∈ T n − 1 , and any output set O ⊆
O n − 1 , we have
[Equation]
Intuitively, an algorithm A satisfies ( ϵ, δ ) -differential privacy if for any node j and dataset D , A ( D ) and D − j do not reveal “much" information about d j .
For low-rank matrices, [20] used a relaxed notion joint ( ϵ, δ )
-differential privacy : an algorithm A satisfies joint ( ϵ, δ ) -differential privacy if for any node j and dataset D , A − j ( D ) (the output for the other n − 1 nodes) and D − j (data entries of the other n − 1 nodes) do not reveal “much" information about d j .
Relaxing ( ϵ, δ ) -differential privacy to joint ( ϵ, δ ) -differential privacy is reasonable for the matrix completion problem since the j -th column for the j -th node can reveal a lot of information about d j .
2.3.2 Differential Privacy for Low-rank Matrix Completion
We would like to point out that joint ( ϵ, δ ) -differential privacy in Def.
3 can be further refined.
For a low-rank matrix M , its column subspace S ( M ) is global information , which is shared across all n 2 nodes and can be easily inferred from A − j
( D ) and D − j .
Note that the differential privacy notion aims to protect individual information, rather than global information.
We adapt it to low-rank matrices by excluding the shared column subspace.
Low-rank matrices have linearly dependent columns, and this dependency is reflected in the fact that they share a common column subspace.
Formally, a rank- r matrix M = U Σ V ⊤ can be expressed as M = UC where U ∈ R n 1 × r and C = Σ V ⊤ ∈ R r × n 2 ; alternatively, a column can be expressed as M j = UC j , for j = 1 , ..., n 2 , where C j is the coefficient vector (individual information) of the j -th node in the column subspace with basis U (global information).
The following subspace-aware joint ( ϵ, δ ) -differential privacy considers the coefficient vectors C j for j = 1 , ..., n 2 , i.e., D in Def.
3 corresponds to the coefficient matrix C ∈ R r × n 2 .
Definition 4.
(Subspace-aware joint ( ϵ, δ ) -differential privacy).
Assume n 2 nodes’ data vector form a rank- r matrix M ∈ R n 1 × n 2 with M = USV ⊤ =
UC where U ∈ R n 1 × r and C =
SV ⊤ ∈ R r × n.
A matrix completion algorithm
A satisfies subspace-aware ( ϵ, δ ) -joint differential privacy if for any node j , any two possible coefficient vectors C j , C ′ j ∈ R r for node j , any tuple of coefficient vectors for all other nodes C − j ∈ R r × ( n 2 − 1) , and any output set O ⊆ R r × n 2 that consists of estimated coefficient vectors in a column subspace with basis U , we have
[Equation]
Novel Homomorphic Framework for Matrix Completion
We propose a homomorphic encryption-decryption scheme: a node performs local encryption and decryption, and uploads an encrypted vector to a server to perform the matrix completion computation.
3.1 Our Idea: Hiding a Low-rank Data Matrix in a Larger Subspace
To preserve the privacy of a low-rank data matrix M ∈ R n 1 × n 2 with rank r , our idea is to hide
M (lies in an r -dimensional subspace) into a larger subspace of dimension r , such that r ≥ r and r, r ≪ n 1 .
A sound approach would be enlarging the original subspace of the data matrix (i.e., the plaintext) as follows: a cloud server generates a random matrix K ∈ R n 1 × k as public keys, k ≪ n 1 , and broadcasts K to all n 2 nodes; then, node j generates k random numbers as private keys R j ∈ R k , and encrypts its vector M j ∈ R n 1 as follows (a version with missing entries will be given in (12))
[Equation]
In the encryption scheme (11), M is added up with KR , resulting in a matrix M with rank r ≤ r + k .
Since r ≪ n 1 , M is also low-rank, it is possible to recover M from a subset of entries.
3.2 Proposed Homomorphic Encryption-Decryption Scheme
We propose a homomorphic encryption-decryption scheme that consists of the following steps, while the pseudocodes are summarzied in Alg. 1 and Alg. 2.
• First, in line 1 of Alg. 1, the cloud server generates a random matrix K ∈ R n 1 × k as public keys, then broadcasts K to all n 2 nodes.
Second, in lines 1-5 of Alg. 2, after receiving K ∈ R n 1 × k from the server (line 1 in Alg. 1), the j -th node locally carries out an encryption with k private keys (i
e., R j ∈ R k ).
As shown in Fig.
2, the j -th node locally encrypts its incomplete vector P Ω j ( M j ) as follows
[Equation]
where
R j i.i.d ∼ N ( 0 , σ 2 I k
, P Ω
j (
KR
j ) means
keeping
the
entries in Ω
j
and setting the entries in the complement set of Ω
j
to be zeros
thus P Ω j
M
j )
has the same set of missing entries as P Ω j ( M j ) .
Note that R j is stored locally, which are private keys that will NOT be shared with any other node.
Then, each node uploads its encrypted vector P Ω j ( M j ) to the cloud server.
Third, in lines 2-5 of Alg. 1, after receiving all n 2 encrypted vectors P Ω j ( M j ) , j = 1 , ..., n 2 , the server forms an incomplete matrix M Ω with Ω =
(cid:83) j =1 ,...,n 2 Ω j .
Then, the server carries out a matrix completion task in (7), and sends the recovered vector (cid:99)
X j back to the j -th node, j = 1 , ..., n 2 .
• Finally, in lines 11-13 of
Alg. 2, using the locally stored private keys R j , and the public keys K , the j -th node decrypts its own vector, i.e., (cid:99)
X
j = g − 1 ( (cid:99)
X j ) = (cid:99)
X j − KR j , j = 1 , ..., n 2 .
Homomorphism Property Holds at Price of More Samples
We prove that the homomorphism property holds for the proposed scheme, which guarantees exact recovery on cyphertexts at a cost of more samples.
The detailed proofs are given in Appx.
C.
Overview : Starting from a necessary and sufficient condition in Lemma 1, we relax to a sufficient condition in Lemma 3 for the homomorphism property to hold.
Then, we provide a homomorphic version of Rudelson Selection Estimation Theorem in Theorem 2 that guarantees Lemma 3 with high probability.
Therefore, we obtain a sample complexity for EXACT recovery in Theorem 3, where our interesting finding is that the homomorphism property holds at price of more samples .
4.1 Sufficient Condition for Low-rank Matrix Completion
We start from a necessary and sufficient condition for low-rank matrix completion.
Note that a similar necessary and sufficient condition for sparse vector recovery is discussed in compressive sensing [3, 8, 46].
Here, we apply a similar argument to obtain Lemma 1.
We define a set of matrices with rank at most r and a rank-descent cone as follows (cid:40)
[Equation]
where M is the closure of the manifold of rank- r matrices.
Accordingly, for M , we have
[Equation]
Lemma 1.
( Necessary and sufficient condition for low-rank matrix completion )
M is the unique optimal solution to (6) if and only if Ω ⊥ ∩ D M ( M ) = { 0 } , where Ω ⊥ denotes Ker ( P Ω ) .
Geometric interpretation : M is the unique optimal solution to problem (6) if and only if starting from M , the rank of M + D increases for all directions D ∈ Ω ⊥ , where D is nonzero.
Therefore, the homomorphism property of low-rank matrix completion in problem (7) holds if
[Equation]
Lemma 2.
([15, 7] (Theorem 6 . 1 ))
Let M = U Σ V ⊤ be the compact SVD of matrix M .
The tangent cone T M ( M ) of the set M at M is a linear subspace given by
[Equation]
and its complementary space is denoted by T ⊥ .
Since the rank-descent cone is a subset of the tangent cone defined in (16) ([17], Theorem 4 . 8 ),
D M ( M ) ⊆ T, and D M ( M ) ⊆ T , we relax (15) to the following sufficient condition.
Lemma 3.
A sufficient condition for the homomorphic property of matrix completion under the proposed scheme in Alg. 1 and Alg. 2 is Ω ⊥ ∩ T = { 0 } .
Interpretation : if Ω ⊥ ∩ T = { 0 } holds, then we know that M = M + KR is the unique optimal solution to problem (7) and M is the unique optimal solution to problem (6).
Since M = M + KR is a one-to-one mapping , a decryption scheme M − KR will return the desired true matrix M .
4.2 Homomorphic Version of Rudelson Selection Estimation Theorem
The Rudelson selection estimation theorem [39] investigates the number of random points needed to bring a convex body into a nearly isotropic position.
Such an approximate isometry property is fundamentally useful to characterize the number of entries needed to complete a low-rank matrix.
Definition 5.
(Coherence) Let U ∈ R n × r be the r left singular vectors of M (corresponds to the column subspace S ( M ) ) and P U be the orthogonal projection onto U .
Then the coherence of U (or
S ( M ) , respectively) is defined as
[Equation]
since ( U ⊤ U ) − 1 =
I and U is orthonormal.
The concept “coherence" measures the relationship between a low-dimensional space and the observation operator P Ω , namely the cosine (with a scaling factor nr ) of the principal angle between the low-dimensional space and a standard basis.
M is said to satisfy the standard incoherence condition with parameter µ 0 if
[Equation]
A small µ 0 ensures that the information of the row/column spaces of M is not too concentrated on a small number of rows/columns.
It characterizes the contribution of an entry in recovering M : a small µ 0 means that each entry provides approximated the same amount of information.
Theorem 1.
( Rudelson selection estimation theorem [3]) Assume that Ω ∼ Ber ( p ) with p =
Θ( mn 1 n 2 ) , and M obeys the standard incoherence condition (18) with parameter µ 0 .
There is a constant C R such that for β > 1 , with probability ≥ 1 − 3 n − β 2 , we have
[Equation] [Equation]
We derive the following homomorphic variant of the Rudelson selection estimation theorem [39] and will use it to guarantee Lemma 3.
Our new contribution here is to derive the conditions when the approximate isometry property will hold simultaneously for both cyphertexts and plaintexts.
Theorem 2.
( Homomorphic version of Rudelson selection estimation theorem ) Assume that Ω ∼ Ber ( p ) with p =
Θ( mn 1 n 2 ) , M and M satisfy the standard incoherence condition (18) with parameter µ 0 and µ 0 , respectively.
Under the proposed scheme in Alg. 1 and Alg. 2, there are constants C R , C ′ R such that for β > 1 , with probability ≥ 1 − 3 n − β 2 , we have
[Equation]
, which implies
[Equation]
Note that || p − 1 P T P Ω P T − P T || < 1 implies that the sufficient condition Ω ⊥ ∩ T = { 0 } holds.
4.3 Sample Complexity for EXACT Recovery
Then, we prove Theorem 3 that the homormophism property holds for the proposed scheme in Alg.and Alg. 2, provided that there are sufficient number of observations ( | Ω | is large enough).
Theorem 3.
For Alg. 1 and Alg. 2, with probability ≥ 1 − 3 n − β 2 , the homomorphism property holds if p ≥ C 0 µ 0 r ( β log n 2 ) n, where C 0 is a positive constant.
Next, we characterize the coherence change of µ 0 and provide the sample complexity for the EXACT recovery in Alg. 1 and Alg. 2.
Lemma 4.
The new coherence under the proposed scheme in Alg. 1 and Alg.
2 satisfies
[Equation]
Combining Theorem 3 and Lemma 4, we characterize the required number of entries.
Corollary 1.
For Alg. 1 and Alg. 2, with probability ≥ 1 − 6 n − β 2 − cn − 3 2 log n 2 , the homomorphism property holds if p ≥ C 0 ( rµ 0 + C max( k, log n 2 ))
( β log n 2 ) n, where c, C 0 , C are positive constants.
Differential Privacy Property Holds
In this section, we show that the differential privacy holds for the proposed scheme in Alg. 1 and Alg.
It is well-known that one can achieve ( ϵ, δ ) -differential privacy by adding Gaussian noise.
Definition 6.
( Privacy loss as a random variable [12])
Considering a mechanism A on a pair of databases D, D ′ .
For an outcome o ∈ O , the privacy loss on o is defined as the logarithmic ratio between the probability to observe o on input D compared to that on input D ′ :
[Equation]
where P ( A ( D )
= o ) is a probability density over a continuous set O .
Two potential issues of the proposed scheme in Alg. 1 and Alg. 2 is the projection recovery and the rank value r may be unknown.
Namely, one can do a corresponding projection to obtain the real data after a single encryption.
Therefore, we execute the proposed scheme twice and introduce two parameters σ 1 and σ 2 :
• First-round encryption: the server randomly generats a matrix K 1 ∈ R n 1 × k and each node generates k random numbers R 1 j i.i.d ∼ N ( 0 , σ 2 1 I k ) .
Then, we have P Ω ( M ) =
P Ω ( M ) + P Ω ( K 1 R 1 ) .
• Second-round encryption: the server obtains the column space of M as K 2 ∈ R n 1 × r with r = r + k
and then each node generates r + k random numbers R 2 j i.i.d ∼ N
( 0 , σ 22 I r + k ) .
Then we have P Ω ( M ) =
P Ω ( M ) + P Ω ( K 2 R 2 ) .
[Figure]
Theorem 4 states that the proposed scheme satisfies the subspace-aware joint ( ϵ, δ ) -differential privacy in Section 2.3.2.
The detailed proofs are given in Appx.
D, where the key is to quantify σ under which the random variable privacy loss in (22) is bounded by ϵ , with probability at least 1 − δ .
Theorem 4.
Let ϵ ∈
(0 , 1) and c 2 > 2 ln(1 .
25 /δ ) .
Assume the true data matrix M ∈ R n 1 × nhas rank r and each column has bounded ℓ 2 -norm, i.e., ∆ = max 1 ≤ j ≤ n 2 || M j || 2 ≤ L .
Let R 1 j ∼
N ( 0 , σ 2 1 I k ) with σ 1 ≥ 2 cL (cid:112) 2 ln(2 /δ )
/ϵ
and R 2 j ∼ N ( 0 , σ 2 2 I k + r ) with σ 2 ≥ 2 cL (cid:112) 2 ln(2 /δ )
/ϵ , then the encryption and decryption scheme in Alg.
1 and Alg. 2, satisfies the subspace-aware joint ( ϵ, δ ) -differential privacy property.
A substantial improvement is: for the same level of privacy (the same ϵ , δ parameter in the above joint ( ϵ, δ ) -DP property), our algorithms are able to achieve EXACT recovery.
Note that by proving the homomorphism property and characterising the sample complexity, we reduce the error bound
O ( 10 (cid:112) n 31 n 2 ) from [20] to ZERO since we have EXACT recovery.
Performance Evaluation
We evaluate the proposed scheme on synthetic data and real-world datasets using two matrix completion algorithms [41, 18], verifying the homomorphism property of the proposed scheme.
6.1 Experimental Settings
Datasets .
We experiment with synthetic data and real-world datasets.
The synthetic data is generated randomly according to the low-rank 1 , 000 × 1 , 000 matrix model and serves as well-controlled inputs for verification.
The real-world datasets include two benchmark datasets for recommendation systems, namely the MovieLens10M (Top 400) 3 and Netflix (Top 400) datasets.
The MovieLens dataset contains ratings of 400 most rated movies made by approximately 7 , 000 users, and the Netflix dataset contains ratings of 400 most rated movies made by approximately 480 thousand users.
Matrix completion algorithms .
For the matrix completion on the server, we use nuclear-norm minimization (NN) and alternating minimization (AM) algorithms.
In Section 6.2, we compare both algorithms with their homomorphic versions.
In Section 6.3, on the real-world datasets, we also include the private Frank-Wolf (FW) algorithm [20] for comparison.
Performance metric .
We measure the recovery error via the relative square root error RSE =
|| (cid:99) M − M || F || M || F .
All experiments are executed for ten times and we report the average results.
6.2 Results on Synthetic Data
We experiment with randomly generated low-rank matrices on NN and AM algorithms and their homomorphic versions HNN and HAM.
We vary the rank r of the generated matrix and the percentage of observed entries from 1 , 5 , to 95 .
As shown in Fig.
6.2, we observe two trends: 1) for a certain rank r , the success rate increases as the percentage of observed entries increases; and 2) for a certain percentage of observed entries, the success rate decreases as the rank r increases.
On the other hand, we find that the HNN and HAM need slightly more observed entries to reach the success threshold,
[Figure]
which verifies Theorem 3 that the scheme guarantees exact recovery at a cost of more samples.
As an interpretation, the homomorphic version is to hide the plaintext matrix into a larger space, namely from rank r to rank r + k .
In this case, given that we set k = 10 for the experiments, we find that the results of HNN and HAM can be obtained by shifting the results of their counterparts left one grid.
6.3 Results on MovieLens10M and Netflix Datasets
Fig.
4 shows the results on MovieLens10M and Neflix datasets.
For the newly introduced compared algorithm FW, we set the privacy parameter ϵ =
2 log(1 /δ ) and δ = 10 − 6 .
For the NN and AM algorithms, the setting is the same in Section 6.2.
First of all, we observe that the homomorphic algorithms can achieve significantly lower recovery errors than the error of FW algorithm.
This points out the difference between the proposed scheme and existing strategies, in which we do not sacrifice the recovery error to improve the privacy.
On the other hand, we find that the homomorphic algorithms can reach the same level of recovery error as the vanilla algorithms on plaintexts, but need more samples.
Such a performance is consistent with our theoretical proofs and our observations in Section 6.2.
Moreover, we analyze the impact of increasing the percentage of observed entries on three types of algorithms, as shown in Fig.
For
AM and FW algorithms, the recovery error decreases smoothly as the percentage increases (note that the y-axis decreasing in log ).
However, the NN algorithm demonstrates a significant error drop as we increase the percentage of observed entries.
Conclusion and Future Work
This work studied the problem of privacy-preserving data completion in a distributed manner.
To address the privacy concern, we define the homomorphic matrix completion problem and propose a homomorphic encryption-decryption scheme.
Unlike existing works that preserve privacy by sacrificing recovery accuracy, our work guarantees the EXACT recovery while making a tradeoff between privacy and the number of samples.
Then, we theoretically prove that the proposed scheme satisfies the homomorphism and differential privacy properties.
Experimentally, we show that the proposed scheme is compatible with two matrix completion algorithms, namely the nuclear norm minimization and alternating minimization, and verify the homomorphism property.
In the future, it would be interesting to extend this homomophic framework to the tensor completion problem [31, 32].
It would also be practically interesting to study federated learning application [24] and develop high-performance implementations for high-dimensional data analysis [50, 49, 36, 50].
Acknowledgement
Xiao-Yang Liu would like to thank Prof. John Wright (Department of Electrical Engineering at
Columbia University) for his insightful sharing.
Broader Impact Statement
This paper is within the area of private machine learning, which calls for privacy-preserving data completion by proposing a homomorphic encryption-decryption scheme.
Due to the wide application areas of the matrix completion problem, this work may have broad practical impact in recommendation systems, global positioning, system identification and mobile social networks, etc.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] Not aware of foresee negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Preliminaries
Notations : Lower case bold letters denote vectors, while capital bold face letters denote matrices, e.g., x ∈ R n and X ∈ R n 1 × n 2 .
Let e i denote the i -th standard basis.
For matrix X , its ( i, j ) -th element is X ij or X ( i, j ) and its j -th column is X j .
The identity matrix I is a square matrix, sometimes we use I n to specify its size n × n , and the identity linear operator is denoted by I .
The transpose of a vector/matrix is indicated by a superscript ⊤ , e.g., x ⊤ and X ⊤ .
The concatenation of two matrices
A ∈ R n 1 × n 2 and B ∈ R n 1 × n 3 is denoted by [ A , B ] ∈ R n 1 × ( n 2 + n 3 ) .
By with high probability (w.h.p.)
we mean that with probability at least 1 − c 1 n − c 2 for some constants c 1 , c 2 > 0 .
Denote the Gaussian distribution by N (0 , σ 2 ) , with mean 0 and standard deviation σ .
We use an overline to represent the encrypted version of a variable.
Variables before encryption are called plaintexts , e.g.,
X , while the encrypted variables are called cyphertexts , e.g., X .
We use an overline to represent the encrypted version of a variable on a cloud server.
Variables before encryption are called plaintexts , e.g., X , while the corresponding encrypted variables are called cyphertexts , e.g., X .
Let set Ω ⊆ { (1 , 1) , (1 , 2) , ..., ( n 1 , n 2 ) } index the observed entries.
We represent the observed entries as M Ω , while for analysis purpose we define a corresponding linear operator P Ω : R n 1 × n 2 → R n 1 × n 2 to represent the observation model as follows
[Equation]
We assume the true matrix M is low-rank, i.e., rank ( M ) =
r ≪ min( n 1 , n 2 ) .
The singular value decomposition (SVD) is M = USV ⊤ , where U ∈ R n 1 × r denotes the r left singular vectors (corresponding to the column subspace), V ∈ R n 2 × r denotes the r right singular vectors (corresponding to the row subspace), and S = diag ( σ i )
∈ R r × r where σ i is the i -th largest singular value and σ 1 ≥ σ 2 ≥ · · · ≥ σ r ≥ 0 .
The nuclear norm of M is || M || ∗ =
r (cid:80) i =
1 σ i .
The ℓ 2 -norm of a vector is || x
|| 2 , while the Frobenius norm of a matrix is || M || F = (cid:113)(cid:80) i,j
| M ij | 2 .
The operator norm (spectral norm) of a matrix is equal to its first singular value, i.e., for M ∈ R n 1 × n 2 ,
[Equation]
24) while more generally, the operator norm of a linear operator L is
[Equation] [Equation]
Definition 7.
( Column subspace and null space [28]) Let A ∈ R n 1 × n 2 .
The set
[Equation]
(26) is the column space or range of A , and the set
Ker
[Equation]
(27) is the kernel or (right) null space of A .
Definition 8.
( Row subspace and null space [28]) Let A ∈ R n 1 × n 2 .
The set
[Equation]
(28) is the row space of A , and the set
[Equation] [Equation]
is the left null space of A .
According to Definition 7, the kernel space of the linear operator P Ω is Ker ( P Ω ) =
{ Z ∈ R n 1 × n 2 | P Ω ( Z ) = 0 } , which is denoted as Ω ⊥ for simplicity.
We adopt the notation Ω ⊥ since
Ker ( P Ω ) equals to a space on the complement set of Ω .
Let Ω ∼ Uni ( m ) denote a set with m entries, which is sampled uniformly from all sets of m entries, and Ω ∼ Ber ( p ) denote a set with E | Ω | = m entries, each sampled independently according to the
Bernoulli model in Definition 9.
Definition 9.
( Bernoulli model [4]).
Let { γ ij , i = 1 , .., n 1 , j = 1 , ..., n 2 } be a sequence of independent identically distributed (i.i.d.)
binary Bernoulli random variables with
[Equation]
Correspondingly, we define a set Ω
= { ( i, j ) | γ ij = 1 } with E | Ω | = m.
Definition 10.
( Random orthogonal model [4])
The set { u j ∈ R n , j = 1 , ..., r } is selected uniformly at random among all sets of r orthonormal vectors.
B More about Homomorphic Matrix Completion
Now, we define the homomorphic matrix completion problem , and further specify the homomorphism property in the context of matrix completion. Definition 11. ( Homomorphic matrix completion (HMC) ) Assume the true matrix M is low-rank with rank ( M ) =
r ≪ min( n 1 , n 2 ) , the observed entries P Ω ( M ) ∈ R n 1 × n 2 , Ω ⊆ [ n 1 ] × [ n 2 ] , and let the matrix completion problem (6) be formally expressed as (cid:99) X = f
( P Ω ( M ) , r ) where f ( · ) is a mapping 4 .
Denote the pair of encryption and decryption schemes 5 by { g ( · ) , g − 1 ( · ) } , the homomorphic matrix completion problem estimates (cid:99) X
= g − 1 ( (cid:99) X ) , where (cid:99) X
= f ( P Ω ( g ( M )) , r ) denotes the following problem
Find a
[Equation]
where P Ω ( M ) =
P Ω ( g ( M )) , and r = rank ( M ) may be slightly bigger than r (this change is made by the encryption scheme g ( · ) .
Definition 12.
( The homomorphism property )
With the same assumptions in Def. 11, we say that the homomorphism property holds for the HMC problem if there exists a pair of encryption and decryption schemes { g ( · ) , g − 1 ( · ) } , such that f ( P Ω ( M ) , r ) =
g − 1 ( f ( g ( M ) , r )) , and both f ( P Ω ( M ) , r ) and f ( g ( M ) , r ) are functions 6 .
Remark 1.
Our HMC problem shares a similar paradigm with the fully homomorphic encryption (FHE) scheme [14, 45], but they are essentially different.
The FHE scheme provides an one-to-one mapping between plaintexts and cyphertexts for both the addition and multiplication operations, i.e.,
[Equation]
where g ( · ) and g − 1 ( · ) denote the encryption and decryption schemes, respectively.
However, throughout the paper, the HMC problem concerns the equivalence between (cid:99) X and g − 1 ( (cid:99) X ) , where the recovery of M and M are essentially different from the addition and multiplication operations.
B.1 Using the Bernoulli Model to Approximate the Uniform Model
If one can bound the probability of failure for the Bernoulli model, then the failure probability for the uniform model will be no more than twice as large, appeared in Section II.C of [5].
Following a similar argument, we formally provide the following two lemmas.
Lemma 5.
[5] The probability that the matrix completion problem (6) fails when the set of observed entries is sampled uniformly from the collection of set of size m (i.e., Ω ∼ Uni ( m ) ) is less than or equal to twice the probability that (6) fails when m entries are sampled according to the Bernoulli model in Def.
9 (i.e., Ω ′ ∼ Bern ( p ) with p =
mn 1 n 2 ).
Proof.
The proof follows the argument in in Section II.C of [5].
Let Ω of size m be drawn using the uniform model, and Ω ′ be drawn from the Bernoulli model Ω ′ ∼ Bern ( p ) with p =
mn 1 n 2 ).
We have
[Equation]
where Ω k denote a set of entries of size k sampled uniformly at random with | Ω k | = k .
We observe the following two inequalities:
• P ( Failure (Ω k )) is a nonincreasing function of k , i.e.,
[Equation]
• Since m = p · n 1 n 2 is an integer, it is the median of | Ω ′ |
[5], i
[Equation]
e.
[Equation]
Then, we have
[Equation]
Lemma 6.
[5] Let n be the number of Bernoulli trials and suppose that Ω ∼ Ber ( m/n ) .
Then, with probability at least 1 − n − 10 , | Ω | =
Θ( m ) , provided that m ≥ c log n for a constant c .
C Proofs for SectionC.1 Proof of Lemma
Proof.
Problem (6) is equivalent to the following problem
[Equation]
where D ∈ Ω ⊥ = { D ∈ R n 1 × n 2 : P Ω ( M + D ) =
P Ω ( M ) } (i.e., P Ω ( D ) = 0 )) is equivalent to the constraint P Ω ( X ) =
P Ω ( M ) in (6).
Then, M is the unique optimal solution to (6) is equivalent to D = 0 .
We know that rank ( M +
D ) ≤ r is equivalent to D ∈ D M ( M ) according to (13).
Therefore, D = 0 is equivalent to Ω ⊥ ∩ D M ( M ) = { 0 } , namely, set D M ( M ) and set Ω ⊥ intersect uniquely at 0 .
The lemma is proved.
Interested readers can check a similar proof for Lemma 3.34 (page 116) of [46].
C.2 Proof of Lemma
We provide a geometric interpretation of the homomorphism property in the context of matrix completion problems.
• The homomorphism property under the scheme in Alg.
1 and Alg.
2 can be mathematically interpreted as: M is the unique optimal solution to problem (6), while simultaneously M + KR is the unique optimal solution to problem (7).
According to Lemma 1, the necessary and sufficient condition would be Ω ⊥ ∩ D M ( M ) = { 0 } and Ω ⊥ ∩ D M ( M ) = { 0 } .
• Since it always holds
that D M ( M ) ⊆ T and D M ( M )
⊆
T according to Lemma 7 and Lemma 8, a sufficient condition for the homomorphic property is that Ω ⊥ ∩ T = { 0 } and Ω ⊥ ∩ T = { 0 } hold simultaneously.
The scheme in Alg. 1 and Alg.
2 satisfies that T ⊆ T , thus Ω ⊥ ∩ T = { 0 } implies Ω ⊥ ∩ T = { 0 } .
• If the sampling probability p is large enough, with high probability, it is guaranteed that Ω ⊥ ∩ T = { 0 } , which also implies Ω ⊥ ∩ T = { 0 } since T ⊆ T .
First, according to Lemma 7 and Lemma 8, we have
[Equation]
Lemma 7 states that the tangent cone of the set M evaluated at M has a closed-form expression, namely, it is just the space T in (4).
Lemma 8 claims that the tangent cone of the set M evaluated at
M is slightly larger than the cone cone ( M − { M } ) , where { M } denotes a set.
Definition 13.
A matrix Ξ ∈ R n 1 × n 2 is tangent to M at M if there exists a sequence { M n } contained in M and converging to M , and a sequence { a n } of nonnegative numbers, such that the sequence a n ( M n − M ) converges to Ξ .
Then, the tangent cone of the set M at point M , denoted by T M ( M ) , is formally defined as follows
[Equation]
Lemma 7.
[7] (Theorem. 6 . 1 ))
Let M = U Σ V ⊤ be the skinny SVD (or reduced SVD) of matrix
M .
The tangent cone T M ( M ) of the set M = { X ∈ R n 1 × n 2 : rank ( X ) ≤ r } at M is a linear subspace given by
[Equation]
where the complementary space is denoted by T ⊥ .
Lemma 8.
([17], Theorem 4 . 8 )
Let M be a non-empty subset of a real normed space.
If M is star-shaped w.r.t.
some M ∈ M , i.e., t ( M − { M } ) ⊆
M − { M } for all t ∈ [0 , 1] , then if follows
D M ( M ) = cone ( M − { M } ) ⊆ T M ( M ) .
Second, we point out that the scheme in Alg. 1 and Alg.
2 satisfies T ⊆ T .
It leads to our argument that Ω ⊥ ∩ T = { 0 } implies Ω ⊥ ∩ T = { 0 } , since 0 belongs to Ω ⊥ , T and T by definition.
This condition can be interpreted as follows: with the same observation set Ω , recovering M = M + KR (a matrix in a larger subspace) is more difficult than recovering M .
This is formulated in Lemma 3.
C.3 Proof of Theorem
Proof.
Applying Theorem 1 for cyphertext M , then we obtain that with probability at least 1 − 3 n − β 2 ,
[Equation]
provided that ϵ ′ < 1 .
Then, under the proposed scheme in Alg. 1 and Alg. 2, we show that ϵ ′ < 1 implies ϵ < 1 , and we have M = M + KR and r > r .
Combining the coherence change in Section 4.3, we can easily know that µ 0 < µ 0 .
Since r < r = rank ( M ) , we obtain that µ 0 r < µr .
Therefore, there are constants C R , C ′ R such that for β > 1 , ϵ ′ < 1 implies ϵ < 1 .
The proof is completed.
C.4 Proof of Theorem
Proof.
Recall that Lemma 3 states that a sufficient condition for the homomorphic property is Ω ⊥ ∩ T = { 0 } .
We use Theorem 2 to derive a sufficient condition p ≥ C 0 µ 0 r ( β log n 2 ) n.
According to Theorem 2, if Ω ∼ Ber ( p ) , with probability at least 1 − 3 n − β 2 , we have
[Equation]
(44) provided that p ≥ C 0 µ
[Equation]
r (
[Equation]
n
[Equation]
n.
Note that I = P Ω + P Ω ⊥ , where P Ω ⊥ denote the projection onto Ω ⊥ , then
P
[Equation]
45) then by the triangle inequality of the operator norm, we obtain that
[Equation]
where we have ||P T
||
= 1 according to Eq.
24, since for X ∈ T , we have ||P T ( X ) ||
= X .
Since ||P Ω ⊥ P T || 2 = ||P T P Ω ⊥ P Ω ⊥ P T || ≤ ||P T P Ω ⊥ P T || , then ||P Ω ⊥ P T || < √ 1 − p + ϵ ′ p < 1 .
Note that ||P Ω ⊥ P T || < 1 implies that Ω ⊥ ∩ T = { 0 } , which holds by contradiction: if there exists a nonzero matrix X ∈ Ω ⊥ ∩ T , then ||P Ω ⊥ P T ( X ) ||
=
X and ||P Ω ⊥ P T ||
= 1 .
We have shown that if p ≥ C 0 µ 0 r ( β log n 2 ) n, with probability at least 1 − 6 n − β 2 , it holds for Ω ⊥ ∩ T = { 0 } .
Combining Lemma 3, the proof is completed.
C.5 Proof of Lemma
We provide key lemmas of coherence change for the union of two subspaces, which will be used to characterize the coherence change under the scheme in Alg. 1 and Alg. 2.
The concept “coherence" measures the relationship between a low-dimensional space and the observation operator P Ω , namely the cosine (with a scaling factor nr ) of the principal angle between the low-dimensional space and a standard basis.
Definition 14.
(Coherence [6]) Let U ∈ R n × r be the r left singular vectors of M (corresponds to the column subspace S ( M ) ) and P U be the orthogonal projection onto U .
Then the coherence of U (or S ( M ) , respectively) is defined as
[Equation]
(47) since ( U ⊤ U ) − 1 =
I and U is orthonormal.
Remark 2.
Note that the left singular vectors U (or right singular vectors V , respectively) are not uniquely defined and all candidates are unitary transform to each other, i.e., M = ( UW ) S ( V W )
⊤ =
USV ⊤ where W ∈ R r × r is an orthonormal matrix, i.e., W W ⊤ =
W ⊤ W =
I .
The coherence of the column subspace µ ( S ( M ))
(or the row subspace µ ( S ( M ⊤ )) , respectively) is uniquely defined, since the coherence is invariant under unitary transformation.
Specifically, given two orthonormal matrices W ∈ R r × r and U ∈ R n 1 × r , then µ ( UW ) = µ ( U ) , since || P ( UW ) · e i || 2 = ||
UW (( UW ) ⊤ UW ) − 1 ( UW )
⊤ e
i
||
2 =
||
U ⊤ e
i || 2 .
Therefore, we will use µ ( U ) instead of µ ( S ( M )) .
Lemma 9.
( Incoherence change under the union operation ) Let r = r 1 + r 2 , the matrix U
=
[ U 1 , U 2 ] ∈ R n × ( r 1 + r 2 ) be the concatenation of two non-overlapping orthonormal matrices U 1 ∈ R n × rand U 2 ∈ R n × r 2 .
Then, we have µ ( U ) ≤ r 1 r µ ( U 1 ) + r 2 r µ ( U 2 ) , and it applies to the union of two subspaces,
i.e., µ ( S ( U )) ≤ r 1 r µ ( S ( U 1 ))
+ r 2 r µ ( S ( U 2 )) .
Proof.
From Def.
14, we have
[Equation]
where µ ( U 1 ) and µ ( U 2 ) are given in Def.
For the union of subspaces, one can easily verify the incoherence change according to Remark 2.
Definition 15.
( Random orthogonal model [6])
The set { u j ∈ R n , j = 1 , ..., r } is selected uniformly at random among all sets of r orthonormal vectors.
Lemma 10.
( Random subspaces span incoherence subspaces [6]) Set r ′ = max( r, log n ) .
There exist positive constants C and c , the random orthogonal model in Def. 15 obeys
[Equation]
, which yields
[Equation]
with probability ≥ 1 − cn − 3 log n .
Here we formally prove Lemma 4.
Proof.
The scheme in Alg. 1 and Alg. 2 can be represented as M = M + KR .
Let M = USV ⊤ and M = U S V ⊤ , we have the column subspaces S ( M )
=
S ( U ) , S ( M ) =
S ( U ) , and the row subspaces S ( M ⊤ ) =
S ( V ) , S ( M ⊤ ) =
S ( V ) .
Let KR =
U ′ S ′
( V ′ )
⊤ .
Since K and R are randomly generated matrices, thus we know that U ′ and V ′ obey the random orthogonal model in
Def. 15.
First, with probability 1 , we have
[Equation]
[Equation]
, specifically ,
[Equation]
Secondly, we bound the coherence changes.
Note that with probability 1 , rank ( M ) =
r = r + k .
From (50), we know that U ∈ R n 1 × r can be represented as the union of U ∈ R n 1 × r and U ′ ∈
R n 1 × k , i.e., U =
[ U U ′ ] , where U ′ obeys the random orthogonal model and the non-overlapping requirement in Lemma 9.
From Lemma 10, we know that µ ( U ′ ) ≤ C · max(1 , log n 1 k ) .
According to Lemma 9, we then obtain that
[Equation]
Similarly, for the row subspace, we have
[Equation]
Therefore,
[Equation]
D Proof for SectionD.1 Proof of Theorem
A potential issue of the proposed scheme in Alg.
1 and Alg.
2 is the projection recovery.
Namely, one can do a corresponding projection to obtain the real data after a single encryption.
Therefore, we execute the proposed scheme twice and introduce two parameters σ 1 and σ 2 in Theorem 4:
• First-round encryption: we randomly generate a matrix K 1 ∈ R n 1 × k and k random numbers R 1 i.i.d ∼ N ( 0 , σ 21 I k ) .
Then we obtain P Ω ( M ) =
P Ω ( M ) + P Ω ( K 1 R 1 ) .
• Second-round encryption: we obtain the column space of M as K 2 ∈ R n 1 × k and k random numbers R 2 i.i.d ∼ N ( 0 , σ 22 I k ) .
Then we obtain P Ω ( M ) =
P Ω ( M ) + P Ω ( K 2 R 2 ) .
As an overview of the proof strategy, we first detail the proof for a single-round encryption case shown in Theorem 5.
Note that Theorem 4 can be easily derived by applying Theorem 5 twice.
Theorem 5.
(Gaussian mechanism [12])
Given an arbitrary n -dimensional function A : T n → O n , define its ℓ 2 -norm sensitivity as ∆ = max dist ( D,D ′ )
=
1 ||A ( D ) − A
( D ′ )
|| 2 where dist ( D, D ′ )
=means that D, D ′ ∈ T n differ from each other by one data entry.
The Gaussian mechanism with parameter σ adds Gaussian noise w i.i.d ∼ N ( 0 , σ 2 I n ) to the output.
Let ϵ ∈
(0 , 1) and c 2 > 2 ln(1 .
25 /δ ) , then the Gaussian mechanism, i.e., A ( D ) + w , with parameter σ ≥ c ∆ (cid:112) 2 ln(2 /δ )
/ϵ satisfies ( ϵ, δ ) -differential privacy.
Proof.
Let v ∈
O n be a vector with || v || 2 ≤ ∆ .
For a pair of databases D, D ′ ∈ T n with dist ( D, D ′ )
= 1 , we would like to obsecure the vector v =
A ( D ) − A
( D ′ ) by adding Gaussian noise w i.i.d ∼ N ( 0 , σ 2 I n ) , i.e., A ( D ) + w .
Thus an adversary user will not be able to differentiate D from D ′ by observing A ( D ) + w or A ( D ′ )
+ w ′ .
According to the joint ( ϵ, δ )-differential privacy in
Def. 3, the key is to quantify σ under which the random variable privacy loss in (22) is bounded by ϵ , with probability at least 1 − δ .
Explicitly, we would like to show that
[Equation]
holds with with probability at least 1 − δ .
Note [Table] , σ 2 I n ) , combining the fact that the distribution of a spherically symmetric normal is independent of the orthogonal basis from which its constituent normals are drawn , we choose to work in a basis that is aligned with v .
For a basis { B 1 , B 2 , ..., B n } (or alternatively represented as a matrix B ∈ R n × n ) with B 1 being parallel to w , we draw w by drawing signed scalars
λ i ∼ N (0 , σ 2 ) for i
= 1 , ..., n , then defining w i
=
λ i B i ∈ R n and w = n (cid:80) i =
1 w i ∈ R n .
Consider the right triangle with base v + w i and edge n (cid:80) i
=
w i ∈ R n , its hypotenuse is v + w .
Then, we have
[Equation]
[Equation]
where we used the fact B 1 is parallel to v and
|| w 1 || 22 = λ 21 .
Recall that ||
v || 2 ≤ ∆ and λ i ∼ N (0 , σ 2 )
for i = 1 , ..., n , the bound in (55) becomes
[Equation]
The quantity in (57) is bounded by ϵ whenever | λ 1 | < σ 2 ϵ ∆ − ∆2 .
Set σ = c ∆ /ϵ
and c 2 > 2 ln(1 .
25 /δ ) .
Let us partition R into R =
R 1 ∪ R 2 , where R 1 = { λ ∈ R | | λ | ≤ c ∆ /ϵ } and R 2 = { λ ∈ R | | λ | > c ∆ /ϵ } , and define O
= O 1 ∪ O 2 where
[Equation]
To ensure the privacy loss is bounded by ϵ with probability at least 1 − δ , one requires
[Equation]
or alternatively, one requires
[Equation]
Taking the integration of (57), one obtains that
[Equation]
yielding the joint ( ϵ, δ ) -differntial privacy for the Gaussian mechanism.