Original: Differentially Private Multi-Party Data Release for Linear Regression
Revised: Differentially Private Multi-Party Data Release for Linear Regression

Original: Abstract
Revised: Abstract

Original: Differentially Private (DP) data release is a promising technique to disseminate data without compromising the privacy of data subjects.
Revised: Differentially Private (DP) data release is a promising technique to disseminate data without compromising the privacy of data subjects.

Original: However the majority of prior work has focused on scenarios where a single party owns all the data.
Revised: However the majority of prior work has focused on scenarios where a single party owns all the data.

Original: In this paper we focus on the multi-party setting, where different stakeholders own disjoint sets of attributes belonging to the same group of data subjects.
Revised: In this paper we focus on the multi-party setting, where different stakeholders own disjoint sets of attributes belonging to the same group of data subjects.

Original: We propose two differentially private algorithms within the context of linear regression that allow all parties to train models on the complete data without the ability to infer private attributes or identities of individuals.
Revised: Within the context of linear regression that allow all parties to train models on the complete data without the ability to infer private attributes or identities of individuals, we start with directly applying Gaussian mechanism and show it has the small eigenvalue problem.

Original: We prove that our DP algorithms asymptotically converge to the optimal (non-private) solutions with increasing dataset size and substantiate these results through experiments on both artiﬁcial and real-world datasets.
Revised: We further propose our novel method and prove it asymptotically converges to the optimal (non-private) solutions with increasing dataset size. We substantiate the theoretical results through experiments on both artiﬁcial and real-world datasets.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: The machine learning community has greatly beneﬁted from open and public datasets [Chapelle and Chang, 2011, Real et al., 2017, Fast and Horvitz, 2017, Kong et al., 2020].
Revised: The machine learning community has greatly beneﬁted from open and public datasets [Chapelle and Chang, 2011, Real et al., 2017, Fast and Horvitz, 2017, Kong et al., 2020].

Original: Unfortunately the privacy concern of data release signiﬁcantly limits the feasibility of sharing many rich and useful datasets to the public, especially in privacy-sensitive domains like health care, ﬁnance, and government etc.
Revised: Unfortunately the privacy concern of data release signiﬁcantly limits the feasibility of sharing many rich and useful datasets to the public, especially in privacy-sensitive domains like health care, ﬁnance, and government etc.

Original: This restriction considerably slows down the research in those areas as well as the general machine learning research given many of today’s algorithms are data-hungry.
Revised: This restriction considerably slows down the research in those areas as well as the general machine learning research given many of today’s algorithms are data-hungry.

Original: Recently, legal and moral concerns on protecting individual privacy become even greater.
Revised: Recently, legal and moral concerns on protecting individual privacy become even greater.

Original: Most countries have imposed strict regulations on the usage and release of sensitive data, e.g. CCPA [Legislature, 2018], HIPPA [Act, 1996] and GDPR [Parliament and of the European Union, 2016].
Revised: Most countries have imposed strict regulations on the usage and release of sensitive data, e.g. CCPA [Legislature, 2018], HIPPA [Act, 1996] and GDPR [Parliament and of the European Union, 2016].

Original: The tension betweenprotecting privacy and promoting research drives the com- +
Revised: The tension between +

Original: Alice:
Revised: Alice:

Original: Bob: …
Revised: Bob: …

Original: Blood test results:
Revised: Blood test results:

Original: Charlie:
Revised: Charlie:

Original: Alice:
Revised: Alice:

Original: Bob: …
Revised: Bob: …

Original: CT images:
Revised: CT images:

Original: Charlie:
Revised: Charlie:

Original: Alice:
Revised: Alice:

Original: Bob: … The degree of liver cancer:
Revised: Bob: … The degree of liver cancer:

Original: Charlie: + + health care example. Clinics have the same set of patients, but different attributes such as blood test results, CT images and  the degree of liver cancer. munity as well as many ML practitioners into a dilemma.
Revised: Charlie: + + protecting privacy and promoting research drives the community as well as many ML practitioners into a dilemma.

Original: Differential privacy (DP)
Revised: Differential privacy (DP)

Original: [Dwork, 2011, Dwork et al., 2006, 2014, Sheffet, 2017, Lee et al., 2019, Xu et al., 2017, Kenthapadi et al., 2012] is shown to be a promising direction to release datasets while protecting individual privacy.
Revised: [Dwork, 2011, Dwork et al., 2006, 2014, Sheffet, 2017, Lee et al., 2019, Xu et al., 2017, Kenthapadi et al., 2012] is shown to be a promising direction to release datasets while protecting individual privacy.

Original: DP provides a formal deﬁnition of privacy to regulate the trade-off between two conﬂicting goals: protecting sensitive information and maintaining data utility.
Revised: DP provides a formal deﬁnition of privacy to regulate the trade-off between two conﬂicting goals: protecting sensitive information and maintaining data utility.

Original: In a DP data release mechanism, the shared dataset is a function of the aggregate of all private samples and the DP guarantees regulate how difﬁcult for anyone to infer the attributes or identity of any individual sample.
Revised: In a DP data release mechanism, the shared dataset is a function of the aggregate of all private samples and the DP guarantees regulate how difﬁcult for anyone to infer the attributes or identity of any individual sample.

Original: With high probability, the public data would be barely affected if any single sample were replaced.
Revised: With high probability, the public data would be barely affected if any single sample were replaced.

Original: Despite the ongoing progress of DP data release, the majority of the prior work mainly focuses on the single-party setting which assumes there is only one party that would release datasets to the public.
Revised: Despite the ongoing progress of DP data release, the majority of the prior work mainly focuses on the single-party setting which assumes there is only one party that would release datasets to the public.

Original: However in many real-world scenarios, there exist multiple parties who own data relevant to each other and want to collectively share the data as a whole to the public.
Revised: However in many real-world scenarios, there exist multiple parties who own data relevant to each other and want to collectively share the data as a whole to the public.

Original: For example, in health care domain, some patients may visit multiple clinics for specialized treatments (Figure 1), and each clinic only has access to its own attributes (e.g. blood test and CT images) collected from the patients.
Revised: For example, in health care domain, some patients may visit multiple clinics for specialized treatments (Figure 1), and each clinic only has access to its own attributes (e.g. blood test and CT images) collected from the patients.

Original: For the same set of patients, attributes combined from all clinics can be more useful to train models.
Revised: For the same set of patients, attributes combined from all clinics can be more useful to train models.

Original: In general, the multi-party setting assumes multiple parties own disjoint sets of attributes (features or labels) belonging to the same group of data subjects (e.g. patients).
Revised: In general, the multi-party setting assumes multiple parties own disjoint sets of attributes (features or labels) belonging to the same group of data subjects (e.g. patients).

Original: One straightforward approach to release data in a multi-party setting is combining data from all parties in a centralized place (e.g. one of the data owners or a third-party), and then releasing it using a private single-party data release approach.
Revised: One straightforward approach to release data in a multi-party setting is combining data from all parties in a centralized place (e.g. one of the data owners or a third-party), and then releasing it using a private single-party data release approach.

Original: However, in a privacy-sensitive organization like a clinic, sending data to another party is prohibited by policy.
Revised: However, in a privacy-sensitive organization like a clinic, sending data to another party is prohibited by policy.

Original: An alternative approach is to let each party individually release its own data to the public through adding sample-wise
Revised: An alternative approach is to let each party individually release its own data to the public through adding sample-wise

Original: Gaussian noise, and then ML practitioners can combine the data together to train models.
Revised: Gaussian noise, and then ML practitioners can combine the data together to train models.

Original: However the resulting models trained on the data combined in this way would show a signiﬁcantly lower utility compared to the models trained on non-private data (conﬁrmed by experiments in Section 5).
Revised: However the resulting models trained on the data combined in this way would show a signiﬁcantly lower utility compared to the models trained on non-private data (conﬁrmed by experiments in Section 5).

Original: To bridge this utility gap, we propose new algorithms specifically designed for multi-party setting.
Revised: To bridge this utility gap, we propose new algorithms specifically designed for multi-party setting.

Original: In summary, we study DP data release in multi-party setting where parties share attributes of the same data subjects publicly through a DP mechanism.
Revised: In summary, we study DP data release in multi-party setting where parties share attributes of the same data subjects publicly through a DP mechanism.

Original: It protects the privacy of all data subjects and can be accessed by the public, including any party involved.
Revised: It protects the privacy of all data subjects and can be accessed by the public, including any party involved.

Original: To this end, we propose the following two differentially private algorithms, both based on Gaussian DP Mechanism [Dwork et al., 2014] within the context of linear regression.
Revised: To this end, we propose the following two differentially private algorithms, both based on Gaussian DP Mechanism [Dwork et al., 2014] within the context of linear regression.

Original: First, in De-biased Gaussian Mechanism for Ordinary Least Squares (DGM-OLS) , each party adds Gaussian noise directly to its data.
Revised: First, in De-biased Gaussian Mechanism for Ordinary Least Squares (DGM-OLS) , each party adds Gaussian noise directly to its data.

Original: The learner with the public data is able to remove a calculated bias from the Hessian matrix.
Revised: The learner with the public data is able to remove a calculated bias from the Hessian matrix.

Revised: However, we show that bias removal brings the small eigenvalue problem.

Original: Second, in Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS) , a random Bernoulli projection matrix is shared to all parties, and each party uses it to project its data along sample-wise dimension before adding Gaussian noise.
Revised: Hence, we propose the sencond method Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS) . A random Bernoulli projection matrix is shared to all parties, and each party uses it to project its data along sample-wise dimension before adding Gaussian noise.

Original: We prove that both algorithms are guaranteed to produce solutions that asymptotically converge to the optimal solutions (i.e. nonprivate) as the dataset size increases.
Revised: We prove that both algorithms are guaranteed to produce solutions that asymptotically converge to the optimal solutions (i.e. non-private) as the dataset size increases.

Original: Through extensive experiments on both synthetic and real-world datasets, we show our algorithms achieve the theoretical claims.
Revised: Through extensive experiments on both synthetic and real-world datasets, we show the later method achieves the theoretical claims and outperforms the ﬁrst method that naively adapts Gaussian mechanism.

Original:  PRELIMINARY
Revised:  PRELIMINARY

Original: A sequence { X n } of random variables in R d is deﬁned to converge in probability towards the random variable X if for all β > 0 ,
Revised: A sequence { X n } of random variables in R d is deﬁned to converge in probability towards the random variable X if for all β > 0 ,

Original: [Equation]
Revised: [Equation]

Original: The norm notation (cid:107)·(cid:107) denotes (cid:96) 2 norm in our paper.
Revised: The norm notation (cid:107)·(cid:107) denotes (cid:96) 2 norm in our paper.

Original: We denote this convergence as plim n →∞ X n = X .
Revised: We denote this convergence as plim n →∞ X n = X .

Original: Differential privacy (DP; [Dwork et al., 2006, 2014]) is a quantiﬁable and rigorous privacy framework, which is formally deﬁned as follows.
Revised: Differential privacy (DP; [Dwork et al., 2006, 2014]) is a quantiﬁable and rigorous privacy framework, which is formally deﬁned as follows.

Original: Deﬁnition 1 ( ( ε, δ ) -differential privacy) .
Revised: Deﬁnition 1 ( ( ε, δ ) -differential privacy) .

Original: A randomized mechanism M : D → R with domain D and range R satisﬁes ( ε, δ ) -differential privacy if for any two adjacent datasets D, D (cid:48) ∈ D , which differ at exactly one data point, and for any subset of outputs S ⊆ R , it holds that 
Revised: A randomized mechanism M : D → R with domain D and range R satisﬁes ( ε, δ ) -differential privacy if for any two adjacent datasets D, D (cid:48) ∈ D , which differ at exactly one data point, and for any subset of outputs S ⊆ R , it holds that P [ M ( D ) ∈ S ] ≤ e ε · P [ M ( D (cid:48) )

Original: [Equation]
Revised: ∈ S ] + δ.

Original: Gaussian mechanism [Dwork et al., 2014] is a post-hoc mechanism to convert a deterministic real-valued function f : D → R m to a randomized algorithm with differential privacy guarantee.
Revised: Gaussian mechanism [Dwork et al., 2014] is a post-hoc mechanism to convert a deterministic real-valued function f : D → R m to a randomized algorithm with differential privacy guarantee.

Original: It relies on sensitivity of f , denoted by
Revised: It relies on sensitivity of f , denoted by

Original: S f , which is deﬁned as the maximum difference of output (cid:107) f
Revised: S f , which is deﬁned as the maximum difference of output (cid:107) f

Original: ( D ) − f
Revised: ( D ) − f

Original: ( D (cid:48) )
Revised: ( D (cid:48) )

Original: (cid:107) .
Revised: (cid:107) .

Original: We deﬁne Gaussian mechanism for differential privacy as below.
Revised: We deﬁne Gaussian mechanism for differential privacy as below.

Original: Lemma 1 (Gaussian mechanism) .
Revised: Lemma 1 (Gaussian mechanism) .

Original: For any deterministic real-valued function f : D → R m with sensitivity S f , we can deﬁne a randomized function by adding Gaussian noise to f : f dp ( D ) :
Revised: For any deterministic real-valued function f : D → R m with sensitivity S f , we can deﬁne a randomized function by adding Gaussian noise to f : [Equation]

Original: = f ( D ) + R, where R is sampled from a multivariate normal distribution
Revised: where R is sampled from a multivariate normal distribution

Original: N (cid:16) 0 , S 2 f σ 2 · I (cid:17) .
Revised: N (cid:16) 0 , S 2 f σ 2 · I (cid:17) .

Original: When σ ≥ √ 2log(1 .
Revised: When σ ≥ √ 2log(1 .

Original: 25 /δ )
Revised: 25 /δ )

Original: ε , f dp is ( ε, δ ) - differentially private for 0 < ε ≤ 1 and δ > 0 .
Revised: ε , f dp is ( ε, δ ) - differentially private for 0 < ε ≤ 1 and δ > 0 .

Original: To simplify notations, we deﬁne σε,δ := √ 2log(1 .
Revised: To simplify notations, we deﬁne σ

Original: 25 /δ ) ε .
Revised: [Equation]

Original: Johnson-Lindenstrauss lemma (JL; [Johnson and Lindenstrauss, 1984, Achlioptas, 2003]) is a technique to compress a set of vectors S
Revised: Johnson-Lindenstrauss lemma (JL; [Johnson and Lindenstrauss, 1984, Achlioptas, 2003]) is a technique to compress a set of vectors S

Original: = { v 1 , · · · , v l } with dimension d to a lower dimension space k < d .
Revised: = { v 1 , · · · , v l } with dimension d to a lower dimension space k < d .

Original: With a proper selection k , it is able to approximately preserve the inner product between any two vectors in the set S with high probability.
Revised: With a proper selection k , it is able to approximately preserve the inner product between any two vectors in the set S with high probability.

Original: We speciﬁcally introduce the Bernoulli version of JL Lemma, which is extended from Theorem 1.1 in Achlioptas [2003].
Revised: We speciﬁcally introduce the Bernoulli version of JL Lemma, which is extended from Theorem 1.1 in Achlioptas [2003].

Original: Lemma 2 (JL Lemma for inner-product preserving (Bernoulli)) .
Revised: Lemma 2 (JL Lemma for inner-product preserving (Bernoulli)) .

Original: Suppose S be an arbitrary set of l points in R d and suppose s is an upper bound for the maximum L2-norm for vectors in S .
Revised: Suppose S be an arbitrary set of l points in R d and suppose s is an upper bound for the maximum L2-norm for vectors in S .

Original: Let B be a k × d random matrix, where B ij are independent random variables taking value from 1 or − 1 with probability 1 / 2 respectively.
Revised: Let B be a k × d random matrix, where B ij are independent random variables taking value from 1 or − 1 with probability 1 / 2 respectively.

Original: With the probability at least 1 − ( l + 1) 2 exp (cid:16) − k (cid:16) [Equation] 4 − β 3 6 (cid:17)(cid:17) , ∀ u , v ∈ S , we have
Revised: With the probability at least 1 − ( l + 1) 2 exp (cid:16) − k (cid:16) β 2 4 − β 3 6 (cid:17)(cid:17) , ∀ u , v ∈ S , we have

Original: [Equation]
Revised: [Equation]

Original:  NOTATION AND PROBLEM SETUP
Revised:  NOTATION AND PROBLEM SETUP

Original: Notations.
Revised: Notations.

Original: Denote D j , j = 1 , · · · , m , as data matrices for m parties, where D j ∈ R n × d j and m ≥ 2 .
Revised: Denote D j , j = 1 , · · · , m , as data matrices for m parties, where D j ∈ R n × d j and m ≥ 2 .

Original: They are aligned by the same set of subjects but have different attributes and they have the same number of samples.
Revised: They are aligned by the same set of subjects but have different attributes and they have the same number of samples.

Original: Deﬁne
Revised: Deﬁne

Original: D = (cid:2) D 1 , · · · , D m (cid:3) ∈ R n × ( d +1) as the collection of all datasets, where d = d 1 + · · · + d m − 1 .
Revised: D = (cid:2) D 1 , · · · , D m (cid:3) ∈ R n × ( d +1) as the collection of all datasets, where d = d 1 + · · · + d m − 1 .

Original: We deﬁne d by subtracting 1 from the total number of attributes because one column is label which we need to treat it separately.
Revised: We deﬁne d by subtracting 1 from the total number of attributes because one column is label which we need to treat it separately.

Original: Deﬁne d max = max j ∈ [ m ] d j , and D i as the i -th row of D , we make the following assumption on data distribution:
Revised: Deﬁne d max = max j ∈ [ m ] d j , and D i as the i -th row of D , we make the following assumption on data distribution:

Original: Assumption 1.
Revised: Assumption 1.

Original: D i , i = 1 , · · · , n , are i.i.d sampled from an underlying distribution P over R d +1 .
Revised: D i , i = 1 , · · · , n , are i.i.d sampled from an underlying distribution P over R d +1 .

Original: Dataset release algorithm.
Revised: Dataset release algorithm.

Original: A private multi-party data release algorithm needs to protect both inter-party and intraparty communications.
Revised: A private multi-party data release algorithm needs to protect both inter-party and intraparty communications.

Original: The general workﬂow of our proposed algorithms is designed as the following:
Revised: The general workﬂow of our proposed algorithms is designed as the following:

Original: Pre-generate random variable B .
Revised: Pre-generate random variable B .

Original: The pre-generated one or more random variables will be shared among parties.
Revised: The pre-generated one or more random variables will be shared among parties.

Original: Privatize the dataset locally with the algorithm A priv .
Revised: Privatize the dataset locally with the algorithm A priv .

Original: Each party applies the same privatizing algorithm
Revised: Each party applies the same privatizing algorithm

Original: A priv that takes the local dataset D j ∈ R n × d j and the random matrix B as the inputs and then outputs k (predeﬁned) “encrypted” samples (cid:0) D pub (cid:1)
Revised: A priv that takes the local dataset D j ∈ R n × d j and the random matrix B as the inputs and then outputs k (predeﬁned) “encrypted” samples (cid:0) D pub (cid:1)

Original: j :
Revised: j :

Original: =
Revised: =

Original: A priv ( D j ; B ) ∈
Revised: A priv ( D j ; B ) ∈

Original: R k × d j . 3.
Revised: R k × d j . 3.

Original: Release the dataset.
Revised: Release the dataset.

Original: All parties jointly release D pub =
Revised: All parties jointly release D pub =

Original: [ (cid:0) D pub (cid:1) 1 , · · · , (cid:0) D pub (cid:1) m
Revised: [ (cid:0) D pub (cid:1) 1 , · · · , (cid:0) D pub (cid:1) m

Original: ] ∈ R k × ( d +1) to the public.
Revised: ] ∈ R k × ( d +1) to the public.

Original: Note that we need to specially design random variable B and the privatizing algorithm A priv , which we will introduce in the next section.
Revised: Note that we need to specially design random variable B and the privatizing algorithm A priv , which we will introduce in the next section.

Original: In addition, the random variable B allows the dependencies between the randomized output from all parties, which can be utilized to guarantee the ﬁnal utility.
Revised: In addition, the random variable B allows the dependencies between the randomized output from all parties, which can be utilized to guarantee the ﬁnal utility.

Original: Privacy constraint.
Revised: Privacy constraint.

Original: Since the public will observe the released dataset D pub , for each j ∈ [ m ] , (cid:0) D pub (cid:1)
Revised: Since the public will observe the released dataset D pub , for each j ∈ [ m ] , (cid:0) D pub (cid:1)

Original: j should not leak the information of the private dataset D j .
Revised: j should not leak the information of the private dataset D j .

Original: Formally we require ∀ j ∈ [ m ] , A priv ( D j ; B ) is differentially private, where two neighbouring datasets D j and (cid:0) D j (cid:1) (cid:48) differ at one row (sample).
Revised: Formally we require ∀ j ∈ [ m ] , A priv ( D j ; B ) is differentially private, where two neighbouring datasets D j and (cid:0) D j (cid:1) (cid:48) differ at one row (sample).

Original: However the multi-party setting requires more than the above guarantee because each party j (cid:48) (cid:54) =
Revised: However the multi-party setting requires more than the above guarantee because each party j (cid:48) (cid:54) =

Original: j not only observes D j but also the shared random variable B .
Revised: j not only observes D j but also the shared random variable B .

Original: Thus we need to further require that given B , each party j cannot infer information about other private datasets D j .
Revised: Thus we need to further require that given B , each party j cannot infer information about other private datasets D j .

Original: In terms of differential privacy, it is required that condition on B for any possible sample value I ,
Revised: In terms of differential privacy, it is required that condition on B for any possible sample value I ,

Original: A priv ( D j ; B ) is ( ε, δ ) -differentially private, i.e. for any two neighbouring datasets D j and (cid:0) D j (cid:1) (cid:48) and B , we have
Revised: A priv ( D j ; B ) is ( ε, δ ) -differentially private, i.e. for any two neighbouring datasets D j and (cid:0) D j (cid:1) (cid:48) and B , we have

Original: P ( A priv ( D j ; B ) | B ) ≤ e ε · P (cid:16)(cid:16)
Revised: P ( A priv ( D j ; B ) | B ) ≤ e ε · P (cid:16)(cid:16)

Original: A priv (cid:16)(cid:0) D j (cid:1) (cid:48) ; B (cid:17)(cid:12)(cid:12)(cid:12) B (cid:17) + δ.
Revised: A priv (cid:16)(cid:0) D j (cid:1) (cid:48) ; B (cid:17)(cid:12)(cid:12)(cid:12) B (cid:17) + δ.

Original: Utility target.
Revised: Utility target.

Original: We aim to guarantee the performance of the linear regression task  on the  released dataset  D j .
Revised: We aim to guarantee the performance of arbitrary linear regression task (arbitrarily selected label and features) on the joint released dataset (cid:2) D 1 , · · · , D m (cid:3) .

Original: The label in the linear regression task isone of the attributes which we assume to be the last dimension, and the features are the rest of the attributes.
Revised: Out of the notation simplicity, we assume the label in the linear regression task is the last attribute, and the features are the rest of the attributes.

Original: Under this assumption, the joint private dataset D can be written as [ X, Y ] , where X ∈ R n × d is the private feature matrix and Y ∈ R n is the private label vector.
Revised: Under this assumption, the joint private dataset D can be written as [ X, Y ] , where X ∈ R n × d is the private feature matrix and Y ∈ R n is the private label vector.

Original: Similarly the public dataset D pub can be written as [ X pub , Y pub ] , where
Revised: Similarly the public dataset D pub can be written as [ X pub , Y pub ] , where

Original: X pub ∈ R k × d and Y pub ∈ R k .
Revised: X pub ∈ R k × d and Y pub ∈ R k .

Original: We deﬁne the loss function by the expected squared loss:
Revised: We deﬁne the loss function by the expected squared loss:

Original: [Equation]
Revised: [Equation]

Original: where the data point is sampled from the distribution P in Assumption 1.
Revised: where the data point is sampled from the distribution P in Assumption 1.

Original: We make two more assumptions for the distribution P : the standard normalization and the no perfect multicollinearity assumption.
Revised: We make two more assumptions for the distribution P : the standard normalization and the no perfect multicollinearity assumption.

Original: The latter is common in the literature of linear regression [Farrar and Glauber, 1967,
Revised: The latter is common in the literature of linear regression [Farrar and Glauber, 1967,

Original: Chatterjee and Hadi, 2006].
Revised: Chatterjee and Hadi, 2006].

Original: Assumption 2.
Revised: Assumption 2.

Original: The absolute values of all attributes | D ij | are bounded by 1 .
Revised: The absolute values of all attributes | D ij | are bounded by 1 .

Original: Assumption
Revised: Assumption

Original: [Equation]
Revised: [Equation]

Original: positive deﬁnite.
Revised: positive deﬁnite

Original:  Under Assumption 3, derived by setting ∇ w L ( w ; P ) = 0 , the optimal solution w ∗ to the loss in Equation 1 has the following explicit form:
Revised: [Equation] Under Assumption 3, derived by setting ∇ w L ( w ; P ) = 0 , the optimal solution w ∗ to the loss in Equation 1 has the following explicit form:

Original: [Equation]
Revised: [Equation]

Original: The utility target (for the trained linear regression model) is determined by our release algorithm ( B, A priv ) .
Revised: The utility target (for the trained linear regression model) is determined by our release algorithm ( B, A priv ) .

Original: For a given the public dataset D pub released by our algorithms, we deﬁne our utility target as the existence of a training algorithm
Revised: For a given the public dataset D pub released by our algorithms, we deﬁne our utility target as the existence of a training algorithm

Original: A lr that achieves the asymptotic property for the trained model weights ˆ w n :=
Revised: A lr that achieves the asymptotic property for the trained model weights ˆ w n :=

Original: A lr (cid:0) D pub (cid:1) as the dataset size n → ∞ .
Revised: A lr (cid:0) D pub (cid:1) as the dataset size n → ∞ .

Original: The asymptotic property is commonly studied in differential privacy [Chaudhuri and Hsu, 2011, Bassily et al., 2014, Feldman et al., 2020] and we restate it as follows:
Revised: The asymptotic property is commonly studied in differential privacy [Chaudhuri and Hsu, 2011, Bassily et al., 2014, Feldman et al., 2020] and we restate it as follows:

Original: ˆ w n converges to w ∗ in probability as the size of dataset n increases, i.e. 
Revised: ˆ w n converges to w ∗ in probability as the size of dataset n increases, i.e. ∀ β > 0 , lim n →∞

Original: [Equation]
Revised: [Equation]

Original: The randomness from the above property comes from data sampling P , dataset release algorithm ( B, A priv ) , and the training algorithm
Revised: The randomness from the above property comes from data sampling P , dataset release algorithm ( B, A priv ) , and the training algorithm

Original: A lr .
Revised: A lr .

Original:  METHODOLOGY
Revised:  METHODOLOGY

Original: We now describe our data release algorithms which both satisfy the differential privacy and yield asymptotically optimal solutions to the linear regression task.
Revised: We now describe our data release algorithms which both satisfy the differential privacy and yield asymptotically optimal solutions to the linear regression task.

Original: We start with the ﬁrst algorithm De-biased Gaussian Mechanism for Ordinary
Revised: We start with the ﬁrst algorithm De-biased Gaussian Mechanism for Ordinary

Original: Least Squares (DGM-OLS) , which directly applies Gaussian mechanism when releasing the data and then de-biasesthe Hessian matrix when training the model.
Revised: Least Squares (DGM-OLS) , which directly applies Gaussian mechanism when releasing the data and then de-biases

Original: However the Dataset Release do 3:
Revised: Dataset [Table] j do 3:

Original: The party j computes (cid:0) D dgm (cid:1) j := D j + R j , where
Revised: The party j computes (cid:0) D dgm (cid:1) j := D j + R j , where

Original: R j ∈ R n × d j is a random Gaussian matrix and elements in R j are i.i.d sampled from N (cid:16) 0 , 4 d max · σ 2 ε,δ (cid:17) 
Revised: R j ∈ R n × d j is a random Gaussian matrix and elements in R j are i.i.d sampled from N (cid:16) 0 , 4 d max · σ 2 ε,δ (cid:17) .

Original: : end for
Revised: : end for

Original: Training Algorithm
Revised: [Table] the Hessian matrix when training the model.

Original:  de-bias operator introduces the possible inverse of a matrix with small eigenvalues, which severely hurts the performance of the learned model.
Revised: However the de-bias operator introduces the possible inverse of a matrix with small eigenvalues, which severely hurts the performance of the learned model.

Original: We therefore propose a novel dataset release algorithm rather than the directly application to Gaussian mechanism – Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS) .
Revised: We therefore propose a novel dataset release algorithm rather than the directly application to Gaussian mechanism – Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS) .

Original: The model learned from the corresponding released public dataset is also guaranteed to be asymptotically optimal, and, more importantly, avoids the problem of small eigenvalues.
Revised: The model learned from the corresponding released public dataset is also guaranteed to be asymptotically optimal, and, more importantly, avoids the problem of small eigenvalues.

Original: 4.1 DE-BIASED GAUSSIAN MECHANISM (DGM-OLS)
Revised: 4.1 DE-BIASED GAUSSIAN MECHANISM (DGM-OLS)

Original: The De-biased Gaussian Mechanism for Ordinary Least
Revised: The De-biased Gaussian Mechanism for Ordinary Least

Original: Squares (DGM-OLS) includes the dataset release algorithm and the corresponding training algorithm.
Revised: Squares (DGM-OLS) includes the dataset release algorithm and the corresponding training algorithm.

Original: Algorithmshows the overview and we will introduce them next.
Revised: Algorithmshows the overview and we will introduce them next.

Original: Dataset release algorithm.
Revised: Dataset release algorithm.

Original: Each party directly applies
Revised: Each party directly applies

Original: Gaussian mechanism to their own dataset D j ( j = 1 , · · · m ) to satisfy the differential privacy.
Revised: Gaussian mechanism to their own dataset D j ( j = 1 , · · · m ) to satisfy the differential privacy.

Original: Consider two neighboring data matrices D j and (cid:0) D j (cid:1) (cid:48) differing at exactly one row with the row index i .
Revised: Consider two neighboring data matrices D j and (cid:0) D j (cid:1) (cid:48) differing at exactly one row with the row index i .

Original: Implied by Assumption 2, we can compute the sensitivity of the data matrix D j : 
Revised: Implied by Assumption 2, we can compute the sensitivity of the data matrix D j : [Equation] ≤ [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: (cid:112) d

Revised: [Equation]

Original: Then each party independently adds a Gaussian noise R j to
Revised: Then each party independently adds a Gaussian noise R j to

Original: D j .
Revised: D j .

Original: Entries in R j are i.i.d sampled from Gaussian distribution N ∼ (0 ,
Revised: Entries in R j are i.i.d sampled from Gaussian distribution N ∼ (0 ,

Original: 4 d max σ 2 ε,δ ) .
Revised: 4 d max σ 2 ε,δ ) .

Original: The dataset release algorithm meets the privacy constraints in section 3.
Revised: The dataset release algorithm meets the privacy constraints in section 3.

Original: No random matrix B is shared among different parties.
Revised: No random matrix B is shared among different parties.

Original: Lemma 1 guarantees that (cid:0) D dgm (cid:1)
Revised: Lemma 1 guarantees that (cid:0) D dgm (cid:1)

Original: j is ( ε, δ ) - differentially private w.r.t.
Revised: j is ( ε, δ ) - differentially private w.r.t.

Original: D j for any 0 < ε ≤ 1 , δ > 0 .
Revised: D j for any 0 < ε ≤ 1 , δ > 0 .

Original: Training algorithm.
Revised: Training algorithm.

Original: Given the dataset released through the above algorithm, there exists an asymptotic linear regression solution.
Revised: Given the dataset released through the above algorithm, there exists an asymptotic linear regression solution.

Original: Denote the feature matrix and the label vector of the private and public joint dataset as [ X, Y ]
Revised: Denote the feature matrix and the label vector of the private and public joint dataset as [ X, Y ]

Original: = D and [ X dgm , Y dgm ]
Revised: = D and [ X dgm , Y dgm ]

Original: = D dgm .
Revised: = D dgm .

Original: Further deﬁne
Revised: Further deﬁne

Original: R := (cid:2) R 1 , · · · , R m (cid:3) ∈ R n × ( d +1) and split R into R X and
Revised: R := (cid:2) R 1 , · · · , R m (cid:3) ∈ R n × ( d +1) and split R into R X and

Original: R Y representing the additive noise to X and Y respectively.
Revised: R Y representing the additive noise to X and Y respectively.

Original: Consider the ordinary least square solution for the public data X dgm and Y dgm , whose explicit form is:
Revised: Consider the ordinary least square solution for the public data X dgm and Y dgm , whose explicit form is:

Original: [Equation]
Revised: [Equation]

Original: Compared with our target solution
Revised: Compared with our target solution

Original: [Equation]
Revised: [Equation]

Original: we can prove that plim n →∞ 1 n (cid:0) X dgm (cid:1) (cid:62) Y dgm =
Revised: we can prove that plim [Equation] x [Equation]

Original: E ( x ,y ) ∼P
Revised: y ) ∼P

Original: [ x · y ] by the concentration of bounded random variables and multivariate normal distribution.
Revised: [ x · y ] by the concentration of bounded random variables and multivariate normal distribution.

Original: Nevertheless, there is a gap between plim 
Revised: Nevertheless, there is a gap between plim n →∞

Original: [Equation]
Revised: [Equation]

Original: dgm
Revised: n

Original: [Equation]
Revised: [Equation]

Revised: and E

Original: X dgm and E ( x ,y ) ∼P (cid:2) xx (cid:62) (cid:3) :plim
Revised: [Equation] x ,y ) ∼P [Equation] (cid:3) :

Original: [Equation]
Revised: [Equation]

Original: where the last equation again holds by the concentration of bounded random variables and multivariate normal distribution.
Revised: where the last equation again holds by the concentration of bounded random variables and multivariate normal distribution.

Original: To reduce the bias 4 d max σ 2 ε,δ · I , we can revise the solution computation in Equation 2 to ˆ w dgm n deﬁned as
Revised: To reduce the bias 4 d max σ 2 ε,δ · I , we can revise the solution computation in Equation 2 to ˆ w dgm n deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is estimated for the inverse of the Hessian matrix E ( x ,y ) ∼P (cid:2) xx (cid:62) (cid:3) , which we denote as ( ˆ H dgm n ) − 1 .
Revised: The ﬁrst term is estimated for the inverse of the Hessian matrix E ( x ,y ) ∼P (cid:2) xx (cid:62) (cid:3) , which we denote as ( ˆ H dgm n ) − 1 .

Original: The asymptotic optimality for the solution ˆ w dgm n is implied by the theorem below and the proof is in the Appendix.
Revised: The asymptotic optimality for the solution ˆ w dgm n is implied by the theorem below and the proof is in the Appendix.

Original: Theorem 1.
Revised: Theorem 1.

Original: When β ≤ c for some variable c that is dependent of σ ε,δ , d , and P , but is independent of n ,
Revised: When β ≤ c for some variable c that is dependent of σ ε,δ , d , and P , but is independent of n ,

Original: [Equation]
Revised: [Equation]

Original: Problem of small eigenvalues.
Revised: Problem of small eigenvalues.

Original: The expectation of ˆ H dgm n is a positive deﬁnite matrix given Assumption 3, but the sample of ˆ H dgm n itself is not guaranteed.
Revised: The expectation of ˆ H dgm n is a positive deﬁnite matrix given Assumption 3, but the sample of ˆ H dgm n itself is not guaranteed.

Original: With a certain probability, it has small eigenvalues that might lead to explosion when computing its inverse.
Revised: With a certain probability, it has small eigenvalues that might lead to explosion when computing its inverse.

Original: In our experiments (section 5), we ﬁnd that ˆ H dgm n suffers from the small eigenvalues even if n is as large as 10 6 .
Revised: In our experiments (section 5), we ﬁnd that ˆ H dgm n suffers from the small eigenvalues even if n is as large as 10 6 .

Original: As a result, the model utility is much more inferior than what is guaranteed theoretically.
Revised: As a result, the model utility is much more inferior than what is guaranteed theoretically.

Original: This motivates us to design the second algorithm. 
Revised: This motivates us to design the second algorithm. Algorithm

Original: Dataset Release 1: Input: D = (cid:2) D 1 , · · · , D m (cid:3) , ε, δ, k .
Revised: Dataset Release 1: Input: D = (cid:2) D 1 , · · · , D m (cid:3) , ε, δ, k .

Original: 2: The ﬁrst party pre-generates a k × n random matrix
Revised: 2: The ﬁrst party pre-generates a k × n random matrix

Original: B where all entries in B are i.i.d.
Revised: B where all entries in B are i.i.d.

Original: sampled from the distribution with probability 1 / 2 for 1 and 1 / 2 for − 1 .
Revised: sampled from the distribution with probability 1 / 2 for 1 and 1 / 2 for − 1 .

Original: Then ﬁrst party sends the random matrix sample B to all parties.
Revised: Then ﬁrst party sends the random matrix sample B to all parties.

Original: 3: for j = 1 , · · · , m do 4:
Revised: 3: for j = 1 , · · · , m do 4:

Original: The party j computes ( D rmgm ) j :=
Revised: The party j computes ( D rmgm ) j :=

Original: BD j / √ k + R j , where R j is a k × d j random matrix and all elements in R j are i.i.d.
Revised: BD j / √ k + R j , where R j is a k × d j random matrix and all elements in R j are i.i.d.

Original: sampled from the multivariate normal distribution N (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .
Revised: sampled from the multivariate normal distribution N (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .

Original: 5: end for 6: Return: D rmgm := (cid:104) ( D rmgm ) 1 , · · · , ( D rmgm ) m (cid:105) .
Revised: 5: end for 6: Return: D rmgm := (cid:104) ( D rmgm ) 1 , · · · , ( D rmgm ) m (cid:105) .

Original: Training Algorithm 1: Input: D rmgm , ε, δ 2: [ X rmgm , Y rmgm ]
Revised: Training Algorithm 1: Input: D rmgm , ε, δ 2: [ X rmgm , Y rmgm ]

Original: =
Revised: =

Original: D rmgm 3: Compute the ordinary least square solution ˆ w rmgm n := (cid:16) ( X rmgm ) (cid:62) X rmgm (cid:17) − 1 ( X rmgm )
Revised: D rmgm 3: Compute the ordinary least square solution ˆ w rmgm n := (cid:16) ( X rmgm ) (cid:62) X rmgm (cid:17) − 1 ( X rmgm )

Original: (cid:62) Y rmgm .
Revised: (cid:62) Y rmgm .

Original: 4: Return: ˆ w rmgm n .
Revised: 4: Return: ˆ w rmgm n .

Original: 4.2 RANDOM MIXING PRIOR TO GAUSSIAN MECHANISM (RMGM-OLS)
Revised: 4.2 RANDOM MIXING PRIOR TO GAUSSIAN MECHANISM (RMGM-OLS)

Original: In previous method’s dataset release stage, when we directly add the Gaussian additive noise R to the data, in order to guarantee DP, the norm of the noise needed has to be the same order (in n ) as the norm of the data matrix D .
Revised: In previous method’s dataset release stage, when we directly add the Gaussian additive noise R to the data, in order to guarantee DP, the norm of the noise needed has to be the same order (in n ) as the norm of the data matrix D .

Original: Both D and R have norm in Θ( √ n ) .
Revised: Both D and R have norm in Θ( √ n ) .

Original: Thus later in the training stage, the additive noise R when compared to the data matrix X would not diminish as n → ∞ and we have to subtract 4 d max σ 2 ε,δ · I from (cid:0) X dgm (cid:1) (cid:62) X dgm to remove this additive noise in order to obtain the optimal model weights.
Revised: Thus later in the training stage, the additive noise R when compared to the data matrix X would not diminish as n → ∞ and we have to subtract 4 d max σ 2 ε,δ · I from (cid:0) X dgm (cid:1) (cid:62) X dgm to remove this additive noise in order to obtain the optimal model weights.

Original: This subtraction is the problematic part that brings training instability (small eigenvalues in the Hessian matrix).
Revised: This subtraction is the problematic part that brings training instability (small eigenvalues in the Hessian matrix).

Original: Instead, we can avoid such subtraction in the training stage by imposing a smaller noise in the data release stage.
Revised: Instead, we can avoid such subtraction in the training stage by imposing a smaller noise in the data release stage.

Original: If we can design the data release stage properly, so that the addictive noise has relatively smaller order in n than D , in the later training stage, the learner would no longer need the problematic de-biasing step.
Revised: If we can design the data release stage properly, so that the addictive noise has relatively smaller order in n than D , in the later training stage, the learner would no longer need the problematic de-biasing step.

Original: Algorithm 2 shows the full details of Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS).
Revised: Algorithm 2 shows the full details of Random Mixing prior to Gaussian Mechanism for Ordinary Least Squares (RMGM-OLS).

Original: We now explain the design of data release and training algorithm based on the above insights.
Revised: We now explain the design of data release and training algorithm based on the above insights.

Original: Dataset release algorithm.
Revised: Dataset release algorithm.

Original: Suppose b is an ndimensional vector in {− 1 , 1 } n .
Revised: Suppose b is an ndimensional vector in {− 1 , 1 } n .

Original: For any two neighbouring daasets D j and (cid:0) D j (cid:1) (cid:48) that are different at row index i , the sensitivity of b (cid:62)
Revised: For any two neighbouring daasets D j and (cid:0) D j (cid:1) (cid:48) that are different at row index i , the sensitivity of b (cid:62)

Original: D j is
Revised: D j is

Original: [Table]
Revised: [Table]

Original: Moreover, when B ∈ {− 1 , 1 } k × n , BD j / √ k has sensitivity 2 √ d max as well.
Revised: Moreover, when B ∈ {− 1 , 1 } k × n , BD j / √ k has sensitivity 2 √ d max as well.

Original: We now introduce the data release algorithm.
Revised: We now introduce the data release algorithm.

Original: Suppose all parties are sharing a random matrix B ∈ {− 1 , 1 } k × n , where all elements in B are i.i.d.
Revised: Suppose all parties are sharing a random matrix B ∈ {− 1 , 1 } k × n , where all elements in B are i.i.d.

Original: sampled from the distribution with probability 1 / 2 for 1 and 1 / 2 for − 1 .
Revised: sampled from the distribution with probability 1 / 2 for 1 and 1 / 2 for − 1 .

Original: Then we deﬁne the local computation for each party j :
Revised: Then we deﬁne the local computation for each party j :

Original: [Equation]
Revised: [Equation]

Original: where R j is a k × d j random matrix and all elements in R j are i.i.d.
Revised: where R j is a k × d j random matrix and all elements in R j are i.i.d.

Original: sampled from the multivariate normal distribution
Revised: sampled from the multivariate normal distribution

Original: N (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .
Revised: N (cid:16) 0 , 4 d max σ [Equation] ,δ

Revised: [Equation]

Original: Gaussian mechanism guarantees for any ﬁxed B ∈ { 1 , − 1 } k × n , ( D rmgm )
Revised: Gaussian mechanism guarantees for any ﬁxed B ∈ { 1 , − 1 } k × n , ( D rmgm )

Original: j is ( ε, δ ) -differentially private w.r.t.
Revised: j is ( ε, δ ) -differentially private w.r.t.

Original: the dataset D j for 0 < ε ≤ 1 , δ > 0 .
Revised: the dataset D j for 0 < ε ≤ 1 , δ > 0 .

Original: Importantly, now the addictive noise R jnow is relatively small than BD j / √ k .
Revised: Importantly, now the addictive noise R j is relatively small than BD j / √ k .

Original: The order of (cid:107) R j (cid:107) is Θ( k ) while the order of (cid:13)(cid:13)(cid:13) BD j / √ k (cid:13)(cid:13)(cid:13) ≈ (cid:107) D j (cid:107) is Θ( n ) (by JL Lemma).
Revised: The order of (cid:107) R j (cid:107) is Θ( k ) while the order of (cid:13)(cid:13)(cid:13) BD j / √ k (cid:13)(cid:13)(cid:13) ≈ (cid:107) D j (cid:107) is Θ( n ) (by JL Lemma).

Original: If we set k = o ( n ) , the additive noise compared to the original data matrix D will diminish as n → ∞ .
Revised: If we set k = o ( n ) , the additive noise compared to the original data matrix D will diminish as n → ∞ .

Original: This implies that the standard ordinary least square solution to the public dataset [ X rmgm , Y rmgm ] would converge to the optimal solution w ∗ without special subtraction.
Revised: This implies that the standard ordinary least square solution to the public dataset [ X rmgm , Y rmgm ] would converge to the optimal solution w ∗ without special subtraction.

Original: Training algorithm.
Revised: Training algorithm.

Original: Given the feature matrix X rmgm and the label vector Y rmgm from the released dataset, wenow show that the vanilla ordinary least square solution
Revised: Given the feature matrix X rmgm and the label vector Y rmgm from the released dataset, we show that the vanilla ordinary least square solution

Original: [Equation]
Revised: [Equation]

Original: is asymptotically optimal, i.e. plim 
Revised: is asymptotically optimal, i.e. plim n →∞

Original: [Equation]
Revised: [Equation]

Original:  n = w [Equation]
Revised: rmgm n = w ∗ .

Original: To prove the above asymptotic optimality, we show
Revised: To prove the above asymptotic optimality, we show

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: x ,y ) P [ x · y ] respectively, and together they prove the optimality.
Revised: y ) P [ x · y ] respectively, and together they prove the optimality.

Original: Deﬁne R =(cid:2)
Revised: Deﬁne R =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , R m (cid:3) ∈
Revised: ∈

Original: R k × ( d +1) and split R into R X and R Y representing the additive noises to BX/ √ k and BY/ √ k respectively.
Revised: R k × ( d +1) and split R into R X and R Y representing the additive noises to BX/ √ k and BY/ √ k respectively.

Original: Because plim n →∞ 1 n X (cid:62) X = E ( x ,y ) P (cid:2) xx (cid:62) (cid:3) , it is sufﬁcient to show plim n →∞ 1 n ( X rmgm ) (cid:62)X rmgm
Revised: Because plim n →∞ 1 n X (cid:62) X = E ( x ,y ) P (cid:2) xx (cid:62) (cid:3) , it is sufﬁcient to show plim n →∞ 1 n ( X rmgm ) (cid:62)

Original: [Equation]
Revised: [Equation]

Original: 1 n X (cid:62) X = 0 .
Revised: n X (cid:62) X = 0 .

Original: Now we decompose 1 n ( X 
Revised: Now we decompose 1 n ( X rmgm ) (cid:62)

Original: [Equation]
Revised: [Equation]

Original: n X (cid:62) X as below:
Revised: n X (cid:62) X as below:

Original: [Equation]
Revised: [Equation]

Original: We informally show how each term converges to 0 as n → ∞ : 1.1 n [Equation] X
Revised: We informally show how each term converges to 0 as n → ∞ : 1.

Original: [Equation]
Revised: [Equation]

Original: X (cid:17) .
Revised: (cid:17) .

Original: If k → ∞ as n → ∞ , the convergence is directly implied by Lemma 2.
Revised: If k → ∞ as n → ∞ , the convergence is directly implied by Lemma 2.

Original: 2. 1 n
Revised: 2. 1 n

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: √ k R X
Revised: √ k R X

Original: + R
Revised: + R

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: √ k X (cid:17) .
Revised: √ k X (cid:17) .

Original: Properties of normal distribution
Revised: Properties of normal distribution

Original: guarantees the approximation B
Revised: guarantees the approximation B

Original: (cid:62) √ k R
Revised: (cid:62) √ k R

Original: X ≈
Revised: X ≈

Original: R
Revised: R

Original: (cid:48)
Revised: (cid:48)

Original: X , where R (cid:48) X ∈ R n × d is a Gaussian matrix with
Revised: X , where R (cid:48) X ∈ R n × d is a Gaussian matrix with

Original: N
Revised: N

Original: ∼ (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .
Revised: ∼ (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .

Original: Then (cid:13)(cid:13)(cid:13) 1 n X (cid:62) B (cid:62)
Revised: Then (cid:13)(cid:13)(cid:13) 1 n X (cid:62) B (cid:62)

Original: [Equation]
Revised: [Equation]

Original: 1 n X
Revised: 1 n X

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: √ n (cid:17) .
Revised: √ n (cid:17) .

Original: 1 n R (cid:62) X R X .
Revised: 1 n R (cid:62) X R X .

Original: If k → ∞ as n → ∞ , (cid:13)(cid:13) 1 k R (cid:62) X R X (cid:13)(cid:13) will converge to 4 d max σ 2 ε,δ · I .
Revised: If k → ∞ as n → ∞ , (cid:13)(cid:13) 1 k R (cid:62) X R X (cid:13)(cid:13) will converge to 4 d max σ 2 ε,δ · I .

Original: On the other hand, when k = o ( n ) , kn · 4 d max σ 2 ε,δ · I will converge to 0 as n → ∞ .
Revised: On the other hand, when k = o ( n ) , kn · 4 d max σ 2 ε,δ · I will converge to 0 as n → ∞ .

Original: Notice that the above convergence relies on the proper selection of k .
Revised: Notice that the above convergence relies on the proper selection of k .

Original: There exists a trade-off: larger k leads to better convergence rate of the ﬁrst term, but worse rate for the diminishing of additive noise – the third term.
Revised: There exists a trade-off: larger k leads to better convergence rate of the ﬁrst term, but worse rate for the diminishing of additive noise – the third term.

Original: The following theorem shows the exact asymptotic rate:
Revised: The following theorem shows the exact asymptotic rate:

Original: Theorem 2.
Revised: Theorem 2.

Original: When β ≤ c for some variable c that is dependent of d and P , but independent of σ ε,δ , n , we have
Revised: When β ≤ c for some variable c that is dependent of d and P , but independent of σ ε,δ , n , we have

Original: P [ (cid:107) ˆ w rmgm n − w
Revised: P [ (cid:107) ˆ w rmgm n − w

Original: [Equation]
Revised: [Equation]

Original: (cid:107) > β ] < exp (cid:32) − O (cid:32) min (cid:40)
Revised: (cid:107) > β ] < exp (cid:32) − O (cid:32) min (cid:40)

Original: kβd 2 , nβ kdd max σ
Revised: kβd 2 , nβ kdd max σ

Original: [Equation]
Revised: [Equation]

Original: , n 1 / 2 β dd
Revised: , n 1 / 2 β dd

Original: [Equation]
Revised: [Equation]

Original: / 2max
Revised: / 2max

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: If we choose k = O (cid:18) n 1 / 2 d 1 / 2 d 1 / 2 max σ ε,δ (cid:19) , then
Revised: If we choose k = O (cid:18) n 1 / 2 d 1 / 2 d 1 / 2 max σ ε,δ (cid:19) , then

Original: P [ (cid:107) ˆ w rmgm n
Revised: P [ (cid:107) ˆ w rmgm n

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: (cid:107) >
Revised: (cid:107) >

Original: [Equation]
Revised: [Equation]

Original: ] < exp
Revised: ] < exp

Original: [Equation]
Revised: [Equation]

Original: In the theorem, k is selected to balance kβ 2 d 

Revised: In the theorem, k is selected to balance kβ 2 d

Original: and nβ kdd max σ 2 ε,δ .
Revised: and nβ kdd max σ 2 ε,δ .

Original: To achieve the optimal rate for f ( β ) with any ﬁxed β , the optimal k is chosen as O
Revised: To achieve the optimal rate for f ( β ) with any ﬁxed β , the optimal k is chosen as O

Original: [Equation]
Revised: [Equation]

Original: Comparison with DGM-OLS.
Revised: Comparison with DGM-OLS.

Original: The near-zero eigenvalue issue is solved since ( X rmgm ) (cid:62) X rmgm (cid:23) 0 holds naturally by its deﬁnition.
Revised: The near-zero eigenvalue issue is solved since ( X rmgm ) (cid:62) X rmgm (cid:23) 0 holds naturally by its deﬁnition.

Original: Moreover, although the convergence rate of n is sacriﬁced, the orders in d, d max and σ ε,δ are much improved.
Revised: Moreover, although the convergence rate of n is sacriﬁced, the orders in d, d max and σ ε,δ are much improved.

Original: In section 5 we show that the RMGM-OLS outperforms DGM-OLS on both synthetic datasets even when n is as large as 3 × 10 6 .
Revised: In section 5 we show that the RMGM-OLS outperforms DGM-OLS on both synthetic datasets even when n is as large as 3 × 10 6 .

Original:  EXPERIMENTAL EVALUATION
Revised:  EXPERIMENTAL EVALUATION

Original: In this section, we evaluate DGM-OLS and RMGM-OLS on both synthetic and real world datasets.
Revised: In this section, we evaluate DGM-OLS and RMGM-OLS on both synthetic and real world datasets.

Original: Our experiments on synthetic dataset are designed to verify the theoretical asymptotic results in section 4 by increasing the training set size n .
Revised: Our experiments on synthetic dataset are designed to verify the theoretical asymptotic results in section 4 by increasing the training set size n .

Original: We further justify the algorithm performance on ﬁve real-world datasets, four from UCI Machine Learning
Revised: We further justify the algorithm performance on ﬁve real-world datasets, four from UCI Machine Learning

Original: Repository 

Revised: Repository

Original: [Dua and Graff, 2017] and one from kaggle.
Revised: [Dua and Graff, 2017] and one from kaggle.

Original: 5.1 EXPERIMENT SET-UP
Revised: 5.1 EXPERIMENT SET-UP

Original: Algorithm set-up.
Revised: Algorithm set-up.

Original: We evaluate both DGM-OLS and
Revised: We evaluate both DGM-OLS and

Original: RMGM-OLS.
Revised: RMGM-OLS.

Original: For k in RMGM-OLS, we set k = √ n σ ε,δ in synthetic dataset experiments and select the best k from { 10 2 , 3 × 10 2 , 10 3 , 3 × 10 3 , 10 4 } in real-world dataset exper iments.
Revised: For k in RMGM-OLS, we set k = √ n σ ε,δ in synthetic dataset experiments and select the best k from { 10 2 , 3 × 10 2 , 10 3 , 3 × 10 3 , 10 4 } in real-world dataset exper iments.

Original: Because of the numerical instability of computing
Revised: Because of the numerical instability of computing

Original: Hessian inverse mentioned early, we add small λ · I with λ = 10 − 5 to all Hessian matrices.
Revised: Hessian inverse mentioned early, we add small λ · I with λ = 10 − 5 to all Hessian matrices.

Original: Baseline.
Revised: Baseline.

Original: In addition, we consider the following baselines to help qualify the performance of proposed algorithms.
Revised: In addition, we consider the following baselines to help qualify the performance of proposed algorithms.

Original: • OLS :
Revised: • OLS :

Original: The explicit solution for linear regression given training data ( X, Y ) and serves as the performance’s upper bound for private algorithms, i.e. non-private solution.
Revised: The explicit solution for linear regression given training data ( X, Y ) and serves as the performance’s upper bound for private algorithms, i.e. non-private solution.

Original: • Biased Gaussian mechanism (BGM-OLS) :
Revised: • Biased Gaussian mechanism (BGM-OLS) :

Original: The same data release algorithm in DGM-OLS, but has a different training algorithm.
Revised: The same data release algorithm in DGM-OLS, but has a different training algorithm.

Original: Given a released dataset ( X bgm , Y bgm ) by Gaussian mechanism, BGM-OLS outputs the vanilla ordinary least square solution ˆ w bgm n
Revised: Given a released dataset ( X bgm , Y bgm ) by Gaussian mechanism, BGM-OLS outputs the vanilla ordinary least square solution ˆ w bgm n

Original: =
Revised: =

Original: (cid:16)(cid:0) X bgm (cid:1) (cid:62) X bgm (cid:17) − 1 (cid:0) X bgm (cid:1) (cid:62) Y bgm .
Revised: (cid:16)(cid:0) X bgm (cid:1) (cid:62) X bgm (cid:17) − 1 (cid:0) X bgm (cid:1) (cid:62) Y bgm .

Original: In other words, it is DGM-OLS without training debiasing.
Revised: In other words, it is DGM-OLS without training debiasing.

Original: Evaluation metric.
Revised: Evaluation metric.

Original: In the experiments on synthetic datasets, we estimate the probability of the (cid:96) 2 distance between the model weights ˆ w n from each algorithm or baseline and the ground truth model weight w ∗ :
Revised: In the experiments on synthetic datasets, we estimate the probability of the (cid:96) 2 distance between the model weights ˆ w n from each algorithm or baseline and the ground truth model weight w ∗ :

Original: P (
Revised: P (

Original: [Equation]
Revised: [Equation]

Original: w ∗ (cid:107) > β ) .
Revised: w ∗ (cid:107) > β ) .

Original: We also evaluate the expectation of the (cid:96) 2 distance between weights for different algorithms:
Revised: We also evaluate the expectation of the (cid:96) 2 distance between weights for different algorithms:

Original: [Equation]
Revised: [Equation]

Original: If an algorithm is asymptotically optimal, we can see both P ( (cid:107) ˆ w n − w ∗ (cid:107) > β ) and E (cid:107) ˆ w n − w ∗ (cid:107) converge towhen n increases.
Revised: If an algorithm is asymptotically optimal, we can see both P ( (cid:107) ˆ w n − w ∗ (cid:107) > β ) and E (cid:107) ˆ w n − w ∗ (cid:107) converge towhen n increases.

Original: For the experiments on real world datasets, we evaluate learned models ˆ w n by the mean squared loss on the test set.
Revised: For the experiments on real world datasets, we evaluate learned models ˆ w n by the mean squared loss on the test set.

Original: 5.2 EVALUATION ON SYNTHETIC DATASETS
Revised: 5.2 EVALUATION ON SYNTHETIC DATASETS

Original: Data generation.
Revised: Data generation.

Original: We deﬁne the feature dimension d = 10 .
Revised: We deﬁne the feature dimension d = 10 .

Original: Each weight value of the ground truth linear model w ∗ is independently sampled from uniform distribution between − 1 /d and 1 /d .
Revised: Each weight value of the ground truth linear model w ∗ is independently sampled from uniform distribution between − 1 /d and 1 /d .

Original: A single data point ( x , y ) is sampled as the following: each feature value in x is independently sampled from a uniform distribution between − 1 and 1 ; label y is computed as ( w ∗ ) (cid:62) x .
Revised: A single data point ( x , y ) is sampled as the following: each feature value in x is independently sampled from a uniform distribution between − 1 and 1 ; label y is computed as ( w ∗ ) (cid:62) x .

Original: Two assumptions for the data distribution P , Assumption 2 and Assumption 3, can be veriﬁed.
Revised: Two assumptions for the data distribution P , Assumption 2 and Assumption 3, can be veriﬁed.

Original: Moreover, we set 6 parties in total, 5 of which have 2 attributes and the remaining one has 1 attribute.
Revised: Moreover, we set 6 parties in total, 5 of which have 2 attributes and the remaining one has 1 attribute.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (cid:18)(cid:16)(cid:18)(cid:18)(cid:16)(cid:20)(cid:18)(cid:16)(cid:22)(cid:18)(cid:16)(cid:24)(cid:18)(cid:16)(cid:26)(cid:19)(cid:16)(cid:18)
Revised: − w (cid:107) > does not show such even training set size n is as large as

Original: [Figure]
Revised: [Figure]

Revised: (cid:3228)(cid:22) (cid:20)(cid:19)

Original: [Figure]
Revised: [Figure]

Original: Minimum Absolute Eigenvalues of
Revised: (cid:20)(cid:19)

Original: [Figure]
Revised: [Table]

Original: further lead to large distance (cid:107) ˆ w n − w ∗ (cid:107) 22 .
Revised: [Table]

Original: Results.
Revised: [Table]

Revised: (cid:691) (cid:43) (cid:81) (cid:20)(cid:19) (cid:19) (cid:20)(cid:19) (cid:20) (cid:20)(cid:19) (cid:21) (cid:95)(cid:95) (cid:691) (cid:90) (cid:81) (cid:3228) (cid:90) (cid:13) (cid:95)(cid:95) (cid:53)(cid:48)(cid:42)(cid:48)(cid:16)(cid:50)(cid:47)(cid:54) (cid:39)(cid:42)(cid:48)(cid:16)(cid:50)(cid:47)(cid:54)

Original: We vary the training set size n ∈ { 10 4 , 3 × 10 4 , 10 5 , 3 × 10 5 , 10 6 , 3 × 10 6 } and privacy budget ε ∈ { 1 ,  3 , 10 } with ﬁxed δ = 10 − 5 .
Revised: ∈ { 1 , 0 . 3 , 0 . 1 } with ﬁxed δ = 10 − 5 .

Original: We estimate the
Revised: We estimate the

Original: P [ (cid:107) w n − w ∗ (cid:107) > β ] and E (cid:107) w n − w ∗ (cid:107) for different algorithms with 1000 random seeds.
Revised: P [ (cid:107) w n − w ∗ (cid:107) > β ] and E (cid:107) w n − w ∗ (cid:107) for different algorithms with 1000 random seeds.

Original: Figure 2 showswhen ε = 1 . 0 , how P [ (cid:107) w n − w ∗ (cid:107) > β ] and E (cid:107) w n − w ∗ (cid:107) of each algorithm changes when training set size n increases.
Revised: Figure 2 shows how P [ (cid:107) w n − w ∗ (cid:107) > β ] and E (cid:107) w n − w ∗ (cid:107) of each algorithm change when training set size n increases.

Original: We put the results of ε ∈ { 0 . 1 , 0 . 3 } in the appendix, which show the similar conclusion.

Original: Regarding two baselines, P [ (cid:107) w n − w ∗ (cid:107) >
Revised: Regarding two baselines, P [ (cid:107) w n − w ∗ (cid:107) >

Original: β ] of OLS solutions, without any private constraint, are close to the ground truth w ∗ under all β with probability 0.
Revised: β ] of OLS solutions, without any private constraint, are close to the ground truth w ∗ under all β with probability 0.

Original: Nonetheless, P [ (cid:107) w n − w ∗ (cid:107) > β ] of BGM-OLS keeps mostly unchanged as n increases.
Revised: Nonetheless, P [ (cid:107) w n − w ∗ (cid:107) > β ] of BGM-OLS keeps mostly unchanged as n increases.

Original: Especially, P [ (cid:107) w n − w ∗ (cid:107) > 0 . 1] stays at 1 for all n .
Revised: Especially, P [ (cid:107) w n − w ∗ (cid:107) > 0 . 1] stays at 1 for all n .

Original: Such results are expected in BGM-OLS’s convergence: plimn →∞ 1 n (cid:0) X bgm (cid:1) (cid:62) X bgm =
Revised: Such results are expected in BGM-OLS’s convergence: plim

Original: E x (cid:2) xx (cid:62) (cid:3) + 4 d max σ 2 ε,δ · I , which introduces a non-diminishing bias 4 d max σ 2 ε,δ · I .
Revised: [Equation] · I , which introduces a non-diminishing bias [Equation]

Original: Next, we compare DGM-OLS and RMGM-OLS.
Revised: Next, we compare DGM-OLS and RMGM-OLS.

Original: RMGMOLS outperforms DGM-OLS at both the convergence of probability P [ (cid:107) w n − w ∗ (cid:107) > β ] (the ﬁrst three ﬁgures in
Revised: RMGMOLS outperforms DGM-OLS at both the convergence of probability P [ (cid:107) w n − w ∗ (cid:107) > β ] (the ﬁrst three ﬁgures in

Original: Figure 2) and the expected distance E [ (cid:107) w n − w ∗ (cid:107) ] (the last ﬁgure in Figure 2).
Revised: Figure 2) and the expected distance E [ (cid:107) w n − w ∗ (cid:107) ] (the last ﬁgure in Figure 2).

Original: RMGM-OLS shows the asymptotic tendencies in all values of β when ε = 1 . 0 .
Revised: RMGM-OLS shows the asymptotic tendencies in all values of β when ε = 1 . 0 .

Original: Although DGMOLS has better rate at n than RMGM-OLS theoretically, n = 3 × 10 6 is not large enough to show the asymptotic tendencies for DGM-OLS.
Revised: Although DGMOLS has better rate at n than RMGM-OLS theoretically, n = 3 × 10 6 is not large enough to show the asymptotic tendencies for DGM-OLS.

Original: DGM-OLS is even much worse than BGM-OLS, which is almost random guess.
Revised: DGM-OLS is even much worse than BGM-OLS, which is almost random guess.

Original: It is caused by the small eigenvalue issue discussed in section 4.
Revised: It is caused by the small eigenvalue issue discussed in section 4.

Original: To illustrate it, Figure 3  shows the scatter plot, where the x -axis is minimum eigenvalues of the Hessian matrix ˆ H n
Revised: To illustrate it, Figure 3 (a) shows the scatter plot, where the x -axis is minimum eigenvalues of the Hessian matrix ˆ H n

Original: and y -axis is the distance between our solutions and the optimal solution (cid:107) ˆ w n − w ∗ (cid:107) .
Revised: and y -axis is the distance between our solutions and the optimal solution (cid:107) ˆ w n − w ∗ (cid:107) .

Original: Each point is processed by a different random seed for DGMOLS and BGM-OLS when n = 10 6 and ε = 1 . 0 .
Revised: Each point is processed by a different random seed for DGM-OLS and BGM-OLS when n = 10 6 and ε = 1 . 0 .

Original: (cid:107) ˆ w n − w ∗ (cid:107) and the minimum absolute eigenvalues of ˆ H n have a strong positive correlation.
Revised: (cid:107) ˆ w n − w ∗ (cid:107) and the minimum absolute eigenvalues of ˆ H n have a strong positive correlation.

Original: With a certain probability, the minimum eigenvalue of DGM-OLS is smaller than 10 −and corresponding (cid:107) w n − w ∗ (cid:107) is larger than 10 .
Revised: With a certain probability, the minimum eigenvalue of DGM-OLS is smaller than 10 − 2 and corresponding (cid:107) w n − w ∗ (cid:107) is larger than 10 .

Original: Overall RMGM-OLS has the best empirical performance across various settings of ε and n on the synthetic data, as its asymptotically optimality is veriﬁed and it consistently outperforms two other private algorithms when n is large enough.
Revised: Overall RMGM-OLS has the best empirical performance across various settings of ε and n on the synthetic data, as its asymptotically optimality is veriﬁed and it consistently outperforms two other private algorithms when n is large enough.

Original: Though DGM-OLS seems to have stronger theoretical guarantee in the aspect of rate in n , its poor empirical performance comes from two aspects: 1.
Revised: Though DGM-OLS seems to have stronger theoretical guarantee in the aspect of rate in n , its poor empirical performance comes from two aspects: 1.

Original: small eigenvalues occur due to the design of the training algorithm; 2. extremely large n is necessary to show the asymptotic optimality due to the worse rates of d, d max and σ ε,δ .
Revised: small eigenvalues occur due to the design of the training algorithm; 2. extremely large n is necessary to show the asymptotic optimality due to the worse rates of d, d max and σ ε,δ .

Original: 5.3 EVALUATION ON REAL WORLD DATASETS
Revised: 5.3 EVALUATION ON REAL WORLD DATASETS

Original: Dataset.
Revised: Dataset.

Original: We experiment with ﬁve datasets:
Revised: We experiment with ﬁve datasets:

Original: • Insurance
Revised: • Insurance

Original: [Lantz, 2019]: predicting the insurance premium from features including age, bmi, expenses, etc.
Revised: [Lantz, 2019]: predicting the insurance premium from features including age, bmi, expenses, etc.

Original: • Bike [Fanaee-T and Gama, 2014]: predicting the count of rental bikes from features such as season, holiday, etc.
Revised: • Bike [Fanaee-T and Gama, 2014]: predicting the count of rental bikes from features such as season, holiday, etc.

Original: • Superconductor [Hamidieh, 2018]: predicting critical temperature from chemical features.
Revised: • Superconductor [Hamidieh, 2018]: predicting critical temperature from chemical features.

Original: • GPU
Revised: • GPU

Original: [Ballester-Ripoll et al., 2019, Nugteren and Codreanu, 2015]: predicting Running time for multiplying two 2048 × 2048 .
Revised: [Ballester-Ripoll et al., 2019, Nugteren and Codreanu, 2015]: predicting Running time for multiplying two 2048 × 2048 .

Original: matrices using a GPU OpenCL SGEMM kernel with varying parameters.
Revised: matrices using a GPU OpenCL SGEMM kernel with varying parameters.

Original: • Music Song [Bertin-Mahieux et al., 2011]: predicting the[Table] release year of a song from audio features.
Revised: • Music Song [Bertin-Mahieux et al., 2011]: predicting the release year of a song from audio features.

Original: We split the original dataset into train and test by the ratio 4 :1 .
Revised: We split the original dataset into train and test by the ratio 4 :

Original: The number of training data n , the number of features d and the number of parties are listed in Table 1.
Revised: The number of training data n , the number of features d and the number of parties are listed in Table 1.

Original: The attributes are evenly distributed among parties.
Revised: The attributes are evenly distributed among parties

Original: All features and labels are normalized into[0 , 1] .
Revised: All features and labels are normalized into

Original: Results.
Revised: Results.

Original: For each dataset, we evaluate OLS and three differentially private algorithms by the mean squared loss on the test split.
Revised: For each dataset, we evaluate OLS and three differentially private algorithms by the mean squared loss on the test split.

Original: Table 1 shows the results for ε ∈ {  1 ,  3 , 10 } and δ = 10 − 5 .
Revised: Table 1 shows the results for ε ∈ { 0 . 1 , 0 . 3 , 1 . 0 } and δ = 10 − 5 .

Original: We can check that the loss of DGM-OLS is usually much larger than others and RMGM-OLS achieves the lowest losses for most cases (12 out of 15).
Revised: We can check that the loss of DGM-OLS is usually much larger than others and RMGMOLS achieves the lowest losses for most cases (12 out of 15).

Revised: Moreover, Figure 3 (b) shows that DGM-OLS has the small eigenvalue problem as well in the real world dataset experiments.

Original: This result is consistent with the result on synthetic dataset.
Revised: These results are consistent with the results on synthetic dataset.

Original: We therefore recommend RMGM-OLS as a practical solution to privately release the dataset and build the linear regression models.
Revised: We therefore recommend RMGM-OLS as a practical solution to privately release the dataset and build the linear regression models.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Differentially private dataset release.
Revised: Differentially private dataset release.

Original: Many recent works [Sheffet, 2017, Gondara and Wang, 2020, Xie et al., 2018, Jordon et al., 2018, Lee et al., 2019, Xu et al., 2017,
Revised: Many recent works [Sheffet, 2017, Gondara and Wang, 2020, Xie et al., 2018, Jordon et al., 2018, Lee et al., 2019, Xu et al., 2017,

Original: Kenthapadi et al., 2012] study the differentially private data release algorithms.
Revised: Kenthapadi et al., 2012] study the differentially private data release algorithms.

Original: However, those algorithms either only serve for data release from a single-party [Sheffet, 2017,
Revised: However, those algorithms either only serve for data release from a single-party [Sheffet, 2017,

Original: Gondara and Wang, 2020], or focus on the feature dimension reduction or empirical improvement [Lee et al., 2019,
Revised: Gondara and Wang, 2020], or focus on the feature dimension reduction or empirical improvement [Lee et al., 2019,

Original: Xu et al., 2017, Kenthapadi et al., 2012], which is orthogonal to the study of asymptotical optimality w.r.t.
Revised: Xu et al., 2017, Kenthapadi et al., 2012], which is orthogonal to the study of asymptotical optimality w.r.t.

Original: dataset size.
Revised: dataset size.

Original: In
Revised: In

Original: Sheffet [2017] and Gondara and Wang [2020], the random
Revised: Sheffet [2017] and Gondara and Wang [2020], the random

Original: Gaussian projection matrices in their method contribute to the differential privacy guarantee, hence the sharing of projection matrix would violate the privacy guarantee between parties.
Revised: Gaussian projection matrices in their method contribute to the differential privacy guarantee, hence the sharing of projection matrix would violate the privacy guarantee between parties.

Original: Nevertheless, without sharing this projection matrix, the utility cannot be guaranteed anymore.
Revised: Nevertheless, without sharing this projection matrix, the utility cannot be guaranteed anymore.

Original: In Xie et al.
Revised: In Xie et al.

Original: [2018] and Jordon et al.
Revised: [2018] and Jordon et al.

Original: [2018], they train a differentially private GAN.
Revised: [2018], they train a differentially private GAN.

Original: However, it is not obvious to rigorously privately share data information during their training when each party holds different attributes but same instances.
Revised: However, it is not obvious to rigorously privately share data information during their training when each party holds different attributes but same instances.

Original: Lee et al.
Revised: Lee et al.

Original: [2019] proposes a random mixing method and also analyzes the linear model.
Revised: [2019] proposes a random mixing method and also analyzes the linear model.

Original: However, the way they mix only works for realizable linear data.
Revised: However, the way they mix only works for realizable linear data.

Original: It is not able to be extended to the general linear regression and the asymptotic optimality guarantee.
Revised: It is not able to be extended to the general linear regression and the asymptotic optimality guarantee.

Original: Xu et al.
Revised: Xu et al.

Original: [2017] and Kenthapadi et al.
Revised: [2017] and Kenthapadi et al.

Original: [2012] focus on the feature dimension reduction, which is orthogonal to the study of asymptotical optimality w.r.t.
Revised: [2012] focus on the feature dimension reduction, which is orthogonal to the study of asymptotical optimality w.r.t.

Original: dataset size.
Revised: dataset size.

Original: Asymptotically optimal differentially private convex optimization.
Revised: Asymptotically optimal differentially private convex optimization.

Original: A large amount of work study differentially private optimization for convex problems [Bassily et al., 2014, 2019, Feldman et al., 2020] or particularly for linear regression [Sheffet, 2017, Kasiviswanathan et al., 2011,
Revised: A large amount of work study differentially private optimization for convex problems [Bassily et al., 2014, 2019, Feldman et al., 2020] or particularly for linear regression [Sheffet, 2017, Kasiviswanathan et al., 2011,

Original: Chaudhuri and Hsu, 2012].
Revised: Chaudhuri and Hsu, 2012].

Original: They mainly differ from our work in the sense that their goal is to release the ﬁnal model while ours is to release the dataset.
Revised: They mainly differ from our work in the sense that their goal is to release the ﬁnal model while ours is to release the dataset.

Original: Linear regression in vertical federated learning.
Revised: Linear regression in vertical federated learning.

Original: Linear regression is a fundamental machine learning task.
Revised: Linear regression is a fundamental machine learning task.

Original: Hall et al.
Revised: Hall et al.

Original: [2011], Nikolaenko et al.
Revised: [2011], Nikolaenko et al.

Original: [2013], Gascón et al.
Revised: [2013], Gascón et al.

Original: [2017] studying linear regression over vertically partitioned datasets based on secure multi-party computation.
Revised: [2017] studying linear regression over vertically partitioned datasets based on secure multi-party computation.

Original: However, cryptographic protocols such as Homomorphic Encryption [Hall et al., 2011, Nikolaenko et al., 2013] and garbled circuits [Nikolaenko et al., 2013, Gascón et al., 2017] lead to heavy overhead on computation and communication.
Revised: However, cryptographic protocols such as Homomorphic Encryption [Hall et al., 2011, Nikolaenko et al., 2013] and garbled circuits [Nikolaenko et al., 2013, Gascón et al., 2017] lead to heavy overhead on computation and communication.

Original: From this aspect, DP-based techniques are more practical.
Revised: From this aspect, DP-based techniques are more practical.

Revised: [Table]

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We propose and analyze two differentially private algorithms under multi-party setting for linear regression, and theoretically both of them are asymptotically optimal with increasing dataset size.
Revised: We propose and analyze two differentially private algorithms under multi-party setting for linear regression, and theoretically both of them are asymptotically optimal with increasing dataset size.

Original: Empirically, RMGM-OLS has the best performance on both synthetic datasets and real-world datasets, while extremely large training set size n is necessary for DGM-OLS.
Revised: Empirically, RMGM-OLS has the best performance on both synthetic datasets and real-world datasets, while extremely large training set size n is necessary for DGM-OLS.

Original: We hope our work can bring more attention to the need for multi-party data release algorithms and we believe that ML practitioners would beneﬁt from such effort in the era of privacy.
Revised: We hope our work can bring more attention to the need for multi-party data release algorithms and we believe that ML practitioners would beneﬁt from such effort in the era of privacy.

Original: Future work.
Revised: Future work.

Original: We focus on linear regression only, and one future direction is to extend our algorithm to classiﬁcation, e.g. logistic regression, while achieving the same asymptotic optimality.
Revised: We focus on linear regression only, and one future direction is to extend our algorithm to classiﬁcation, e g. logistic regression, while achieving the same asymptotic optimality.

Original: In addition, we assume different parties own the same set of data subjects.
Revised: In addition, we assume different parties own the same set of data subjects.

Original: Another future direction is to relax this assumption: the set of subjects owned by different parties might be slightly different.A PROOFS OF COROLLARY IN SECTION 2 AND USEFUL LEMMAS
Revised: Another future direction is to relax this assumption: the set of subjects owned by different parties might be slightly different.

Original: Lemma 1 (Gaussian mechanism) .

Original: For any deterministic real-valued function f : D → R m with sensitivity S f , we can deﬁne a randomized function by adding Gaussian noise to f :

Original: [Equation]

Original: where N (cid:16) 0 , S 2 f σ 2 · I (cid:17) is a multivariate normal distribution with mean 0 and co-variance matrix S 2 f σ 2 multiplying a m × m identity matrix I .

Original: When σ ≥ √ 2log(1 / (1 . 25 δ ))

Original: ε , f dp is ( ε, δ ) -differentially private.

Original: Lemma 2 (JL Lemma for inner-product preserving (Bernoulli)) .

Original: Suppose S be an arbitrary set of l points in R d and suppose s is an upper bound for the maximum L2-norm for vectors in S .

Original: Let B be a k × d random matrix, where B ij are independent random variables, which take value 1 and value − 1 with probability 1 / 2 .

Original: With the probability at least 1 − ( l + 1) 2 exp

Original: [Equation]

Original: Lemma 3.

Original: ∀ x ∈ [0 , 1] , − log (1 − x ) − x ≥ x 2 2 . 2.

Original: ∀ x ∈ [0 , 1] , x − log (1 + x ) ≥ x 2 4 . 3.

Original: ∀ x > 1 , x − log (1 + x ) ≥ x 2 .

Original: Proof.

Original: Deﬁne

Original: [Equation]

Original: x ≥ 0 .

Original: Thus f 1 ( x ) increases on [0 , 1] and f 1 ( x ) ≥ f 1 (0) = 0 .

Original: Deﬁne f 2 ( x ) :

Original: = x − log (1 + x ) − x 2 4 .

Original: f (cid:48) 2 ( x ) =

Original: x (1 − x ) 2(1+ x ) .

Original: f 2 ( x ) increases on [0 , 1] and f 2 ( x ) ≥ f 2 (0)

Original: = 0 .

Original: Deﬁne f 3 ( x ) := x − log (1 + x ) − x 4 . f (cid:48) 3 ( x ) = 3 x − 1 4(1+ x ) > 0 .

Original: f 3 ( x ) increases on [0 , 1] and f 3 ( x ) ≥ f (1) > 0 .

Original: Lemma 4.

Original: Denote ˆ H n = 1 n X (cid:62) X , ˆ C n = 1 n X (cid:62) Y ,

Original: H =

Original: E ( x ,y ) ∼P (cid:2) xx (cid:62) (cid:3) and C = E ( x ,y ) ∼P

Original: [ x · y ] .

Original: Assume (cid:107) ˆ H pub n − ˆ H n (cid:107) ≤ β, (cid:107) ˆ C pub n − ˆ C n (cid:107) ≤

Original: β

Original: with prob 1 − f ( β ) .

Original: We have that when β ≤ 2 (cid:107) C (cid:107)(cid:107) H − 1 (cid:107) +5,

Original: P X, y ∼D ,R 1 ,R 2 (cid:2) (cid:107) ˆ w pub n − w ∗ (cid:107) ≤ β (cid:3) ≥ 1 − h ( β ) , where ˆ w pub n = (cid:16) ˆ H pub n (cid:17) − 1 ˆ C pub n , c := (cid:107) C (cid:107)(cid:107) H − 1 (cid:107) 2 +2 (cid:107) H − 1 (cid:107) and h ( β ) =

Original: f ( β/ 2 c )+ d 2 exp (cid:16) − nβ 2 8 c 2 d 2 (cid:17)

Original: + d exp (cid:16) − nβ 2 8 c 2 d (cid:17) .

Original: Proof.

Original: Hoeffding inequality and union bound together imply that with prob.

Original: 1 − d 2 exp (cid:16) − nβ 2 2 d 2 (cid:17) − d exp (cid:16) − nβ 2 2 d (cid:17) , (cid:107) ˆ H n − H (cid:107) ≤ β, (cid:107) ˆ C n − C (cid:107) ≤ β.

Original: Thus with prob 1 − g ( δ ) , (cid:107) ˆ H pub n − H (cid:107) ≤ β, (cid:107) ˆ C pub n − C (cid:107) ≤ β , where g ( β ) =

Original: f ( β/ 2) + d 2 exp (cid:16) − nβ 2 8 d 2 (cid:17) + d exp (cid:16) − nβ 2 8 d (cid:17)

Original: We further have

Original: [Equation]

Original: ˆ H [Equation] n
Revised: Acknowledgements

Original: [Equation]

Original: −

Original: [Equation]

Original: ·(cid:107) H −

Original: [Equation]

Original: β , which implies that when β ≤

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: • When β ≤

Original: [Equation]

Original: [Equation]

Original: Let b

Original: [Equation]

Original: =

Original: [Equation]

Original: and replace β by b − 1 β , we have that when β ≤

Original: [Equation]

Original: where h ( β ) = g ( β/b ) = [Equation] [Equation]
Revised: The authors thank Xiaowei Zhang for drawing pictures in

Original: (cid:16)

Original: [Equation]

Original: −

Original: [Equation]

Original: Lemma 5.

Original: If r is a random variable sampled from standard normal distribution, we have following concentration bound:

Original: [Equation]

Original: Proof.

Original: It’s shown in page 2 in Pollard [2015].

Original: Lemma 6.

Original: If r 1 , r 2 are two independent random variables sampled from standard normal distribution, r 1 r 2 can be written as c 1 − c 2 2 , where c 1 , c 2 are independent two random variables sampled from chi-squared with degree 1 .

Original: Moreover, (cid:80) ni

Original: =1 r 1 ,n r 2 ,n can be written as c 1 , 1: n − c 2 , 1: n, where c 1 , 1: n , c 2 , 1: n are independent two random variables sampled from chi-squared with degree n .

Original: Proof.

Original: r 1 r 2 =

Original: [Equation]

Original: Because r 1 , r 2 are two independent standard normal random variables, r 1 + r 2 √ 2 , r 1 − r 2 √are two independent standard normal random variables as well.

Original: c 1 := r 1 + r 2 √ 2 and c 2 :=

Original: r 1 − r 2 √ 2 complete the proof for the ﬁrst part.

Original: (cid:80)

Original: [Equation]

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: ,i and c

Original: [Equation]

Original: , 1: n := (cid:80) ni

Original: =1 c 2 ,i ﬁnish the proof.

Original: B PROOFS IN SECTION

Original: We restate the assumptions and theorems for the completeness.

Original: Assumption 1.

Original: D i , i = 1 , · · · , n , are i.i.d sampled from an underlying distribution P over R d +1 .

Original: Assumption 2.

Original: The absolute values of all attributes are bounded by 1 .

Original: Assumption 3.

Original: E ( x ,y ) ∼P (cid:2) xx

Original: (cid:62) (cid:3) is positive deﬁnite.

Original: Theorem 1.
Revised: Figure 1.

Original: When β ≤ c for some variable c that depends on σ ε,δ , d and P , but independent of n ,

Original: [Equation]

Original: Proof of Theorem 1.

Original: Denote (cid:0) max j ∈ [ m ] d j (cid:1) by d max .

Original: Denote R ∈ R n × d is a random matrix s.t.

Original: R i,j ∼ N (cid:16) 0 , 4 d max σ 2 ε,δ (cid:17) .

Original: We split R into R X and R Y representing the addictive noise to X and Y .

Original: [Equation]

Original: For any i

Original: ∈ [ d ] ,

Original: [Equation]

Original: is sampled from chi-square distribution with degree n. From the cdf of chi-square distribution, we have following concentration:

Original: [Equation]

Original: Moreover, for i (cid:54) =

Original: j , Lemma 6 implies that

Original: [Equation]

Original: i

Original: [Equation]

Original: j can be written as c 1 ,

Original: [Equation]

Original: n − c

Original: [Equation]

Original: , where c 1 , 1: n , c 2 , 1: n are independent two random variables sampled from chi-squared with degree n .

Original: Thus

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: implied by Lemma 5.

Original: 5. Similar to 1,

Original: [Equation]

Original: One can simplify − log (1 − x ) − x and x − log (1 + x ) by Lemma 3.

Original: Set β 1 = 12 β, β 2 = 14 β, β 3 = β 4 = 14 β, β 5 = 12 β .

Original: The above concentrations together imply that when β < 8 dd max σ 2 ε,δ , (cid:107) ˆ H pub n − ˆ H n (cid:107) ≤ β, (cid:107) ˆ C pub n − ˆ C n (cid:107) ≤ β with prob at least 1 − f ( β ) , where f ( β ) =

Original: exp (cid:16) − min (cid:110) O (cid:16) n · β

Original: [Equation]

Original: d

Original: [Equation]

Original: d 2max σ 4 ε,δ (cid:17) + ˜ O

Original: (1) (cid:111)(cid:17) .

Original: [-With the application of Lemma 4: when β ≤ 2 (cid:107) C (cid:107)(cid:107) H − 1 (cid:107) +
-]
Revised: [-With the application of Lemma 4: when β ≤ 2 (cid:107) C (cid:107)(cid:107) H − 1 (cid:107) +
-]

Original: P X

Original: [Equation]

Original: y ∼D

Original: [Equation]

Original: R

Original: [Equation]

Original: R 2 [

Original: [Equation]

Original: n −

Original: [Equation]

Original: ≤

Original: [Equation]

Original: ] ≥ 1 − h ( β ) , where h ( β ) is:

Original: when β < 16 bdd max σ 2 ε,δ , h ( β ) = exp (cid:16) − O (cid:16) n ·

Original: [Equation]

Original: d 2 d 2max σ 4 ε,δ

Original: [Equation]

Original: + ˜ O

Original: (1) (cid:17) ; 2. when β ≥ 16 bdd max σ 2 ε,δ , h ( β ) = exp (cid:16) − O (cid:16) n ·

Original: [Equation]

Original: δ

Original: [Equation]

Original: + ˜ O

Original: (1) (cid:17) ;

Original: where b =

Original: [Equation]

Original: is a distribution dependent constant.

Original: In the other word, when β ≤

Original: [Equation]

Original: Theorem 2.

Original: When β ≤ c for some variable c that depends on d and P

Original: [Equation]

Original: but independent of n and σ ε,δ ,

Original: P [ (cid:107) w rmgm n

Original: [Equation]

Original: If we take k

Original: [Equation]

Original: Proof of Theorem 2.

Original: ˆ w rmgm n

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Deﬁne M := 1 √ k B .

Original: Then we can make the analysis one by one.

Original: JL-lemma applied by Bernoulli random variables implies that with probability 1 − ( d + 2) 2 exp (cid:16) − k (cid:16) β 21 64 d − β

Original: [Table]

Original: √ d (cid:17)(cid:17) , (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: Proof.

Original: Lemma 2 implies that with prob 1 − ( d + 2) 2 exp (cid:16) k (cid:16) β 2 4 − β 3 6 (cid:17)(cid:17) , for any u, v ∈ { X (cid:62) i | i ∈

Original: [ d ] } ∪ { Y } ,

Original: [Equation]

Original: This further implies that

Original: [Equation]

Original: β 1 = 4 √ dβ helps ﬁnish the proof.

Original: JL-lemma applied by Bernoulli random variables implies that with probability

Original: [Table]

Original: [Table]

Original: 1 nX

Original: [Table]

Original: M

Original: [Table]

Original: MX

Original: [Table]

Original: nX

Original: [Table]

Original: X

Original: [Table]

Original: ≤ β 2 . 3.

Original: With prob.

Original: 1 − 2 kd (cid:113) 2 kdd max σ 2 ε,δ πnβexp

Original: [Equation]

Original: − n

Original: [Equation]

Original: [Equation]

Original: ε

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:62) X R X

Original: [Equation]

Original: [Table]

Original: ≤ β 3 .

Original: Proof.

Original: To simplify the proof, let’s assume R X is a standard gaussian matrix.

Original: Because P ( | ( R X ) ij | ) ≤ β ) ≥ 1 −√ 2 πβ exp (cid:0) − β 2 / 2 (cid:1) shown in Lemma 5,

Original: [Equation]

Original: It’s equivalent that

Original: [Equation]

Original: Plug-in the variance of R X leads to the targeted inequality.

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Proof.

Original: Denote c := R (cid:62) X BY n √ k and further c i := ( ( R X ) i ) (cid:62) b √ k , where ( R x ) i is the i th column for R X and b = BYn . [Equation]
Revised: RW and KQW are supported by grants from the National Science Foundation NSF (IIS-2107161, III1526012, IIS- and IIS-1724282) and the Cornell

Original: Then

Original: [Equation]

Original: Similarly,

Original: [Equation]

Original: Union bound gives the conclusion

Original: [Equation]

Original: [Equation] which is implied similar to 4. With prob
Revised: Center for Materials Research with funding from the NSF

Original: [Equation]

Original: Proof.

Original: To simplify the proof, let’s assume R X and R Y is a standard gaussian matrix ﬁrst.

Original: Because P ( | ( R X ) ij | ) ≤ β ) ≥ 1 −

Original: [Equation]

Original: [Equation]

Original: shown in Lemma 5,

Original: [Equation]

Original: It’s equivalent that

Original: [Equation]

Original: Plug-in the variance of R X and R Y leads to the targeted inequality.

Original: Deﬁne

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: The above analysis implies that, with prob.

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: we have [Equation] Let β 2 = 2 β 3 , β 3 = 1 6 , β 5 = β 12 and [-β 1 = β 
 = β 6 = β 4 .-]
Revised: MRSEC program (DMR-1719875) and [-β 1 = β 
 = β 6 = β 4 .-]SAP America.

Original: We will have (cid:107) ˆ H pub n

Original: [Equation]

Original: , (cid:107) ˆ C pub n − ˆ C n (cid:107) ≤ β , with prob.

Original: [Equation]

Original: β

Original: [Equation]

Original: , ∀

Original: [Equation]

Original: ≤ 4 √ d (implies β 1 ≤ √ d and β 2 ≤ d ), where

Original: [Equation]

Original: 48 kd

Original: [Equation]

Original: Lemma 4 implies that for β ≤ min { 8 b √ d, 2 (cid:107) C (cid:107)(cid:107) H − 1 (cid:107) +5} , we have

Original: P [

Original: [Equation]

Original: w n

Original: [Equation]

Original: β ] ≥

Original: [Equation]

Original: − h ( β ) , where h ( β ) is:

Original: [Equation]

Original: where ˜ O (1) includes log terms of n, d, d max , k, β .

Original: If we take k =

Original: O

Original: [Equation]

Original: C EXTRA RESULTS ON SYNTHETIC DATASETS

Original: Figure 4 shows when ε ∈ { 0 . 1 , 0 .

Original: 3 } how P [ (cid:107) w n − w ∗ (cid:107) >

Original: β ] and E (cid:107) w n − w ∗ (cid:107) of each algorithm changes when training set size n increases.

Original: RMGM-OLS shows asymptotic tendencies plim n →∞ P [ (cid:107) ˆ w rmgm n − w ∗ (cid:107) > β ] = 0 for all settings except the most strict setting ( ε, β )

Original: = (0 . 1 , 0 . 1) , while DGM-OLS doesn’t show such tendencies even when training set size n