Hypothesis Testing for Differentially Private Linear Regression
Abstract
In this work, we design differentially private hypothesis tests for the followingproblems in the general linear model: testing a linear relationship and testingfor the presence of mixtures.
The majority of our hypothesis tests are basedon differentially private versions of the F -statistic for the general linear modelframework, which are uniformly most powerful unbiased in the non-private setting.
We also present another test for testing mixtures, based on the differentially privatenonparametric tests of Couch, Kazan, Shi, Bray, and Groce (CCS 2019), which isespecially suited for the small dataset regime.
We show that the differentially private
F -statistic converges to the asymptotic distribution of its non-private counterpart.
As a corollary, the statistical power of the differentially private F -statistic convergesto the statistical power of the non-private F -statistic.
Through a suite of Monte
Carlo based experiments, we show that our tests achieve desired signiﬁcance levelsand have a high power that approaches the power of the non-private tests as weincrease sample sizes or the privacy-loss parameter.
We also show when our testsoutperform existing methods in the literature.
Introduction
Linear regression is one of the most fundamental statistical techniques available to social scientistsand economists (especially econometricians).
One of the goals of performing regression analysis isfor use in decision-making via point estimation (i e., getting a single predicted value for the dependentvariable).
To increase the conﬁdence of decision-makers and analysts in such estimates, it is oftenimportant to also release accompanying uncertainty estimates for the point estimates [18, 19, 20, 6].
In this work, we aim to provide differentially private linear regression uncertainty quantiﬁcationvia the use of hypothesis tests.
Given the realistic possibility of reconstruction, membership andinference attacks [45, 25], we can rely on Differential Privacy (DP), a rigorous approach to quantifyingprivacy loss [24, 22].
The task of DP linear regression is to, given datapoints { ( x i , y i ) } ni =1 , estimate 25 point or uncertainty estimates for linear regression while satisfying differential privacy.
The majorityof our tests will rely on generalized likelihood ratio test F -statistics.
In previous works [41, 48, 13, 3], differentially private estimators for linear regression are exploredand key factors (such as sample size and variance of the independent variable) that affect the accuracyof these estimators are identiﬁed.
The focus of these previous works is for point estimate prediction.
The predictive accuracy of such estimators can be measured in terms of a conﬁdence bound ormean-squared error.
We continue the study of the utility of such estimators for use in uncertaintyquantiﬁcation via hypothesis testing [41].
(See Section B for background on hypothesis testing.)
Earlier work on uncertainty quantiﬁcation for linear regression was done by Sheffet [41], whoconstructed conﬁdence intervals and hypothesis tests based on the t -test statistic, can used to test a linear relationship.
The random projection routine in [41], based on the Johnson–Lindenstrauss(JL) transform, only starts to correctly reject the null hypothesis when the sample size is very large (orthe variables have a large spread).
This observation is also supported by the work of [21].
Furthermore,the random projection routine requires extra parameters (e.g., for specifying the dimensions of therandom matrix).
In our work, we use the F -statistic and our framework can be used to test mixturemodels, amongst other tests.
We provide a general framework for DP tests based on the F -statistic.
In addition, we will consider hypothesis testing for linear regression coefﬁcients on both small andlarge datasets.
For the mixture model tester, we additionally adapt and evaluate a nonparametricmethod, a DP analogue of the Kruskal-Wallis test due to Couch, Kazan, Shi, Bray, and Groce [21],which is especially suited for the small dataset regime.
To the best of our knowledge, our tests arethe ﬁrst to differentially privately detect mixtures in linear regression models, with accompanyingexperimental validation.
1.1 Our Contributions
In this work, we show that for the problem of differentially private linear regression, we can performhypothesis testing for two problems in the general linear model: (1) Testing a Linear Relationship :is the slope of the linear model equal to some constant (e.g., slope is 0)?
(2) Testing for Mixtures :does the population consist of one or more sub-populations with different regression coefﬁcients?
We provide a differentially private analogue of the F -statistic which we, under the general linear model, show converges in distribution to the asymptotic distribution of the F -statistic (Theorem 2.2).
Furthermore, the DP regression coefﬁcients converge, in probability, to the true coefﬁcients(Lemma D.11).
In particular, in Lemma D.11, we show a 1 / p n statistical rate of convergence forthe DP regression coefﬁcients used for our hypothesis tests.
This matches the optimal rate [38].
Wethen use our DP F -statistic and parametric estimates to obtain DP hypothesis tests using a Monte
Carlo parametric bootstrap, following Gaboardi, Lim, Rogers, and Vadhan [30].
The Monte Carloparametric bootstrap is used to ensure that our tests achieve a target signiﬁcance level of ↵ (i.e., datagenerated under the null hypothesis is rejected with probability ↵ ).
We experimentally comparethese tests to their non-private counterparts for univariate linear regression (i.e., one independentvariable and one dependent variable).
To the best of our knowledge, our tests are the ﬁrst that use the 63 F -statistic to perform tests on the problem of linear regression while ensuring privacy of the datasubjects.
In addition, our F -statistic based tests can be adapted to work on design matrices in anydimension i.e., the design matrix can be cast in the form X 2 R n ⇥ p for any integer p  2 where 66 n represents the number of individuals in the dataset and p represents the number of features perindividual; we leave experimental evaluation of the multivariate case for future work.
Experimental evaluation of our hypothesis tests is done on: Synthetic Data : We generate synthetic datasets with different distributions on the independent (or explanatory) variables.
Speciﬁcally, we consider uniform, normal, and exponentialdistributions on the independent variables.
We also vary the noise distribution of thedependent variable.2.
Opportunity Insights (OI) Data : We use a simulated version of the data used by theOpportunity Insights team (an economics research lab) to release the Opportunity Atlas tool,primarily used to predict social and economic mobility.
We chose to use these datasets sincethey come from a real deployment of privacy-preserving statistics.
The census tract-leveldatasets from these states can have a very small number of datapoints.3.
Bike Sharing Dataset : We use a real-world dataset publicly available in the UCI machinelearning repository.
The dataset consists of daily and hourly counts (with other informationsuch as seasonal and weather information) of bike rentals in the Capital bikeshare system inyears 2011 and 2012.
Our experimental ﬁndings are as follows:
Signiﬁcance and Power :
Across a variety of experimental settings, the signiﬁcance is
below the target signiﬁcance level of 0.05.
Thus, we have a high conﬁdence that we will not falsely reject the null hypothesis, should the null hypothesis be true.
Our tests are designedto be conservative in the sense that they err on the side of failing to reject the null (e.g., whenthe DP estimate of a variance is negative).
Consistently, the power of our tests increase aswe increase sample sizes (from hundreds up to tens of thousands) or as we relax the privacyparameter.
The behavior of our DP tests tends toward that of the non-DP tests.
The powerof the DP OLS linear relationship tester increases as the slope of the model increases and asthe noise in the dependent variable decreases.
But when the DP estimate of the noise in thedependent variable is negative, the tests err on the side of failing to reject the null, leadingto a lower power.
The power of the mixture model tests also increases as the differencebetween the slopes in the two groups increases.
And the more uneven the group sizes are,the lower the power.2. Alternative Methods : (1) We compare our DP linear relationship tester, based on the F -statistic, to a test that builds on a DP parametric bootstrap method for conﬁdence intervalestimation.
Ferrando, Wang, and Sheldon [29] prove that these intervals are consistent (inthe asymptotic regime) and experimentally show that these intervals have good coverage.
We can rely on such parametric bootstrap methods to decide to reject or fail to reject thenull hypothesis.
Such methods achieve the desired signiﬁcance levels.
However, we observethat the method is less powerful than the F -statistic approach.
This behavior could beattributed to the differences in the bootstrap process: whereas we use estimates of sufﬁcientstatistics under the null in the bootstrap procedure, [29] uses the entirety of the sufﬁcientstatistics estimated for the parametric model.
The DP nonparametric method we provided,adapting the work of Couch, Kazan, Shi, Bray, and Groce [21], has a higher power in thesmall dataset regime than the DP F -statistic method.
As the dataset size increases or thedifference in slopes between the groups increases, the gap closes.
In addition, as the varianceof the independent variable increases, the F -statistic method outperforms the nonparametricmethod.
This observation echos ﬁndings in previous work [3] for the case of predictionrather than hypothesis testing.
In that work, it is shown that parametric methods for linearregression perform well in prediction as n , the variance of the independent variable, or theprivacy loss parameter increase.
1.2 Overview of Techniques
We provide Algorithm 1, a generic framework for DP Monte Carlo tests via a parametric bootstraproutine for estimating sufﬁcient statistics.
Algorithm 1 crucially relies on DPStats , a procedure thatuses statistics of the independent and dependent variables to produce DP statistics.
These statisticscan be used to decide to reject or fail to reject the null hypothesis.
DPStats must satisfy DP—in ourcase ⇢
-zCDP (Zero-Concentrated Differential Privacy) [12].
Algorithms 4 and 5 are SSP (Sufﬁcient
Statistic Perturbation) implementations of DPSS for the linear relationship tester and mixture modeltester, respectively.
(See Section C.2 for more details.)
Algorithm 6 is a DP nonparametric testframework based on the Kruskal-Wallis test statistic for testing mixtures.
We use the standard
Gaussian mechanism to make these algorithms satisfy zCDP
although the Laplace mechanism couldbe used instead (i.e., for pure DP).
To make Algorithm 6 DP, we randomly pair datapoints so thatthe resulting transformation is 1-Lipschitz (See Section C.3 for more details.).
Theorem 2.2 showsthat the DP F -statistic converges to the asymptotic distribution of its non-private counterpart, thechi-squared distribution.
As a corollary, the statistical power of the DP F -statistic converges to thestatistical power of the non-private F -statistic.
To prove this result, we ﬁrst show the 1 / p n statisticalrate of convergence for the DP coefﬁcients.
Then we reformulate both the F -statistic and the DP 130 F -statistic in terms of the 1 / p n convergent quantities, including convergent functions of the Gram
Matrix of the independent variable.
Next, we construct random variables whose ` 2 -norm squaredis distributed as the (non-central) chi-squared distribution.
Then we prove that DP analogues of thesquared ` 2 -norm of such random variables converge to the (non-central) chi-squared distribution.
Finally, using such random variables, the reformulated DP F -statistic, reformulated F -statistic, andthe continuous mapping theorem, we combine the convergent quantities to prove Theorem 2.2.
See
Section D for more details.
1.3 Other Related Work
Differentially Private Linear Regression Sheffet [41] considers hypothesis testing for ordinary 139 least squares for a speciﬁc test: testing for a linear relationship under the assumption that the independent variable is drawn from a normal distribution.
Wang [48] focuses on using adaptivealgorithms for linear regression prediction.
M -estimators [34], motivated by the ﬁeld of robuststatistics [35], are a simple class of statistical estimators that present a general approach to parametricinference.
Dwork and Lei [23, 38] and Chaudhuri and Hsu [17] present differentially private Mestimators with near-optimal statistical rates of convergence.
Avella-Medina [7] generalizes the 145 M -estimator approach to differentially private statistical inference using an empirical notion ofinﬂuence functions to calibrate the Gaussian mechanism.
Alabi, McMillan, Sarathy, Smith, and
Vadhan [3] proposed median-based estimators for linear regression and evaluated their performancefor prediction.
All of these previous works show connections between robust statistics, M -estimators,and differential privacy.
The ordinary least squares estimator is a classical M -estimator for prediction.
Other examples include sample quantiles and the maximum likelihood estimation (MLE) objective.
However, for differentially private hypothesis testing, as we show in this work, the optimal teststatistic for DP linear regression depends on statistical properties of the dataset (such as variance andsample size).
We also present novel differentially private test statistics that converge in distribution tothe asymptotic distribution of the F -statistic.
Bernstein and Sheldon [10] take a Bayesian approach tolinear regression prediction and credible interval estimation.
Through the Bayesian lens, there is alsowork on how to approximately bias-correct some DP estimators while providing some uncertaintyestimates in terms of (private) standard errors [27].
As a motivation for differentially private simplelinear regression point and uncertainty estimation, Bleninger, Dreschsler, Ronning [11] show how anattacker could use background information to reveal sensitive attributes about data subjects used in asimple linear regression analysis.
Gaboardi, Lim, Rogers, and Vadhan [30] 162 study hypothesis testing subject to differential privacy constraints.
The tests they consider are: (1) goodness-of-ﬁt tests on multinomial data to determine if data was drawn from a multinomialdistribution with a certain probability vector, and (2) independence tests for checking whether twocategorical random variables are independent of each other.
Both tests use the chi-squared teststatistic.
Rogers and Kifer [40] develop new test statistics for differentially private hypothesistesting on categorical data while maintaining a given Type I error.
Through the use of the subsampleand-aggregate framework, Barrientos et al.
[9] compute univariate t -statistics by ﬁrst partitioningthe data into M disjoint subsets, estimating the statistic on each subset, truncating the statistic atsome threshold a , and then adding noise from a Laplace distribution to the average of the truncated 171 t -statistics.
Our framework requires a clipping parameter (similar to a ) but does not require anyothers (e.g., the number of subsets).
As noted in that work, the parameter M plays a signiﬁcant rolein the performance—and tuning—of their tests while our tests require no such parameter tuning onpartitioning of the data.
A subset of previous work [46, 14, 44, 21] focus on differentially privateindependence tests between a categorical and continuous variables.
Some of these works producenonparametric tests which require little or no distributional assumptions on the data generationprocess.
Speciﬁcally, Couch, Kazan, Shi, Bray, and Groce [21] develop DP analogues of rank-basednonparametric tests such as Kruskal-Wallis and Mann-Whitney signed-rank tests.
The Kruskal-Wallistest, for example, can be used to determine whether the medians of two or more groups are thesame.
We adapt their test to the setting of linear regression by using it to compare the distributionsof slopes between the two groups.
Wang, Lee, and Kifer [49] develop DP versions of likelihoodratio and chi-squared tests, showing a modiﬁed equivalence between chi-squared and likelihoodratio tests.
In the space of differentially private hypothesis testing, previous work introduce methodsfor differentially private identity and equivalence testing of discrete distributions [1, 2, 4, 5].
Adifferentially private version of the log-likelihood ratio test for the Neyman-Pearson lemma has alsobeen shown to exist [15].
Furthermore, Awan and Slavkovic [8] derive uniformly most powerful DPtests for simple hypotheses for binomial data.
Suresh [43] proposes a hypothesis test, which canbe made to satisfy differential privacy, that is robust to distribution perturbations under Hellingerdistance.
Sheffet and Kairouz, Oh, Viswanath [42, 36] also consider hypothesis testing, although inthe local setting of differential privacy.
1.4 Differential Privacy
For the deﬁnitions below, we say that two databases x and x 0 are neighboring, expressed notationallyas d ( x , x 0 )
= 1 for any x , x 0 2 X n , if x differs from x 0 in exactly one row.
Deﬁnition 1.1 (Differential Privacy [24, 22]) .
Let M : X n !
R be a (randomized) mechanism.
For any ✏  0 ,  2 [0 , 1] , we say M is ( ✏ ,  ) -differentially private if for all neighboring databases x , x 0 2 X n , d ( x , x 0 )
= 1 and every S ✓ R ,
[Equation]
If  = 0 , then we say M is ✏ -DP, sometimes referred to as pure differential privacy.
Typically, ✏ is a 195 small constant (e.g., ✏ 2 [0 . 1 , 1] ) and   1 / poly( n ) is cryptographically small.
Deﬁnition 1.2 (Zero-Concentrated Differential Privacy (zCDP) [12]) .
Let M : X n !
R be a (randomized) mechanism.
For any neighboring databases x , x2 X n , d ( x , x 0 ) = 1 , we say M satisﬁes ⇢
-zCDP if for all ↵
[Equation] where D ↵ ( M ( x ) k M ( x 0 )) is the Rényi divergence of order ↵ between the distribution of M ( x ) and 197 the distribution of M ( x 0 ) . In this paper, we will primarily use ⇢ -zCDP as our deﬁnition of differential privacy, adding noisefrom a Gaussian distribution to ensure zCDP.
Differentially Private Linear Regression Testing
The goal of hypothesis testing is to infer, based on data, which of two hypothesis, H 0 (the nullhypothesis) or H 1 (the alternative hypothesis), should be rejected.
Let P ✓ be a family of probabilitydistributions parameterized by ✓ 2 ⌦ .
For some unknown parameter ✓ 2 ⌦ , let Z ⇠ P ✓ be the 204 observed data.
Then the two competing hypothesis are: H 0 : ✓ 2 ⌦ 0 vs. H 1 : ✓ 2 ⌦ 1 , where 205 ( ⌦ 0 , ⌦ 1 ) form a partition of ⌦ .
We will consider hypothesis testing in the linear model Y =
X  + e , where X 2 R n ⇥ p is a matrixof known constants,  2 R p is the parameter vector that determines the linear relationship between 208 X and the dependent variable Y , and e is a random vector such that for all
i [Equation] ] , E [ e i ] = 0 , 209 var[ e i ]
=  2 e .
Furthermore, for all i 6 =
j 2 [ n ] , cov( e i , e j )
= 0 .
Note that the simple linear regression model, y i
=  2 +  1 · x i + e i for scalars x i , y i and e i {+
+} i 2 [ n ] , 211 can be cast as a linear model as follows:
X [Equation] n ⇥ 2 where the ﬁrst column is an all-ones vector and 212 the second column ( x 1 , . . .
, x n ) T .
We will consider the general linear model:
Y ⇠ N ( X  ,  2 e I n ⇥ n ) , where I n ⇥ n is the n ⇥ n identity 214 matrix.
Let ! be an r -dimensional linear subspace of R p and !
be a q -dimensional linear subspaceof !
such that 0  q < r .
We will consider hypothesis tests of the form: (i) H 0 :  2 ! 0 ; (ii) H 1 : 216  2 !
! 0 .
Let ˆ  and ˆ  N denote the least squares estimates under the alternative and null hypothesis respectively.
In other words,
[Equation]
The test statistic we will use is equivalent to the generalized likelihood ratio test statistic
[Equation]
where S 2 = k Y
X ˆ  k 2 / ( n  r ) .
The vectors Y
X ˆ  and X ˆ
X ˆ  N can be shown to beorthogonal,
so that k Y  X ˆ  N k 2 = k Y
X
ˆ  k 2 + k X ˆ
X ˆ  N k 2 by the Pythagorean theorem [37].
To design a Monte Carlo hypothesis test, we follow a similar route to Gaboardi, Lim, Rogers, and
Vadhan [30].
In Algorithm 1, we provide a framework to perform DP Monte Carlo tests using aparametric bootstrap based on a test statistic.
Let DPStats be a procedure that uses one or more statistics of X, Y to produce DP statistics thatcan be used to reject or fail to reject the null hypothesis.
In this paper, DPStats will satisfy ⇢ -zCDP(Zero-Concentrated Differential Privacy).
T is the test statistic computation procedure.
As donein [30], for example, we will assume the dataset sizes are public information.
Let T
=
T (ˆ ✓ 1 ) be the non-private test statistic procedure given ˆ ✓ 1 = ˆ ✓ 1 ( X, Y ) .
The goal is tocompute T (˜ ✓ 1 ) where ˜ ✓ 1 is an approximation of ˆ ✓ 1 .
DPStats returns ˜ ✓ 0 and ˜ ✓ 1 .
If ˜ ✓ 0 and ˜ ✓ 1 is not ?
( ? is returned whenever the perturbed statistics cannot be used to simulate the null distributions), thenwe use ˜ ✓ 1 to compute the DP test statistic and ˜ ✓ 0 to simulate the null.
P ˜ ✓ 0 represents the distributionfrom which we will sample from to simulate the null distribution.
When ( X, y )
⇠ P ˜ ✓ 0 for ✓ 0 2 ⌦ 0and we set ˜ ✓ 1 = ˜
✓ 1 ( X, Y ) and sample ( X 0 , y 0 )
⇠ P ˜ ✓ 0 , then ˆ ✓ 1 (( X 0 , y 0 )) has approximately the samedistribution as ˆ ✓ 1 (( X, y )) .
˜ T will denote the test statistic applied to the DP sufﬁcient statistics.
9: Select do 11:
[Equation]
[Equation] [Equation] Obtain t k from T ( k ˜ ✓ 1 ) 14: end for 15:
( K ) 16:
Compute threshold t ( r ) where r = d ( K + 1)(1  ↵ ) e 17: if ˜ t > t ( r ) then 18: return Reject the null 19: else 20: return Fail to Reject the null 21: end if
2.1 Differentially Private Algorithms
Lemma 2.1 (Lemma C.1) .
For any ⇢ ,  > 0 , Algorithm 2 satisﬁes ⇢ -zCDP.
In Algorithm 4, we give the full ⇢ -zCDP procedure for computing all necessary sufﬁcient statistics tocompute ˜ T L ( X, Y, ˆ  , ˆ  N , n, r, q, ⇢ ,  ) .
Instantiating Algorithm 1 for the Linear Tester:
If the procedure DPStats L returns ( ? , ? ) ,then we fail to reject the null.
Otherwise, we use the returned statistics ˜ ✓ 1 =
(˜  1 , ˜¯ x, f x 2 , f
S 2 , n ) tocreate the test statistic T (˜ ✓ 1 ) =
˜
2 1 · n · ( f x 2  ˜ ¯ x 2 ) f
Sand use ˜ ✓ 0 = (˜¯ y, ˜¯ x, f x 2 , f
S 20 , n ) to simulate the nulldistributions (to decide to reject or fail to reject the null hypothesis).
P ˜ ✓ 0 is instantiated as a normaldistribution and used to generate k x i distributed as N (˜¯ x, ( n f x 2  n ˜¯ x 2 ) / ( n  1)) and k y i as ˜  2 + e i ,e i ⇠ N (0 , f
S 20 ) for all
i 2 [ n ] .
Instantiating Algorithm 1 for the F -statistic Mixture Tester:
If the procedure DPStats M re- 245 turns ( ? , ? ) , then we fail to reject the null.
Otherwise, we use the returned statistics˜ ✓ 1 =
(˜  1 , ˜  2 , g x 21 , g x 22 , f x 2 , f
S 2 , n 1 , n 2 , n ) to create the test statistic and use˜ ✓ 0 = (˜  1 , ˜¯ x, f x 2 , f
S 20 , n 1 , n 2 , n ) to simulate the null distributions.
P ˜ ✓ 0 is instantiated as a normaldistribution and used to generate k x i distributed as N (˜¯ x, ( n f x 2  n ˜¯ x 2 1 ) / ( n  1)) and to generate k y idistributed as ˜  1 k x i
+ e i , e i ⇠ N (0 , f S 20 ) for either group 1 with size n 1 or group 2 with size n  n 1 .
[Equation]
[Equation]
• ˜¯ x = 1 n
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
0 , ˜ ✓ 1 )
=
( ? , ? ) 9: return ( ˜ ✓ 0 , ˜ ✓ 1 )
2.2 Convergence
We now state and interpret our main theoretical result.
For more details on the proof and consequencesof the theorem, see Section D.
Theorem 2.2 (Theorem D.1) .
Let  e > 0 , r = p = 2 , q = 1 , and  2 R p .
For every n 2 N with 254 n > r , let X n 2 R n ⇥ p be the design matrix where the ﬁrst column is an all-ones vector and the 255 second column is ( x 1 , . . .
, x n ) T .
Let  =  n > 0 be a sequence of clipping bounds, ⇢ = ⇢ n > 0be a sequence of privacy parameters, and ⌘ 2 n = k X n   X n  N k 2  2 e .
Under the general linear model(GLM), Y n ⇠ N ( X n  ,  2 e I n ⇥ n ) . Let ˜  and ˜  N be the DP least-squares estimate of  , obtainedin Algorithm 4, under the alternative and null hypotheses, respectively.
Let ˜ T
=
˜
T n be the DP
F -statistic computed from DP sufﬁcient statistics via Algorithm 4 and Equation (5).
Suppose thefollowing conditions hold:
9 c x , c x 2 2 R such that
¯ x !
c x , x 2 !
c x 2 , and c x 2 > c 2 x ,2. 9 ⌘ 2 R such that ⌘ 2 n !
⌘ 2 ,3.
2 n ⇢ n n ,  4 n ⇢ n n ! 0 ,4. P [ 9 i 2 [ n ] , y i /
[   n ,  n ]] !
and 8 i 2 [ n ] , x i
[   n ,  n ] .
Then we obtain the following results: Under the null hypothesis:
˜  N = ˜
Nn P !
.
Under thealternative hypothesis: ˜  = ˜
n P !
.
˜ T =
˜ T n D !
2 r  q ( ⌘ 2 ) r  q .
Note that condition that P [ 9 i 2 [ n ] , y i /
[   n ,  n ]] !
(Condition 4 in Theorem 2.2), holds by a
Gaussian tail bound (Claim D.4), if for all i 2 [ n ] ,  n  |  1 x i +  2 | +  e p log 2 n O (1) . [Figure]
T n is the non-private F -statistic while ˜ T n is the DP F -statistic constructed from DP sufﬁcientstatistics obtained via Algorithm 4.
The main theorem in this section is Theorem 2.2, which shows theconvergence, in distribution, of ˜ T n to the asymptotic distribution of T n , the chi-squared distribution.
As a corollary, the statistical power of ˜ T n converges to the statistical power of T n .
While Theorem 2.2is specialized to the simple linear regression setting (i.e., p = 2 ), it can easily be extended to multiplelinear regression.
Experimental Evaluation
In the appendix (Section E), we provide a more thorough suite of experiments to support our claims. In this section, we present a snippet of these results.
General Parameter Setup for Synthetic Data: For experimental evaluation on synthetic datasets, 279 we generated datasets with sizes between n = 100 and n = 10 , 000 .
For both the linear relationship and mixture model tests on synthetic data below, we consider thefollowing values of ⇢ : { 0 . 1 2 / 2 , 1 2 / 2 , 5 2 / 2 , 10 2 / 2 } .
We draw the independent variables x 1 , . . .
, x naccording to a few different distributions: Normal, Uniform, Exponential.
For all tests below, the clipping parameter is either set to  = 2 or  = 3 .
For the synthetic data,the dependent variable Y is generated using the linear or mixture model speciﬁcation described inprevious sections and by ﬁxing or varying parameters (such as  e ).
3.1 Testing a Linear Relationship on Synthetic Data
Evaluating the Signiﬁcance for Normally Distributed Independent Variables : Generally, we see 288 that the signiﬁcance remains below the target signiﬁcance level, on average, for all values of ⇢ .
Forthe linear relationship tester, when the standard deviation of the dependent variable (  e ) is small(Figure 1a), we see that the true signiﬁcance level is well below the target signiﬁcance of 0.05, whichis ﬁne (but conservative).
We conjecture that this happens because when  e is small: (i) we fail toreject when the noisy estimate of  e is  0 ; or (ii) the test statistic under the null distribution will be 293 almost always
since under the null (even without privacy), the standard deviation of the test statisticis proportional to  e .
In Figures 1a and 1b, we see the signiﬁcance of the linear tester attains thetarget (of 0.05) as we vary the noise in the dependent variable  e .
3.2 Testing Mixture Models on Synthetic Data
Evaluating the Power as we Increase the Variance of the Independent Variable :
In Figures 2a 298 and 2b, we see that the F -statistic method outperforms the KW method when the variance of theindependent variable is much larger (10x) than previously.
[Figure] [Figure]
3.3 Testing on Opportunity Insights Data
[Figure] The Opportunity Insights (OI) team gave us simulated data for census tracts from the followingstates in the United States: Idaho, Illinois, New York, North Carolina, Texas, and Tennessee.
Thedependent and independent variables are the child and parent national income percentiles, respectively.
Figures 3a, and 3b show the probability of rejecting the null as we increase the parameter ⇢ whenusing the DP linear tester.
Conclusion
We have developed differentially private hypothesis tests for testing a linear relationship in data andfor testing for mixtures in linear regression models.
We also show that the DP F -statistic convergesto the asymptotic distribution of the non-private F -statistic.
Through experiments, we show thatour Monte Carlo tests achieve signiﬁcance that is less than the target signiﬁcance level across awide variety of experiments.
Furthermore, our tests generally have a high power, getting higheras we increase the dataset size and/or relax the privacy parameter.
Even on small datasets (in thehundreds) with small slopes, our tests retain the small signiﬁcance while having a good power.
Wehave provided formal statements for the DP F -statistic in the asymptotic regime.
We leave to futurework the task of theoretically analyzing the procedures in the non-asymptotic regime.
Experimental evaluation is done on simulated data for the Opportunity Atlas tool, UCI datasets, andon synthetic datasets of varying distributions on the independent variable (normal, exponential, anduniform).
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See conclusion and experimentssection (our DP tests do not perform well for certain data distributions).(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We discussthat in our work the size of the dataset n is public whereas the record-level informationremains private.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See all theoremand lemma statements.
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section E. We specify data splits for the mixture model testsand hyperparameters for both linear and mixture model testers.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Our error bars correspond to the probability of rejectingthe null hypothesis over thousands of trials.
See Section E.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section E.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] [Table]