Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference
Abstract
A surprising phenomenon in modern machine learning is the ability of a highly overparameterized model to generalize well (small error on the test data) even when it is trained to memorize the training data (zero error on the training data).
This has led to an arms race towards increasingly overparameterized models (c.f., deep learning).
In this paper, we study an underexplored hidden cost of overparameterization: the fact that overparameterized models may be more vulnerable to privacy attacks , in particular the membership inference attack that predicts the (potentially sensitive) examples used to train a model.
We signiﬁcantly extend the relatively few empirical results on this problem by theoretically proving for an overparameterized linear regression model in the Gaussian data setting that membership inference vulnerability increases with the number of parameters.
Moreover, a range of empirical studies indicates that more complex, nonlinear models exhibit the same behavior.
Finally, we extend our analysis towards ridge-regularized linear regression and show in the Gaussian data setting that increased regularization also increases membership inference vulnerability in the overparameterized regime.
Introduction
As more machine learning models are being trained on sensitive user data (e g., customer behavior, medical records, personal media), there is a growing concern that these models may serve as a gateway for malicious adversaries to access the models’ private training data [1, 2].
For example, the possibility of performing membership inference (MI), the task of identifying whether a speciﬁc data point was included in a model’s training set or not, can be greatly detrimental to user privacy.
In settings like healthcare, knowledge of mere inclusion in a training dataset (e g., hospital visitation records) already reveals sensitive information.
Moreover, MI can enable the extraction of verbatim training data from a released model [1].
The general motivation for this paper is the recent realization that privacy issues around machine learning might be exacerbated by today’s trend towards increasingly overparameterized models that have more parameters than training data points and so can be trained to memorize (attain zero error on) the training data.
Surprisingly, some overparameterized models (e.g., large regression models [3], massive deep networks [4, 5]) generalize extremely well [6, 7].
Limited empirical evidence suggests that overparameterization may lead to greater privacy vulnerabilities [1, 8–11].
However, there has been little to no analytical work on this important problem.
In this paper, we take the ﬁrst steps towards an analytical understanding of how the number of parameters in a machine learning model relates to MI vulnerability.
In a theoretical direction, we prove for linear regression on Gaussian data in the overparameterized regime that increasing the number of parameters of the model increases its vulnerability to MI (Theorem 3.2).
In a supporting empirical direction, we demonstrate that the same behavior holds for a range of more complex models: a latent space model, a time-series model, and a nonlinear random ReLU features model (Section 5).
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
We also extend our analysis towards the techniques of ridge regularization and noise addition in
Section 4.
We ﬁrst prove that MI vulnerability similarly increases with the number of parameters for ridge-regularized models.
Surprisingly, we also demonstrate analytically that additional regularization increases this vulnerability in overparameterized linear models.
Hence, ridge regularization is not always an effective defense against MI and can even be harmful.
Afterwards, we show that the privacy-utility trade-off induced by reducing the number of parameters of a linear regression model is equivalent to that obtained by adding independent Gaussian noise to the output of the model when using all available parameters.
Overall, our analyses show that there are settings where reducing the number of parameters of a highly overparameterized model is a simple strategy to protect a model against MI.
As the trend towards increasingly overparameterized machine learning models accelerates, our results make the case for less overparameterization when privacy is a concern.
Related work.
The problem of membership inference has received considerable attention, and we refer the reader to [12] for a survey of the prior art.
Many works have demonstrated how deep neural networks can be highly vulnerable to MI attacks (cf., [13–16] for example).
Various types of attacks, such as shadow models [13], conﬁdence-based attacks [17, 18], and label-only attacks [19] have risen to further expose privacy vulnerability of machine learning models.
Our analysis in this work is based on optimal MI attacks, a framework also employed by other studies of MI [9, 14, 10].
While we study the effect parameterization has on MI vulnerability, several studies examine how other aspects of the machine learning pipeline, such as data augmentation [20], dropout [15], sparsity [21], and ensembles [22] affect MI vulnerability.
[16] show that pruning a neural network can defend against MI, lending further experimental support to the link between overparameterization and membership inference.
[9] analyzes the relationship between overﬁtting (as measured by generalization error) and membership inference but does not connect this to the number of parameters.
Much recent work has shown though that overparameterization does not always lead to increased generalization error and indeed sometimes can even decrease it [23, 6, 3, 7], suggesting that more work is needed linking overparameterization and membership inference beyond mere generalization error.
Despite some empirical evidence that overparameterized models are vulnerable to MI attacks [1, 11], to the best of our knowledge, there has been no theoretical study of the effect of this connection.
Differential privacy (DP) [24] is another popular framework frequently used to study the privacy properties of machine learning algorithms, and models that have DP also enjoy MI guarantees [14].
[25] bounds the minimum dataset size as a function of dimension to ensure privacy, however its results do not extend to data of continuous values (e.g., regression data) as is done in this work and does not account for the randomness in the data generating process, which as argued in [14], is a necessary condition to bound the optimal MI risk.
While our theoretical analysis focuses on linear regression, many works have studied overparameterized linear models as an interpretable and tractable test case of more complex overparameterized models such as deep networks [26, 27, 3, 6, 23], suggesting links of our results to more complex nonlinear models.
Contributions.
In this paper, we establish the ﬁrst theoretical connection between overparameterization and vulnerability to membership inference attacks.
Our contributions are as follows:
We derive the optimal MI accuracy for linear regression on Gaussian data and show that increasing the number of parameters increases the vulnerability to MI.
We empirically show for more complex models that increasing the number of parameters also increases MI vulnerability.
We theoretically show for overparameterized linear models that increased ridge regularization increases MI vulnerability in the Gaussian data setting.
We use our analysis to show that reducing the number of parameters of a linear regression model yields an equivalent privacy-utility trade-off as adding independent noise to the output of a model using all available parameters.
Our code is provided in https://github.com/tanjasper/parameters_or_privacy .
The Membership Inference Problem
We now introduce our notation and formally state the membership inference (MI) problem.
Let
D 2 Z + denote the data dimension.
Let z
=
( x , y ) 2 R D ⇥ R denote a data point, and let D be a distribution over the data points.
Consider a set S sampled S ⇠ D n of size n 2 Z + data points and denote it by S
= { z 1 , z 2 , ..., z n } = { ( x 1 , y 1 ) , ..., ( x n , y n ) } .
Let f S represent a machine learning model obtained by applying a training algorithm T on the dataset S .
In particular, f S is a deterministic function in R D ! R .
Typically, it is a function that minimizes P ni {+=
 `+} ( f ( x i ) , y i ) over a family of functions f for some loss function ` : R ⇥ R !
R  0 .
We denote b y i =
f S ( x i ) .
Given a data point z 0 and a trained model f S , membership inference is the task of identifying if zwas included in the set S used to train f S .
Different MI attacks are characterized by the types of information accorded to the adversary.
In this work, we focus on a blind black-box setting where the adversary has access to x 0 , f S ( x 0 ) , n , the training algorithm T , and the data distribution D , but not to the ground truth y 0 , the loss value ` ( f S ( x 0 ) , y 0 ) , or any learned parameters of f S .
Thus, the adversary is a function A : R d ⇥ R ! { 0 , 1 } such that given ( x 0 , f S ( x 0 )) , it outputs 0 or 1, representing its prediction as to whether x 0 is a member of S based on f S ( x 0 ) .
There are two main interests in studying blind MI wherein the adversary does not have access to the loss value.
First, there are many realistic scenarios where the ground truth variable of interest is unknown for the general population, and hence the need for a model f S to predict it.
Second, since we study the interpolation regime where the training loss can be driven down to exactly 0,
MI becomes trivial given the loss value: an adversary that predicts any data point that achievesloss as a member will have perfect accuracy.
Multiple works in the literature also study this blind setting [18, 28].
We emphasize that our results on the blind adversary lower bound the performance of adversaries that additionally have access to more information.
MI is often deﬁned as an experiment to facilitate precise analysis, and we follow the setup of [9] summarized below.
Experiment 1 .
Membership inference experiment.
Given a data distribution D , an integer n , a machine learning training algorithm T , and an adversary A , the MI experiment is as follows:
Sample S ⇠ D n .
Apply the training algorithm T on S to obtain f S . 3.
Sample m 2 { 0 , 1 } uniformly at random.
If m = 0 , sample a data point ( x 0 , y 0 ) ⇠ D .
Else, sample a data point ( x 0 , y 0 ) 2 S uniformly at random.
Observe the adversary’s prediction A ( x 0 , f S ( x 0 ))
We wish to quantify the optimal performance of the adversary A and turn to the popular membership advantage metric deﬁned in [9].
The membership advantage of A is the difference between the true positive rate and false positive rate of its predictions.
Deﬁnition 2.1 ([9]) .
The membership advantage of an adversary A is:
[Equation]
where P ( · ) is taken jointly over all randomness in Exp. 1.
Note that membership advantage is an average-case metric, as opposed to worst-case metrics considered by other privacy frameworks such as differential privacy.
Our analysis is thus focused on average-case privacy leakage, and we do not provide worst-case guarantees.
Theoretical Results 3.1 Optimal Membership Inference Via Hypothesis Testing
In this paper, rather than studying current MI attacks, we study the optimal MI adversary: the adversary that maximizes membership advantage.
As such, our analysis and results are not restricted to the current known attack strategies, which are constantly evolving, and instead serve as upper bounds for the performance of any MI attack, now or in the future.
The following proposition supplies the theoretically optimal MI adversary.
Proposition 3.1.
The adversary that maximizes membership advantage is:
[Equation]
where b y
= f S ( x 0 ) and P denotes the distribution function for b y 0 over the randomness in the membership inference experiment conditioned on x 0 .
As observed by [29, 10], the optimal adversary performs a hypothesis test with respect to the posterior probabilities, with the two hypotheses being: [Equation] D [Equation]
x
[Equation]
so that maximizing membership advantage is achieved by performing a likelihood ratio test.
3.2 Linear Regression with Gaussian Data
We begin our study of overparameterization’s effect on MI with linear regression with Gaussian data.
We ﬁnd that in the sufﬁciently overparameterized regime, increasing the number of parameters increases vulnerability to MI.
We denote by n, p, D the number of data points, number of model parameters, and data dimensionality, respectively.
Let n, p, D 2 Z + be given such that p  D , and let  > 0 be given.
Consider a D -dimensional random vector  ⇠ N  0 , 1 D I D  , representing the true coefﬁcients.
We consider data points ( x i , y i ) where x i ⇠ N (0 , I d ) and y i
=
x > i  + ✏ i , where ✏ i ⇠ N (0 ,  2 ) .
Sampling n data points, we denote by X the n ⇥ D matrix whose i th row is x > i and by y the n -dimensional vector of elements y i .
Further, let X p be the n ⇥ p matrix containing the ﬁrst p columns of X .
Least squares linear regression ﬁnds the minimum-norm p -dimensional vector ˆ  that minimizes k y  X p  k 22 , which is ˆ  =
X † p y , where X † p denotes the Moore-Penrose inverse of X p .
Then, for a given feature vector x 0 , the model predicts b y
= x > 0 ,p ˆ  , where x 0 ,p is the vector containing the ﬁrst p elements of x 0 .
In this setting, we can derive the membership advantage of the optimal adversary as a function of the number of parameters p used by the model.
Theorem 3.2.
Let n, p, D 2 Z + be given such that n + 1 < p  D .
Let x 0 be a given Ddimensional vector.
Let  ⇠ N (0 , 1 D I D ) , x i ⇠ N (0 , I D ) , ✏ i ⇠ N (0 ,  2 ) , and y i = x > i  + ✏ i for i 2 { 1 , 2 , ..., n } .
Let m be a variable whose value is either 0 or 1 such that, if m = 1 , x k is set to x 0 for k chosen uniformly at random from 1 , 2 , ..., n .
Let X be the n ⇥ D matrix whose rows are x > 1 , x > 2 , ..., x > n .
Finally, let b y
= x > 0 ,p X † p y where y is the n -dimensional vector with elements
y i .
Then, as n, p, D ! 1 such that pn !  2 (1 , 1 ) , we have:
[Equation]
where
[Equation]
Consider the case when  1 >  0 .
The resulting optimal membership inference algorithm
A ⇤ is
[Equation]
i where ↵ :=
[Equation]
with membership inference advantage Adv :
[Equation]
where  ( · ) denotes the CDF of the standard normal, and ↵ ,  0 ,  1 are conditioned on x 0 .
Remark 3.3 .
The case where  1 <  0 occurs when  is small ( p ⇡ n ).
The same membership advantage result holds in this, reversing the roles of  0 and  1 in Adv ( A ⇤ ) in Eqs.
(3) and (4) and reversing the inequality in Eq.
Remark 3.4 .
The above result holds using the asymptotic distributions as n, p, D ! 1 .
In Lemma C. in the Appendix, we derive the non-asymptotic distributions for the predictions of the minimum-norm least squares interpolator, though they cannot be written in closed form.
[Figure]
In Theorem 3. Eq.
(1) shows the posterior distributions of the outputs test points (when m = 0 ) and training points (when m = 1 ) are both 0 -mean Gaussians but with variances  20 and  respectively given in Eq.
from Prop.
1 that the optimal MI adversary is a likelihood ratio test (LRT) between the distribution of the model’s output for new test points ( reused training points standard sufﬁcient statistic for an between two 0 -mean Gaussian distributions with  0 and  1 .
Finally, we compute the membership advantage of this adversary in Eq.
(4) by taking an expectation over the random draw of x 0 , as deﬁned in Experiment 1, noting that  0 ,  1 , and ↵ are all functions of x 0 .
Membership advantage is deﬁned to be the difference between the true and false positive probabilities in Defn.
This difference is given by  ( ↵ /  0 )
( ↵ /  1 ) for each side of the
Gaussian distributions, leading to the expression in Eq.
To understand the implication of the result for MI, ﬁrst observe that when m = 1 and x 0 is a training point that is memorized by the model, the variance of the model’s output is equal to the variance of the measurement y {+
+}
itself independent of p .
On the contrary, when m = 0 and x 0 is a test point, the variance of the model’s output is decreasing with p (Figure 1a).
Hence, though the output distribution means stay the same, as the variance for the m = 0 case decreases far below that of the m = 1 case, an LRT can easily distinguish these two distributions.
In the extreme case, suppose p, D  n , then  20 ! 0 , while  21 remains a nonzero value.
Hence Adv ( A ⇤ ) ! 1 .
We conﬁrm our derivations of the output distributions numerically in Figure 1 where we plot empirical histograms of the predicted distributions from Eq.
(1) by repeatedly computing the minimum norm least squares solution over 20,000 independent trials 1 .
We also plot the standard deviations from
Eq.
(2) in Fig.
1a.
Visually, from the distributions in Figure 1, given f ( x 0 ) , MI reduces to identifying whether it is more likely that a sample came from the blue distribution or from the orange with the adversary simply predicting the more likely outcome.
These distributions intersect at ± ↵ , so it is sufﬁcient to compare
[Equation]
In Figure 2a, we plot the membership advantage for  > 1 using Eq.
Since the difference between the variances of the model’s output when m = 0 and m = 1 increases with  for  >
[Figure]
(cf., Fig.
1a), it becomes easier to distinguish the two distributions.
This results in an increase in the membership advantage.
The initial decrease in membership advantage when   2 is a consequence of  1   0 in this regime as shown in Fig.
1a.
Since  0 is decreasing in p , initially, this decrease makes the train and test output distributions harder to distinguish, leading to lower membership advantage and 0 advantage for  = 2 .
However, for  > 2 ,  0 decreases past  1 and membership advantage approaches 1 as  ! 1 .
In practice, the  < 2 regime (only slightly overparameterized) is less interesting since models suffer larger generalization error in this setting (cf., Theorem 1 of [23]) and are rarely (if ever) used in practice.
A key takeaway is that for linear regression in the
Gaussian data setting, extreme overparameterization increases the vulnerability of a machine learning model to MI.
While Theorem 3.2 operates in the asymptotic regime, we empirically approximate membership advantage for n = 100 and D = 3 , 000 by approximating the posterior distributions with histograms of samples.
As we see in Figure 2b, the asymptotic formula agrees very closely with the nonasymptotic experiment.
Experimental details are provided in Appendix E.
Mitigating Membership Inference Attacks
Next, we extend our analysis from the previous section towards two methods commonly used for preserving privacy: regularization and noise addition.
We present two key results.
First, we show that for the overparameterized Gaussian data setting, ridge regularization actually increases membership advantage and is thus detrimental to privacy.
Second, we show that the privacy-utility trade-off induced by reducing the number of parameters of a linear regression model with Gaussian data is equivalent to that of adding independent Gaussian noise to the output of a model that uses all available features.
4.1 Ridge-Regularized Linear Regression
We analyze membership inference in ridge-regularized linear regression in the same setting as
Section 3.2 except that, now, the estimate is ˆ   =
( X > p X p + n  I p ) † X > p y , where  is a regularization parameter.
Larger values of  yield greater regularization, and {+ ! 
+} reduces to the case of Section 3.2.
Regularization is a common method to reduce overﬁtting and has thus been proposed in previous works as a defense mechanism to protect models from MI attacks [13, 30].
A surprising observation resulting from our analysis is that, in the highly overparameterized regime (   1 ), ridge regularization actually increases the model’s vulnerability to MI attacks for linear
[Figure]
regression with Gaussian data.
In Theorem D.1, presented in the appendix, we derive an analogous result for the distributions of the ridge-regularized predictions as Theorem 3.2 demonstrated for the unregularized case.
In particular, if A ⇤  is the adversary for the  -regularized problem, its membership advantage Adv ( A ⇤  ) is an increasing function in  when   1 .
To visualize this, we plot the theoretical membership advantages for 100 sampled x 0 ’s (each with iid standard normal elements) for differing regularization strengths in Figure 3 with the same setting as in Figure 2a.
In Figure 3a, we plot membership advantage as a function of the overparameterization ratio  for a few n  values.
Conversely, In Figure 3b we plot membership advantage versus regularization n  for a few different values of  .
In particular, note how in both subplots, increasing  never decreases the membership advantage.
This observation has been made empirically (but not analytically) for techniques that have similar regularizing effects, such as dropout [15], ensembling [22], and weight decay [30].
Intuitively, the reason ridge regularization increases MI vulnerability is because while it decreases the variance of the model’s output on training points, it also signiﬁcantly decreases the variance for test points such that the two output distributions become more distinguishable.
In Figure 5 in the appendix, we plot the gap between the variances for the m =
and m = 1 cases for different regularization amounts  to visualize this effect.
4.2 Noise Addition vs. Feature Reduction
A consequence of Thm.
3.2 is that, in the Gaussian data setting, one can reduce vulnerability to MI attacks by simply decreasing the number of parameters/features.
However, this comes at the cost of decreased generalization performance (utility) due to the “double descent” effect [6, 23, 7], wherein generalization error decreases with increased overparameterization.
An alternative and popular method to increase a model’s privacy is adding independent noise to the model output [24, 31], but this also decreases generalization performance.
Interestingly, we show in this subsection that for
Gaussian data, the privacy-utility trade-offs induced by both feature reduction and noise addition are actually equivalent.
Feature reduction: To characterize the privacy-utility trade-off of feature reduction, we ﬁrst need the generalization error for a given number of parameters, provided in Corollary 2.2 of [23]:
Proposition 4.1.
(Adapted from Corollary 2.2 of [23]).
In the same setting as Theorem 3.2, for x 0 ⇠ N (0 , I D ) , the generalization error is given by
[Equation]
Alternatively, this expression can be written as 1 +  2 + E x {+
+} [  20 ]  2 nD , with  20 as in Theorem 3.2.
Noise addition: Next, we consider noise addition performed by perturbing the m = 0 model output with independent noise before releasing it: [-(cid:98) y 
-][Equation]
where ¯ ✏ ⇠ N
[Equation]
0 , ¯  2 ) .
Recall that MI is possible for overparameterized linear regression models because the variance of the output of test points ( m = 0) is lower than the variance of the output for training points ( m = 1) (Fig. 1).
Thus, adding random noise to the model output when a test point x 0 is queried can make the output distributions harder to distinguish.
In the following lemma, we compute the membership advantage and generalization error of noise addition using the full set of features p
= D .
Lemma 4.2.
Consider the setup of Theorem 3.2 restricted to p
=
D ,  1 >  0 , and  > 1 , and suppose independent 0 -mean, variance ¯  2 Gaussian noise is added to the outputs when m =
( x 0 is not a member of the training set).
Then, ( b y [-
-]0 | m = 0 , x 0 )
⇠ N (0 ,  20 + ¯  2 ) .
The optimal adversary is
A ⇤ ¯  ( x 0 , b y 0 ) :
= ⇥
ˆ y 02 > ↵ 2¯
⇤ where ↵ ¯  :=
[Equation]
Its membership inference advantage is
[Equation]
Additionally, for x 0 ⇠ N (0 , I D ) , the generalization error incurred is
[Equation]
The terms in Lemma 4.2 are similar to those in Theorem 3.2 except for the added dependence on the added noise’s variance ¯  2 in the case that m =
( x 0 is not a member).
Increasing noise variance ¯  2 decreases membership advantage (Eq.
at the cost of increased generalization error (Eq.
Indeed, it is possible to add sufﬁcient noise such that  21 =  20 + ¯  2 , rendering the membership advantage 0, though possibly at the cost of impermissible generalization performance.
In Figure 3c, we plot the membership advantage vs. generalization error trade-offs for both feature reduction (blue) and noise addition (orange).
The plots follow the setting of Fig. 2b.
For the blue curve, we employ the expressions in Thm 3.2 and Prop 4.1 while varying the overparameterization ratio  .
For the orange curve, we use Lemma 4.2, using all available features ( p = D ) while varying the noise variance ¯  2 .
Fig.
3c shows that the trade-offs induced by both feature reduction and noise addition are essentially equivalent.
More Complex Models
In this section, we present three more complex data models wherein we empirically observe increased overparameterization leading to increased MI vulnerability.
For each data model, we perform the following experiment.
We ﬁrst sample a x 0 vector, which is the data point we wish to perform MI on.
Then we sample a training dataset X , measurements y , as well as other random elements according to the data model.
To obtain an m = 0 prediction, we learn a model on X that we then apply on x 0 .
To obtain an m = 1 prediction, we ﬁrst replace a row of X with x 0 and the corresponding element of y with y 0 before learning the model and applying it on x 0 .
Keeping x 0 ﬁxed throughout the experiment and resampling all other random data (such as X ) many times, we collect a large set of m =
and m = 1 prediction samples.
We build a histogram of these samples by assigning them into ﬁne discrete bins to obtain approximations of the conditional densities ˆ P ( b y
| m = 1 , x 0 ) and ˆ P ( b y
| m = 0 , x 0 ) needed for the optimal adversary (cf., Prop 3.1).
To approximate membership advantage, we sum up the differences ˆ P[-( (cid:98) y 
-]
[-| m 1 , x 0 ) − ˆ P ( (cid:98) y 
-][Equation]
over all the histogram bins where ˆ P ([-(cid:98) y 
-]
[Equation]
[-ˆ P ( (cid:98) y 
 | m 0 , x 0 ) .-] We repeat this experiment 20 times, each with a newly sampled x 0 , and plot the means (as points) and the estimated standard errors (as shaded error regions) of the membership advantage values across the 20 experiments in
Figure 4.
We next discuss the data models in detail.
Linear Regression on Latent Space Model:
We ﬁrst consider the latent space model from [3], where the output variable y
i is a noisy linear function of a data point’s d latent features z i , but one
[Figure]
only observes a vector x
i containing p  d covariates rather than the direct features z i .
Let Z be an n ⇥ d matrix where each row is the vector of latent features for an observation.
We then have:
[Equation]
where w j is a d -dimensional vector, and u i,j is a noise term.
In this experiment, we set n = 200 , d = 20 , and vary p .
For each experiment, we sample a single x 0 ⇠ N (0 , I d ) and a single set of w j vectors, each from N (0 , I d ) , and keep them ﬁxed.
We leave the other variables random with the following distributions: z i ⇠ N (0 , I d ) , ✏ i ⇠ N (0 ,  2 ) ,  ⇠ N  0 , 1 d I d  and u i,j ⇠ N (0 , 1) .
In Fig.
4a, we plot the empirical membership advantage values, which increase with the number p of features in the overparameterized regime.
Noise-Free Time-Series Regression:
In this experiment, we consider a model that aims to interpolate a time-series signal using frequency components as the features.
For example, consider a patient who visits a hospital at irregular times t
i to get their blood glucose level measured.
After obtaining a number of measurements taken over time, the hospital ﬁts a time-series signal representing the patient’s blood glucose level at any time.
Using this learned model, an adversary wishes to identify whether the patient visited the hospital at a particular time t 0 , that is, if t 0 is one such time point included in the hospital’s training dataset for learning the patient’s blood glucose level.
To formalize this, we ﬁx D = 1024 as the number of frequency components each signal contains and let M = 2 D  1 .
Let W be the D ⇥ D matrix whose elements W kl = 1 p M +2 cos  2 ⇡ klM  .
That is, each column of the matrix is half a period of an M -dimensional discrete cosine with frequency 2 ⇡ kM .
Sample a  ⇠ N  0 , 1 D I D  , and let z =
W  denote the true length- D signal.
Thus, the signal is a random linear combination of cosines.
We randomly select n = 128 indices from 1 , 2 , . . .
, D uniformly without replacement, and let X be the n ⇥ D matrix whose rows are the rows of W at these selected indices.
Then, y =
X  is the signal observed at the randomly selected n indices.
The regressor learns ˆ  = X † p y and then predicts the signal at any other time point t 0 as [-(cid:98) y 
-][Equation]
where x 0 is row t 0 of W .
Thus, identifying whether t 0 was a time point in the training dataset is equivalent to identifying if x 0 was in X .
The membership advantage values for this task, plotted in
Figure 4b, increases with the number p of frequency components included in the model.
Random ReLU Features: We next consider a nonlinear data model based on Random ReLU feature networks [32, 33].
Let Z be a random n ⇥ D matrix whose elements are iid standard normal.
Let
V be a random D ⇥ p matrix whose rows are sampled iid from the surface of the unit sphere in R p .
Let X = max( ZV , 0) , where the max is taken elementwise.
The target variables are given by y
= Z  + ✏ , where  ⇠ N  0 , 1 D I D  and ✏ ⇠ N (0 , I n ) .
Finally, for the data point x 0 , let its prediction be b y
= x > 0 X † y .
We plot the membership advantage in Figure 4c with n = 100 ,
D = 5000 , and  = 1 .
We again observe that in the highly overparametrized regime, membership advantage increases with parameters.
Discussion and Conclusions
We have shown theoretically for (regularized) linear regression with Gaussian data and empirically for more complex models (latent space regression, time-series regression using Fourier components, and random ReLU features) that increasing the number of model parameters renders them more vulnerable to membership inference attacks.
Thus, while overparameterization may be attractive for its robust generalization performance, one must proceed with caution to ensure the learned model does not lead to unintended privacy leakages.
More speculatively, we hypothesize that the same overparameterization/vulnerability tradeoff should exist in many machine learning models (e.g., deep networks) beyond those we have studied.
Intuitively, the output of a model that achieves zero training error but generalizes well must i) ﬁt to any noise (e.g. additive Gaussian noise) present in the training data to get a perfect ﬁt to the noisy training data but also ii) eliminate the effect of noise in the training data in predicting for unseen data to achieve good generalization.
This disparate behavior towards training and non-training data points leads to different output distributions when the input is or is not among the training data and is universal for overparameterized models.
Ultimately, this causes a difference in the distributions of training and test predictions that can be leveraged to perform a membership attack.
There are still many open questions in this line of research.
While we have shown multiple settings where reducing the number of parameters can increase privacy, it remains to be veriﬁed that the phenomenon holds widely for other types of machine learning settings such as language tasks or large-scale image recognition.
Another interesting next step would be to investigate how increased overparameterization affects privacy for models trained with privacy-preserving techniques or membership inference defense schemes other than ridge regularization.
We believe the ﬁndings of our work can provide insights towards developing the next generation of privacy-preserving techniques.
It is our hope that the observations and analyses in this paper take a step towards keeping sensitive training data safer in a world increasingly intertwined with machine learning.
Acknowledgements
This work was supported by NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants
[-= 1 if P ( (cid:98) y 
-]
[-| m = 1 , x 0 ) − P ( (cid:98) y 
-]
[-x 0 is part of training set, and so (cid:98) y 
-]
[-Since ϵ ∼ N (0 , σ 2 ) , we have that (cid:98) y 
-]
[-0 , 1 D || x 
-]
[-[Equation] for ∥ x ∥ A := √ x ⊤ Ax for any semidefinite matrix A where we have additionally conditioned over any randomness in the choice of x 0 . In the m = 1 case, where x 0 is sampled uniformly from the rows of X , we have that x ⊤ 0 ,p ˆ β = y 
-]N00014-18-12571, N00014-20-1-2534, and MURI N00014-20-1-2787; AFOSR grant FA9550-22-10060; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
[-Finally, we collect the (cid:98) y 
-]
[-The orange histograms are formed the same way except that the first row of X is replaced with x 0 and the first element of y is replaced with y 
-]
[-From these samples, the discrete histograms for ( (cid:98) y 
-]
[-| m = 0) and ( (cid:98) y 
-]
[-| m = 1) for a given γ are then formed by splitting the interval between the minimum and maximum values over both ( (cid:98) y 
-]
[-| m = 0) and ( (cid:98) y 
-]