Differentially Private Linear Sketches: Efficient Implementations and Applications
Anonymous Author(s)
AffiliationAddress email
Abstract
Linear sketches have been widely adopted to process fast data streams, and theycan be used to accurately answer frequency estimation, approximate top K items,and summarize data distributions.
When data are sensitive, it is desirable toprovide privacy guarantees for linear sketches to preserve private information whiledelivering useful results with theoretical bounds.
We show that linear sketchescan ensure privacy and maintain their unique properties with a small amount ofnoise added at initialization.
From the differentially private linear sketches, weshowcase that the state-of-the-art quantile sketch in the turnstile model can also beprivate and maintain high performance.
Experiments further demonstrate that ourproposed differentially private sketches are quantitatively and qualitatively similarto noise-free sketches with high utilization on synthetic and real datasets
Introduction
Data sketches are fundamental tools for data analysis, statistics, and machine learning [Cormode and
Yi, 2020].
Two of the most widely studied problems in data summaries are frequency estimation andquantile approximation.
Many real world applications need to estimate the frequency of each itemin the database and understand the overall distribution of the database.
These applications includestream processing [Das et al., 2009, Bailis et al., 2017], database management [Misra and Gries, 1982,
Metwally et al., 2005, Zhao et al., 2021a], caching [Zakhary et al., 2020], network monitoring [Guptaet al., 2016, Ivkin et al., 2019], federated learning [Rothchild et al., 2020], among others.
On one hand, the motivation for data sketch algorithms is to efficiently process a large databaseand extract useful knowledge, since computing the exact information for a large amount of data isboth time and memory intensive.
For instance, Munro and Paterson [1980] proved that to find thetrue median of a database with n items using p sequential passes requires at least Ω( n 1 /p ) memory.
On the other hand, to protect user-level privacy, privacy-preserving algorithms limit the disclosureof private information in the database so that an observer cannot infer much about an individual.
Recent works have shown that data sketches can be integrated with privacy-enhancing technologiesto provide insightful information and preserve individual privacy at the same time [Cormode, 2022].
Differential privacy [Dwork et al., 2006] is a widely-accepted definition of privacy.
Recently, researchers have observed that some data sketches are inherently differentially private [Blocki et al.
,2012, Smith et al., 2020], while many other data sketches need modifications to the algorithm to bedifferentially private.
In particular, a substantial amount of literature has focused on differentiallyprivate data sketches for tasks such as linear algebra [Upadhyay, 2014, Arora et al., 2018], cardinality estimation [Mir et al., 2011, Pagh and Stausholm, 2020, Dickens et al., 2022] and quantileapproximation [Tzamos et al., 2020, Gillenwater et al., 2021, Alabi et al., 2022].
In this paper, we introduce new differentially private algorithms that support both insertions anddeletions for frequency, top k, and quantile approximation.
While many data sketches assume aninsertion-only model [Greenwald and Khanna, 2001, Shrivastava et al., 2004, Karnin et al., 2016] or abounded-deletion model
[Jayaram and Woodruff, 2018, Zhao et al., 2021a,b], our algorithms build ontop of linear sketches [Charikar et al., 2002, Cormode and Muthukrishnan, 2005] and operate in theturnstile model, which allows an arbitrary number of insertions and deletions into the database.
Earlier,researchers attempted to prove Count-Median [Charikar et al., 2002] itself preserves differentialprivacy, but the authors acknowledged that there are issues in the proof [Li et al., 2019].
Instead ofproving that linear sketches, i.e., both Count-Median and Count-Min, are inherently differentiallyprivate, we add a small amount of Gaussian noise at their initialization to provide a privacy guarantee,while maintaining linear sketches’ original properties, providing high utility for frequency and top
K estimations, and keeping update and query algorithms unchanged.
We also demonstrate that ouranalysis provides the tight uniform bound (Section 3.1 and Appendix E).
In addition, we propose thefirst differentially private quantile sketch in the turnstile model by leveraging the differentially privatelinear sketch.
Our differentially private sketches can be queried an arbitrary number of times withoutaffecting privacy guarantees based on the post-processing immunity.
Following prior works [Choiet al., 2020, Smith et al., 2020], we assume ideal random hash functions exist, and this assumptioncan be replaced in practice by cryptographic hash functions [Dickens et al., 2022].
Preliminaries
Consider a database X
= { i t } t ∈
[ N ] of N items that are drawn from a large universe of size U , suchas IPv4 address of size 2 32 , and for each insert or delete operations, one item can be inserted intoor deleted from the database X .
To support ordered statistic such as quantile, we assume that theuniverse is some finite totally ordered data universe.
Definition 2.1.
Given a database X , the frequency of an item x is f ( x ) = (cid:80)
Nt =1 π ( i t = x ) where πreturns 1 if i t is x , and 0 otherwise.
Definition 2.2.
Given a database X of items drawn from an ordered universe, the rank of an item xis R ( x ) =
(cid:80)
Nt =1 π ( i t ≤ x ) where π returns 1 if i t is less or equal to x and 0 otherwise.
Given the large size of N , calculating the actual statistics, such as frequency and quantile, is oftenhard, and hence most applications are satisfied with an approximation .
The randomized frequencyestimation problem takes an accuracy parameter γ and a failure probability β such that, for any itemx , | (cid:98) f
( x ) − f ( x ) | ≤ γ · N with high probability 1 − β , where (cid:98) f ( x ) is the estimated frequency and f ( x )is the true frequency [Cormode and Hadjieleftheriou, 2008].
In addition, the randomized quantileapproximation problem also takes an accuracy parameter γ and a failure probability β
such that, forany item x , | (cid:98) R ( x ) − R ( x ) | ≤ γ · N with high probability 1 − β where (cid:98) R ( x ) is the estimated rankand R ( x ) is the actual rank [Karnin et al., 2016].
2.1 Differential Privacy
Definition 2.3.
Databases X and X ′ are neighbors ( X ∼ X ′ ), if they differ in at most one element.
Through this paper, we assume the update/replace definition of differential privacy instead ofadd/remove definition of differential privacy, in which one item in X is updated or replaced byanother item in X ′
[Vadhan, 2017].
Definition 2.4 (Differential Privacy [Dwork et al., 2006]) .
A randomized algorithm M satisfies( ϵ, δ ) -differential
privacy
( ϵ, δ ) -DP)
if for all neighboring databases X , X ′ and for all possibleevents E in the output range of M , we have
[Equation]
When δ = 0 , ϵ -DP is known as pure DP, and when δ > 0 , ( ϵ , δ )-DP is known as approximate DP.
Definition 2.5 (Gaussian Mechanism [Dwork et al., 2006]) .
Define the ℓ 2 sensitivity of a functionf : N X
(cid:55)→ R d as
[Equation]
The Gaussian mechanism M with noise level σ is then given by
[Equation]
Specifically, the Gaussian mechanism is known to satisfy a stronger notion of privacy known as zeroconcentrated differential privacy (zCDP, defined below); zCDP lies between pure and approximate
DP and can be translated into standard DP notations, as shown in Lemma 2.9.
Moreover, zCDPsatisfies cleaner composition theorems, as shown in Lemma 2.7.
Definition 2.6 (zCDP
[Dwork and Rothblum, 2016, Bun and Steinke, 2016]) .
A randomized mechanism M satisfies ρ -Zero-Concentrated Differential Privacy ( ρ -zCDP), if for all neighboring databases
X, X ′ and all α ∈ (1 , ∞ ) ,
[Equation]
where D α is the Renyi divergence [Van Erven and Harremos, 2014].
Lemma 2.7 (Adaptive composition and Post Processing of zCDP
[Bun and Steinke, 2016]) .
Let
M : X n → Y and M ′ :
X n × Y → Z .
Suppose M satisfies ρ -zCDP and M ′ satisfies ρ ′
-zCDP(as a function of its first argument).
Define M ′′ : X n → Z by M ′′ ( x ) =
M ′ ( x, M ( x )) .
Then M ′′satisfies ( ρ + ρ ′ )
-zCDP.
Lemma 2.8 (Privacy Guarantee of Gaussian mechanism [Dwork et al., 2014, Bun and Steinke,2016]) .
Let f : N X (cid:55)→ R d be an arbitrary d-dimensional function with ℓ 2 sensitivity ∆ 2 =sup neighboring X,X ′ ∥
f ( X ) − f ( X ′ )
∥ 2 .
Then for any ρ > 0 , Gaussian Mechanism with parameterσ 2 = ∆ 22 2 ρ satisfies ρ -zCDP.
Lemma 2.9 (Converting zCDP to DP [Bun and Steinke, 2016]) .
If M satisfies ρ -zCDP then Msatisfies ( ρ + 2 (cid:112) ρ log(1 /δ ) , δ ) -DP.
As we use exclusively Gaussian mechanisms and their composition in our proposed algorithms, ourmethod actually satisfies ( ϵ, δ ) -DP guarantees with stronger parameters than what is implied by zCDPvia techniques from [Balle and Wang, 2018, Dong et al., 2019], which reduces the ϵ parameter by asizable fraction in typical parameter regimes.
We stick to zCDP for clarity and generality, because allour results would apply without changes if we modify the noise into other mechanisms satisfyingzCDP, e.g., the Discrete Gaussian Mechanism [Canonne et al., 2020].
2.2 Revisiting Linear Sketches
Charikar et al.
[2002] proposed the Count-Median sketch, a randomized algorithm that summarizesa database and solves the frequency estimation problem.
The Count-Median sketch uses a d × w arrayof counters, i.e, C[d, w], where all the counters are initialized to zero , and has two sets of independenthash functions h and g .
For each row r , the hash function h r maps input items uniformly onto{ 1 , . . .
, w } and the hash function g r maps input items uniformly onto {− 1 , +1 } .
For item x withvalue v ∈ {− 1 , +1 } , Count-Median sketch updates d counters, one per each row, based on the hashvalues such that for a particular row r ,
g r
( x ) will be added or subtracted to the counter at the h r ( x ) thindex depending on whether x is being inserted or deleted respectively, as shown in Algorithm 1.
Hence, the update time is O ( d ) .
To estimate the frequency of item x , Count-Median sketch will outputthe median 1 ≤ r ≤ d g r ( x ) · C [ r, h r ( x )] , as shown in Algorithm 2.
By updating each row’s counterbased on the hashed value of either 1 or − 1 and reporting the median for query, Count-Mediansketch provides an unbiased estimate.
To reduce the failure probability of bad estimations, d is set to
O (log(1 /β ))
and it uses O ( 1 γ log( 1 β )) space to solve the frequency estimation problem.
Cormode and Muthukrishnan [2005] proposed the Count-Min sketch that shares the same initialization, update, and data structure as Count-Median sketch.
Count-Min sketch also uses O ( 1 γ log( 1 β ))space to solve the frequency estimation problem.
A major difference is that Count-Min sketch makes all hash functions in set g return positive 1.
As a result, to estimate the frequency of item x ,
Count-Min sketch returns min 1 ≤ r ≤ d C [ r, h r ( x )] instead of the median, as shown in Algorithm 2.
Inaddition, it has the nice property of never underestimating item’s frequency.
Since linear sketches canapproximate an item’s frequency accurately, they also solves the top K approximation problem byreturning the K items associated with the highest estimated frequency.
Gilbert et al.
[2002] made the connection between frequency and quantiles, in which the quantilerange can be decomposed into at most log U dyadic intervals [Cormode et al., 2019] and the sum ofthe estimated frequencies for these intervals gives the estimated rank.
Wang et al.
[2013] leveragedthe unbiased property of Count-Median sketches and proposed the Dyadic Count-Median (DCS) toestimate the frequencies of each dyadic interval.
For more specific details, Appendix B and [Cormodeand Yi, 2020] provide a comprehensive analysis of quantile sketches.
Private Linear Sketches
In this section, we present new algorithms for differentially private linear sketches.
We highlightthat our Private Count Min/Median only require a different initialization while they share the sameupdate (Algorithm 1) and query (Algorithm 2) with the original Count Min/Median.
Therefore, theimplementation of our algorithms is efficient.
Below we show our private initialization.
In Algorithm 3, the set of arrays we use is C which consists of log(2 /β )
arrays with length 1 /γ ,which has the same space complexity as original Count Min/Median.
Recall that two neighboringdatabases X and X ′ differ by at most one item.
Therefore, after updating all the items respectively,for each corresponding array in C ( X ) and C ( X ′ ) , they differ by at most two elements and thedifference is at most 1 .
Then the ℓ 2 -sensitivity of the set of arrays C is bounded by
[Equation]
By applying the Gaussian Mechanism (Definition 2.5), we can add independent Gaussian noises
N (0 , σ 2 ) to each counter in C , where σ = (cid:113) log(2 /β )
ρ .
Due to the privacy guarantee of Gaussian
Mechanism (Lemma 2.8), it satisfies ∆ 22 2 σ 2 = ρ -zCDP.
Define E ( β, γ, ρ ) = (cid:114) 2log 2 β ρ · (cid:114) log 4 γ log( 2 β ) β , for simplicity, we will use E in Algorithm 3 andthe proof in Appendix A.
The private version of Count-Min can be derived by adding independent
Gaussian noises N ( E, σ 2 ) to each counter of C , while the private version of Count-Median can be derived by adding independent Gaussian noises N (0 , σ 2 ) to each counter of C .
The private versionof Count Min/Median is derived by combining Algorithm 3, Algorithm 1, and Algorithm 2.
3.1 Main results about Private Count Min/Median
We present the privacy guarantee and utility analysis of our Private Count Min/Median below.
Recallthat for each item x , we perform update as in Algorithm 1 and query as in Algorithm 2.
In addition,(cid:98) f ( x ) is the output estimated frequency and f ( x ) is the actual frequency.
To provide a bound for theadditional error due to DP, we define (cid:101) f
( x ) to be the non-private estimated frequency (the output ofthe original Count Min/Median with the same set of hash functions).
We begin with the properties of
Private Count-Min.
Note that all the proofs are deferred to Appendix A.
Theorem 3.1.
Private Count-Min satisfies ρ
-zCDP regardless of the number of queries.
Furthermore, with probability 1 − β , the output (cid:98) f ( x ) satisfies that
[Equation]
In addition, for each item x , with probability 1 − β ,
[Equation]
Comparison to Count-Min.
Comparing our Theorem 3.1 with the conclusion in [Cormode and
Muthukrishnan, 2005], our Private Count-Min preserves the nice property that the output will notunderestimate the frequency with high probability.
Furthermore, within the most popular regimewhere the privacy budget ρ is a constant, the additional error bound due to differential privacy isindependent of the size of database N , therefore it will become negligible as N goes large.
Justification of our Gaussian noise.
Note that with high probability, all the noises we add ( E + σ i,j ,σ i,j ∼ N (0 , σ 2 ) ) will be non-negative.
Therefore, the noise we add and the original error induced by
Count-Min will directly sum up and lead to larger error in evaluation.
However, we claim that theadditional E ensures that with high probability, for all item x , the output will not underestimate theactual frequency.
This nice property enables the good performance of our Private Count-Min whenused in approximate top k task, as shown in Section 5.
Next, Theorem 3.2 shows the properties of Private Count-Median.
Theorem 3.2.
Private Count-Median satisfies ρ -zCDP regardless of the number of queries.
Furthermore, the frequency query from Private Count-Median is unbiased and with probability 1 − β ,
[Equation]
x
[Equation]
) |
[Equation]
E.
In addition, for each item x , with probability 1 − β ,
[Equation]
Comparison to Count-Median.
Comparing our Theorem 3.2 with the conclusion in [Charikar et al.
,2002], our Private Count-Median preserves the nice property that the output will be an unbiasedestimate of the frequency.
This property enables our use of Private Count-Median in quantileestimation below (Section 4).
Furthermore, within the most popular regime where the privacy budgetρ is a constant, the additional error bound due to differential privacy is independent of the size ofdatabase N , thus it will become negligible as N goes large.
3.2 The Uniform Bound of Additional Error
Theorem 3.1 and Theorem 3.2 show a uniform bound sup x | (cid:98) f ( x ) − (cid:101)
f
( x ) | ≤ O ( E ) for linear sketches,which upper bounds the additional error imposed on the estimated frequency due to Differential
Privacy guarantees.
To derive the point-wise bound for | (cid:98)
f ( x ) − f ( x ) | , we combine our result withthe point-wise bound for | (cid:101) f
( x ) − f ( x ) | [Cormode and Muthukrishnan, 2005, Charikar et al.,
2002](note it is straightforward to apply other analyses on the point-wise bound [Minton and Price, 2014,
Larsen et al., 2021] due to the triangle inequality).
Moreover, in Appendix E, we demonstrate thatour analysis provides the tight uniform bound when items are drawn from a large universe.
Private Quantile Sketches
In this section, we apply our Private Count Median to state of the art quantile sketches in the turnstilemodel.
Our private Dyadic Count-Median sketch can estimate all the quantiles accurately at the sametime while ensuring differential privacy.
4.1 Revisiting DCS
In [Wang et al., 2013], it is shown that DCS can return all γ -approximate quantiles with constantprobability using space O (cid:16) 1 γ log 1 .
5 U log 1 .
5 ( log Uγ ) (cid:17) .
More specifically, the sketch structure hereconsists of log U Count Medians, each Count Median uses a counter arrays C , which is d × wcounters.
The choice of d, w follows d = Θ (cid:16) log( log Uγ ) (cid:17) and w = O (cid:16)(cid:113) log U log( log Uγ ) /γ
(cid:17) .
4.2 Private DCS
In this work, we aim to estimate the quantiles accurately while preserving privacy.
We do this by replacing Count Median with Private Count Median, which bases on the same structure as Count
Median discussed above.
Given the privacy budget ρ , the privacy budget of each Private Count
Median is thus ρ 0 = ρ log U , due to composition of zCDP (Lemma 2.7).
The ℓ 2 -sensitivity of each
Private Count Median is
[Equation]
To keep the whole algorithm ρ -zCDP, it suffices to keep each Count Median ρ 0zCDP (Lemma 2.7).
Therefore, Gaussian Mechanism (Definition 2.5) with σ=
O (cid:16) log U log (cid:16) log Uγ (cid:17) /ρ
(cid:17) ensures ρ
-zCDP (Lemma 2.8).
Similar to Lemma A.1, define E ( γ, U ) =
O (cid:32)(cid:114) log U
[Equation]
, we can prove that with constant probability, all the
Gaussian noises we add to all log U Count Medians are bounded by E (for simplicity, we use E torepresent E ( γ, U ) ).
Conditioned on the high probability event above, we prove that for a fixed quantile, the estimated quantile will be accurate with high probability.
As has been proven in Theorem 3.2, the output estimated frequency is unbiased for any item.
Therefore, similar to [Wang et al., 2013], for any item x (corresponding to a fixed Count Median), we have the output (cid:98) f ( x ) of that Count Median satisfies
[Equation]
By a union bound, with probability 1 − log U
× O (cid:16) γ log U (cid:17) = 1 − O ( γ ) , for any item correspondingto this fixed quantile, the error of Count Median is bounded by 1 w · N + E .
Conditioned upon thisevent, by Hoeffding’s inequality, with probability 1 − O (cid:16) γ log U (cid:17) , the sum of log U such independenterrors is bounded by
[Equation]
where E ′ =
O (cid:32) log U log ( log Uγ ) √
[Equation]
log log
[Equation]
log ( log Uγ ) γ (cid:33) .
To sum up, for a fixed quantile, withprobability 1 − O ( γ ) , the estimating error is bounded by γ · N + E ′ .
Finally, apply another union bound on the 1 γ different quantiles, with constant probability, all thequantiles are estimated accurately (within the error bound (2)).
Note that similar to [Wang et al.,
2013], the failure probability here is a constant.
For any failure probability β , we can further increased by a factor of log 1 β to reduce this failure probability to β .
Take-away of Private DCS.
First, our Private DCS has a same space complexity as the original
DCS.
In addition, according to (2), the additional error bound is proportional to log U log 1 γ √ ρ (ignoringlog log terms), and independent to the size of database N .
In the most popular regime where theprivacy budget ρ is a constant, the additional error bound only appears as lower order terms, whichwill become negligible as N goes large.
Evaluation
We have implemented DP linear sketches and DP DCS, and conducted extensive experiments toevaluated the privacy-utility trade-off of our proposed private sketches.
The implementations arewritten in Python with the advantage of fast prototyping and good readability.
5.1 Data Sets
The experimental evaluation is conducted using both synthetic and real world data sets.
We considerthe synthetic Zipf dataset Zipf [2016] with universe size of 2 16 and the source IP addresses from
CAIDA Anonymized Internet Trace 2015 dataset pas with universe size of 2 32 .
For each independentrun in the experiments, we use an input database size N = 10 5 .
5.2 Metrics
In all experiments, we average the various metrics over 5 independent runs to minimize the measurement variance.
The metrics used in the experiments are:
Average Relative Error : Let the set Ψ denotes all unique items in the database.
Average RelativeError (ARE) is computed based on Ψ in which 1Ψ (cid:80) e ∈ Ψ | f ( e ) − (cid:98)
f
( e ) | f
( e ) .
F1 Score : F1 score is the harmonic mean of the precision and recall ( 2 · precision · recall precision + recall ).
Average Rank Error : For each evenly spaced quantile and its associated item, we average thedistance between the true rank and estimated rank.
We use ARE to evaluate sketch performance on frequency estimation and F1 score to evaluate thesketch’s performance in identifying the top 10 items.
For quantile approximation, we consider the mevenly spaced quantiles and items.
For instance, if m = 1 , we consider the rank error for the medianitem; if m = 2 , we consider then average rank error for the 33 rd and 67 th percentile items.
Lower
ARE and average rank error, and higher F1 score indicate better approximation.
5.3 Private Linear Sketches Experiments
To evaluate the utility of DP linear sketches, we compare the average relative error (ARE) and F1score for frequency estimation and identify the top 10 items, respectively.
As shown in Figure 1, thex-axis represents the space budget for each sketch (from 9.2 KB to 147.3 KB), and the y-axis denotes
ARE or F1 score.
The DP linear sketches use ρ ∈ { 0 . 1 , 1 , 10 } in which lower ρ value indicates morenoise need to be added, and all sketches assume β = 1% .
For frequency estimation, the performance of private Count-Median sketches with various privacybudgets is basically equivalent to the performance of the non-private Count-Median sketch.
Underdifferent space and privacy budgets, they have minimal difference in ARE for both Zipf and CAIDAdatasets, meaning that, while providing strong privacy guarantee, the estimated frequencies are stillvery accurate.
The accurate estimation of private Count-Median is primarily due to the unbiasednature of Count-Median in which, by adding Gaussian noise, the private Count-Median still providesunbiased estimation for an item’s frequency as proved in Theorem 3.2.
As shown in both Figure 1(a)and Figure 1(b), the performance of private Count-Min degrades when the space budget increasesor the privacy budget decreases.
This behavior is expected as the upper bound on the frequencyerror in Theorem 3.1 has a dependency on both γ and ρ .
In order to preserve the property of notunderestimating an item’s frequency, the private Count-Min sketch needs to add larger noise to each counter when the number of counters increases.
As a result, the estimated frequencies forlow-frequency items become inflated and this in turn decreases the overall accuracy.
[Figure]
For approximate top 10 items, private Count-Median has similar performance to Count-Median.
Since non-private and private Count-Median are unbiased, they may underestimate the frequency oftrue top K items and decrease the recall.
On the other hand, the property of no underestimation isdesirable for approximate top K items.
In particular, non-private and private Count-Min sketch scorehigh F1 scores for all datasets.
While providing privacy guarantees, private Count-Min achieve 1.
F1 scores for all space and all privacy budgets in Zipf dataset, as shown in Figure 1(c).
5.4 Private Quantile Sketch Experiments
[Figure]
To evaluate the utility of DP DCS, we compare the average rank error.
As shown in Figure 2, thex-axis represents the number of evenly spaced quantiles, and the y-axis denotes the average rank error.
The DP DCS use privacy budget ρ ∈ { 0 . 1 , 1 , 10 } and all sketches assume γ = 1% .
For the quantile approximation, we observe that the increase in the number of evenly spaced quantilesdoes not impact the average rank error, as shown in both Figure 2(a) and Figure 2(b).
Since the
CAIDA dataset universe size ( 2 32 ) is larger than Zipf dataset universe size ( 2 16 ), the average rankerror in the CAIDA dataset is larger than the average rank error in the Zipf dataset.
As shown in
Equation (2), the error bound has a term depending on the universe size in which a large universesize leads to more error.
When the privacy budget decreases, the average rank error increases as morenoise needs to be added.
Comparing DP DCS with strong privacy ( ρ = 0 . 1 ) to DCS, the increase inrank error is relatively small compared to the database size of 10 5 .
In addition, the desired rank errorupper bound is γ · N = 10 3 and all the rank errors are one order of magnitude lower.
Related Works
In this section, we discuss and compare our results to previous literature on Private Count-Min
Sketch [Mir et al., 2011, Melis et al., 2015, Ghazi et al., 2019], and the concurrent work on Private
Count-Median Sketch [Pagh and Thorup, 2022].
In fact, Pagh and Thorup [2022] and us bothindependently discovered the same algorithm for Private Count-Median with differences in thetheoretical analysis.
To the best of our knowledge, we are the first to present a DP quantile sketch inthe turnstile model.
Private Count-Min.
Mir et al.
[2011] proposed to add Laplace noise into the Count-Min Sketchestimator to derive the number of heavy hitters with Pan-Privacy [Dwork et al., 2010].
Similarly, Meliset al.
[2015] add independent Laplace noise to each counter of the sketch instead of the estimator.
However, adding Laplace noise breaks the nice property of never underestimation in Count-Min.
In contrast, our private Count-Min guarantees no underestimation with high probability.
[Ghaziet al., 2019] added one-sided binomial noise into each counter of the sketch to preserve the propertyof no underestimation.
However, using the Binomial mechanism inherently implies approximatedifferential privacy [Canonne et al., 2020].
In contrast, by using the Gaussian mechanism, our Private
Count-Min provides the stronger concentrated differential privacy guarantee.
Private Count-Median.
Pagh and Thorup [2022] and us both independently discovered the samealgorithm for private Count-Median.
There is a major difference in the analysis and we believe bothanalyses are valuable, in which Pagh and Thorup [2022] focused on deriving a tight point-wise boundfor | (cid:98) f ( x ) − f ( x ) | , while we focused on deriving a uniform bound for sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | .
Our uniformbound for sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | can derive the point-wise bound for | (cid:98) f
( x ) − f ( x ) | , by combining ourresult with any point-wise bound for | (cid:101)
f ( x ) − f ( x ) | due to the triangle inequality.
However, Pagh and
Thorup [2022]’s analysis can not even imply the point-wise bound for | (cid:98) f
( x ) − (cid:101) f
( x ) | .
More detailedcomparisons are included in Appendix C.
Conclusion
In this work, we demonstrate that linear sketches can be made differentially private and provideuseful information while maintaining their original properties by adding a small amount of Gaussiannoise at initialization.
In addition, leveraging the private Count-Median sketch, we propose the DP
DCS for quantile approximation in the turnstile model.
DP DCS achieves low rank errors even for alarge data universe.
Moreover, for all the proposed algorithms, when the privacy budget is constant,the additional error due to privacy is independent of the database size and the error will becomenegligible when the database grows larger.
Moreover, private linear sketches bring new opportunitiesfor other statistical questions such as the private euclidean distance estimation [Stausholm, 2021]which can be calculate as the dot product of two private linear sketches.
As a result, we believe ourproposed algorithms are efficient and practical for real-world systems and enable these systems toperform data analysis and machine learning tasks privately.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa-
per’s contributions and scope?
[Yes] Claims in abstract and introduction reflect ourcontributions.
(b) Did you describe the limitations of your work?
[Yes] In Theorem 3.1, the private CountMin frequency estimation’s additional error has a dependency of (cid:113) log 1 γ in order tokeep the property of not underestimating item’s frequency.
Assume database size is
fixed, decreasing gamma will increase the average error.
Note the error is independentfrom the database size, and when database size grows, the error will become negligible.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Apply
our proposed algorithms to current systems will protect user privacy.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We put some proves
in Appendix A due to space limits
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will provide
an url to our Github Repo once the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide the parameters used for the experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A] We didn’t use random seed
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[N/A] we didn’t use any external
resources beside a macbook pro.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use
existing assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not use crowdsourcing or conducted research with humansubjects.
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]