Differentially Private Linear Sketches: Efficient Implementations and Applications
Abstract
Linear sketches have been widely adopted to process fast data streams, and they can be used to accurately answer frequency estimation, approximate top K items, and summarize data distributions.
When data are sensitive, it is desirable to provide privacy guarantees for linear sketches to preserve private information while delivering useful results with theoretical bounds.
We show that linear sketches can ensure privacy and maintain their unique properties with a small amount of noise added at initialization.
From the differentially private linear sketches, we showcase that the state-of-the-art quantile sketch in the turnstile model can also be private and maintain high performance.
Experiments further demonstrate that our proposed differentially private sketches are quantitatively and qualitatively similar to noise-free sketches with high utilization on synthetic and real datasets.
Introduction
Data sketches are fundamental tools for data analysis, statistics, and machine learning [Cormode and
Yi, 2020].
Two of the most widely studied problems in data summaries are frequency estimation and quantile approximation.
Many real world applications need to estimate the frequency of each item in the database and understand the overall distribution of the database.
These applications include stream processing [Das et al., 2009, Bailis et al., 2017], database management [Misra and Gries, 1982,
Metwally et al., 2005, Zhao et al., 2022], caching [Zakhary et al., 2020], system monitoring [Gupta et al., 2016, Ivkin et al., 2019, Zhao et al., 2021], federated learning [Rothchild et al., 2020], among others.
On one hand, the motivation for data sketch algorithms is to efficiently process a large database and extract useful knowledge, since computing the exact information for a large amount of data is both time and memory intensive.
For instance, Munro and Paterson [1980] proved that to find the true median of a database with n items using p sequential passes requires at least Ω( n 1 /p ) memory.
On the other hand, to protect user-level privacy, privacy-preserving algorithms limit the disclosure of private information in the database so that an observer cannot infer much about an individual.
Recent works have shown that data sketches can be integrated with privacy-enhancing technologies to provide insightful information and preserve individual privacy at the same time [Cormode, 2022].
Differential privacy [Dwork et al., 2006] is a widely-accepted definition of privacy.
Recently, researchers have observed that some data sketches are inherently differentially private [Blocki et al., 2012, Smith et al., 2020], while many other data sketches need modifications to the algorithm to be differentially private.
In particular, a substantial amount of literature has focused on differentially private data sketches for tasks such as linear algebra [Upadhyay, 2014, Arora et al., 2018], cardinality estimation [Mir et al., 2011, Pagh and Stausholm, 2021, Dickens et al., 2022] and quantile approximation [Tzamos et al., 2020, Gillenwater et al., 2021, Alabi et al., 2022].
In this paper, we introduce new differentially private algorithms that support both insertions and deletions for frequency, top k, and quantile approximation.
While many data sketches assume an insertion-only model [Greenwald and Khanna, 2001, Shrivastava et al., 2004, Karnin et al., 2016] or a bounded-deletion model [Jayaram and Woodruff, 2018, Zhao et al., 2022, 2021], our algorithms build on top of linear sketches [Charikar et al., 2002, Cormode and Muthukrishnan, 2005] and operate in the turnstile model, which allows an arbitrary number of insertions and deletions into the database.
Earlier, researchers attempted to prove CountSketch [Charikar et al., 2002] itself preserves differential privacy, but the authors acknowledged that there are issues in the proof [Li et al., 2019].
Instead of proving that linear sketches, i.e., both Count-Min and CountSketch, are inherently differentially private, we add a small amount of Gaussian noise at their initialization to provide a privacy guarantee, while maintaining linear sketches’ original properties, providing high utility for frequency and top
K estimations, and keeping update and query algorithms unchanged.
We also demonstrate that our analysis provides the tight uniform bound (Section 3.1 and Appendix E).
In addition, we propose the first differentially private quantile sketch in the turnstile model by leveraging the differentially private linear sketch.
Our differentially private sketches can be queried an arbitrary number of times without affecting privacy guarantees based on the post-processing immunity.
Following prior works [Choi et al., 2020, Smith et al., 2020], we assume ideal random hash functions exist, and this assumption can be replaced in practice by cryptographic hash functions [Dickens et al., 2022].
Preliminaries
Consider a database X
= { i t } t ∈
[ N ] of N items that are drawn from a large universe of size U , such as IPv4 address of size 2 32 , and for each insert or delete operations, one item can be inserted into or deleted from the database X .
To support ordered statistic such as quantile, we assume that the universe is some finite totally ordered data universe.
Definition 2.1.
Given a database X , the frequency of an item x is f ( x ) = (cid:80)
Nt =1 π ( i t = x ) where π returns 1 if i t is x , and 0 otherwise.
Definition 2.2.
Given a database X of items drawn from an ordered universe, the rank of an item x is R ( x ) = (cid:80) Nt =1 π ( i t ≤ x ) where π returns 1 if i t is less or equal to x and 0 otherwise.
Given the large size of N , calculating the actual statistics, such as frequency and quantile, is often hard, and hence most applications are satisfied with an approximation .
The randomized frequency estimation problem takes an accuracy parameter γ and a failure probability β such that, for any item x , | (cid:98) f
( x ) − f ( x ) | ≤ γ · N with high probability 1 − β , where (cid:98) f ( x ) is the estimated frequency and f ( x ) is the true frequency [Cormode and Hadjieleftheriou, 2008].
In addition, the randomized quantile approximation problem also takes an accuracy parameter γ and a failure probability β
such that, for any item x , | (cid:98) R ( x ) − R ( x ) | ≤ γ · N with high probability 1 − β where (cid:98) R ( x ) is the estimated rank and R ( x ) is the actual rank [Karnin et al., 2016].
2.1 Differential Privacy
Definition 2.3.
Databases X and X ′ are neighbors ( X ∼ X ′ ), if they differ in at most one element.
Through this paper, we assume the update/replace definition of differential privacy instead of add/remove definition of differential privacy, in which one item in X is updated or replaced by another item in X ′
[Vadhan, 2017].
Definition 2.4 (Differential Privacy [Dwork et al., 2006]) .
A randomized algorithm M satisfies ( ϵ, δ ) -differential
privacy
( ϵ, δ ) -DP)
if for all neighboring databases X , X ′ and for all possible events E in the output range of M , we have
[Equation]
When δ = 0 , ϵ -DP is known as pure DP, and when δ > 0 , ( ϵ , δ )-DP is known as approximate DP.
Definition 2.5 (Gaussian Mechanism [Dwork et al., 2006]) .
Define the ℓ 2 sensitivity of a function f : N X (cid:55)→ R d as
[Equation]
The Gaussian mechanism M with noise level σ is then given by
[Equation]
Specifically, the Gaussian mechanism is known to satisfy a stronger notion of privacy known as zeroconcentrated differential privacy (zCDP, defined below); zCDP lies between pure and approximate
DP and can be translated into standard DP notations, as shown in Lemma 2.9.
Moreover, zCDP satisfies cleaner composition theorems, as shown in Lemma 2.7.
Definition 2.6 (zCDP
[Dwork and Rothblum, 2016, Bun and Steinke, 2016]) .
A randomized mechanism M satisfies ρ -Zero-Concentrated Differential Privacy ( ρ -zCDP), if for all neighboring databases
X, X ′ and all α ∈ (1 , ∞ ) ,
[Equation]
where D α is the Renyi divergence [Van Erven and Harremos, 2014].
Lemma 2.7 (Adaptive composition and Post Processing of zCDP
[Bun and Steinke, 2016]) .
Let
M : X n → Y and M ′ :
X n × Y → Z .
Suppose M satisfies ρ -zCDP and M ′ satisfies ρ ′
-zCDP (as a function of its first argument).
Define M ′′ : X n → Z by M ′′ ( x ) =
M ′ ( x, M ( x )) .
Then M ′′ satisfies ( ρ + ρ ′ )
-zCDP.
Lemma 2.8 (Privacy Guarantee of Gaussian mechanism [Dwork et al., 2014, Bun and Steinke, 2016]) .
Let f : N X (cid:55)→ R d be an arbitrary d-dimensional function with ℓ 2 sensitivity ∆ 2 = sup neighboring X,X ′ ∥
f ( X ) − f ( X ′ )
∥ 2 .
Then for any ρ > 0 , Gaussian Mechanism with parameter σ 2 = ∆ 22 2 ρ satisfies ρ -zCDP.
Lemma 2.9 (Converting zCDP to DP [Bun and Steinke, 2016]) .
If M satisfies ρ -zCDP then M satisfies ( ρ + 2 (cid:112) ρ log(1 /δ ) , δ ) -DP.
As we use exclusively Gaussian mechanisms and their composition in our proposed algorithms, our method actually satisfies ( ϵ, δ ) -DP guarantees with stronger parameters than what is implied by zCDP via techniques from [Balle and Wang, 2018, Dong et al., 2019], which reduces the ϵ parameter by a sizable fraction in typical parameter regimes.
We stick to zCDP for clarity and generality, because all our results would apply without changes if we modify the noise into other mechanisms satisfying zCDP, e.g., the Discrete Gaussian Mechanism [Canonne et al., 2020].
2.2 Revisiting Linear Sketches
Charikar et al.
[2002] proposed the CountSketch , a randomized algorithm that summarizes a database and solves the frequency estimation problem.
The CountSketch uses a d × w array of counters, i.e, C[d, w], where all the counters are initialized to zero , and has two sets of independent hash functions h and g .
For each row r , the hash function h r maps input items uniformly onto { 1 , . . .
, w } and the hash function g r maps input items uniformly onto {− 1 , +1 } .
For item x with value v ∈ {− 1 , +1 } ,
CountSketch updates d counters, one per each row, based on the hash values such that for a particular row r , g r ( x ) will be added or subtracted to the counter at the h r ( x ) th index depending on whether x is being inserted or deleted respectively, as shown in Algorithm 1.
Hence, the update time is O ( d ) .
To estimate the frequency of item x , CountSketch will output the median 1 ≤ r ≤ d g r ( x ) · C [ r, h r ( x )] , as shown in Algorithm 2.
By updating each row’s counter based on the hashed value of either 1 or − 1 and reporting the median for query, CountSketch provides an unbiased estimate.
To reduce the failure probability of bad estimations, d is set to O (log(1 /β )) and it uses O ( 1 γ log( 1 β )) space to solve the frequency estimation problem.
Cormode and Muthukrishnan [2005] proposed the Count-Min sketch that shares the same initialization, update, and data structure as CountSketch.
Count-Min sketch also uses O ( 1 γ log( 1 β )) space to solve the frequency estimation problem.
A major difference is that Count-Min sketch makes all hash functions in set g return positive 1.
As a result, to estimate the frequency of item x , Count-Min sketch returns min 1 ≤ r ≤ d C [ r, h r ( x )] instead of the median, as shown in Algorithm 2.
In addition, it has the nice property of never underestimating item’s frequency.
Since linear sketches can approximate an item’s frequency accurately, they also solves the top K approximation problem by returning the K items associated with the highest estimated frequency.
Gilbert et al.
[2002] made the connection between frequency and quantiles, in which the quantile range can be decomposed into at most log U dyadic intervals [Cormode et al., 2019] and the sum of the estimated frequencies for these intervals gives the estimated rank.
Wang et al.
[2013] leveraged the unbiased property of CountSketch and proposed the Dyadic CountSketch (DCS) to estimate the frequencies of each dyadic interval.
For more specific details, Appendix B and [Cormode and Yi, 2020] provide a comprehensive analysis of quantile sketches.
Private Linear Sketches
In this section, we present new algorithms for differentially private linear sketches.
We highlight that our Private Count-Min and CountSketch only require a different initialization while they share the same update (Algorithm 1) and query (Algorithm 2) with the original Count-Min and CountSketch.
Therefore, the implementation of our algorithms is efficient.
Below we show our private initialization.
In Algorithm 3, the set of arrays we use is C which consists of log(2 /β )
arrays with length 1 /γ , which has the same space complexity as original Count-Min and CountSketch.
Recall that two neighboring databases X and X ′ differ by at most one item.
Therefore, after updating all the items respectively, for each corresponding array in C ( X ) and C ( X ′ ) , they differ by at most two elements and the difference is at most 1 .
Then the ℓ 2 -sensitivity of the set of arrays C is bounded by
[Equation]
By applying the Gaussian Mechanism (Definition 2.5), we can add independent Gaussian noises
N (0 , σ 2 ) to each counter in C , where σ = (cid:113) log(2 /β )
ρ .
Due to the privacy guarantee of Gaussian
Mechanism (Lemma 2.8), it satisfies ∆ 22 2 σ 2 = ρ -zCDP.
Define E ( β, γ, ρ ) = (cid:114) 2log 2 β ρ · (cid:114) log [Equation] , for simplicity, we will use E in Algorithm 3 and the proof in Appendix A.
The private version of Count-Min can be derived by adding independent
Gaussian noises N ( E, σ 2 ) to each counter of C , while the private version of CountSketch can be derived by adding independent Gaussian noises N (0 , σ 2 ) to each counter of C .
The private versions of Count-Min and CountSketch are derived by combining Algorithm 3, Algorithm 1, and Algorithm 2.
3.1 Main results about Private Count-Min and CountSketch
We present the privacy guarantee and utility analysis of our Private Count-Min and CountSketch below.
Recall that for each item x , we perform update as in Algorithm 1 and query as in Algorithm 2.
In addition, (cid:98) f ( x ) is the output estimated frequency and f ( x ) is the actual frequency.
To provide a bound for the additional error due to DP, we define (cid:101) f
( x ) to be the non-private estimated frequency (the output of the original Count-Min and CountSketch with the same set of hash functions).
We begin with the properties of Private Count-Min.
Note that all the proofs are deferred to Appendix A.
Theorem 3.1.
Private Count-Min satisfies ρ
-zCDP regardless of the number of queries.
Furthermore, with probability 1 − β , the output (cid:98) f ( x ) satisfies that
[Equation]
In addition, for each item x , with probability 1 − β ,
[Equation]
Comparison to Count-Min.
Comparing our Theorem 3.1 with the conclusion in [Cormode and
Muthukrishnan, 2005], our Private Count-Min preserves the nice property that the output will not underestimate the frequency with high probability.
Furthermore, within the most popular regime where the privacy budget ρ is a constant, the additional error bound due to differential privacy is independent of the size of database N , therefore it will become negligible as N goes large.
Justification of our Gaussian noise.
Note that with high probability, all the noises we add ( E + σ i,j , σ i,j ∼ N (0 , σ 2 ) ) will be non-negative.
Therefore, the noise we add and the original error induced by
Count-Min will directly sum up and lead to larger error in evaluation.
However, we claim that the additional E ensures that with high probability, for all item x , the output will not underestimate the actual frequency.
This nice property enables the good performance of our Private Count-Min when used in approximate top k task, as shown in Section 5.
Next, Theorem 3.2 shows the properties of Private CountSketch.
Theorem 3.2.
Private CountSketch satisfies ρ -zCDP regardless of the number of queries.
Furthermore, the frequency query from Private CountSketch is unbiased and with probability 1 − β ,
[Equation]
In addition, for each item x , with probability 1 − β ,
[Equation]
Comparison to CountSketch.
Comparing our Theorem 3.2 with the conclusion in [Charikar et al., 2002], our Private CountSketch preserves the nice property that the output will be an unbiased estimate of the frequency.
This property enables our use of Private CountSketch in quantile estimation below (Section 4).
Furthermore, within the most popular regime where the privacy budget ρ is a constant, the additional error bound due to differential privacy is independent of the size of database N , thus it will become negligible as N goes large.
3.2 The Uniform Bound of Additional Error
Theorem 3.1 and Theorem 3.2 show a uniform bound sup x | (cid:98) f ( x ) − (cid:101)
f
( x ) | ≤ O ( E ) for linear sketches, which upper bounds the additional error imposed on the estimated frequency due to Differential
Privacy guarantees.
To derive the point-wise bound for | (cid:98)
f ( x ) − f ( x ) | , we combine our result with the point-wise bound for | (cid:101) f
( x ) − f ( x ) | [Cormode and Muthukrishnan, 2005, Charikar et al., 2002]
(note it is straightforward to apply other analyses on the point-wise bound [Minton and Price, 2014,
Larsen et al., 2021] due to the triangle inequality).
Moreover, in Appendix E, we demonstrate that our analysis provides the tight uniform bound when items are drawn from a large universe.
Private Quantile Sketches
In this section, we apply our Private CountSketch to state of the art quantile sketches in the turnstile model.
Our private Dyadic CountSketch can estimate all the quantiles accurately at the same time while ensuring differential privacy.
4.1 Revisiting DCS
In [Wang et al., 2013], it is shown that DCS can return all γ -approximate quantiles with constant probability using space O (cid:16) 1 γ log 1 .
5 U log 1 .
5 ( log Uγ ) (cid:17) .
More specifically, the sketch structure here consists of log U CountSketches, each CountSketch uses a counter arrays C , which is d × w counters.
The choice of d, w follows d = Θ (cid:16) log( log Uγ ) (cid:17) and w = O (cid:16)(cid:113) log U log( log Uγ ) /γ
(cid:17) .
4.2 Private DCS
In this work, we aim to estimate the quantiles accurately while preserving privacy.
We do this by replacing CountSketch with PrivateCountSketch, which bases on the same structure as CountSketch discussed above.
Given the privacy budget ρ , the privacy budget of each Private CountSketch is thus ρ 0 = ρ log U , due to composition of zCDP (Lemma 2.7).
The ℓ 2 -sensitivity of each Private
CountSketch is
[Equation]
To keep the whole algorithm ρ -zCDP, it suffices to keep each CountSketch ρ 0zCDP (Lemma 2.7).
Therefore, Gaussian Mechanism (Definition 2.5) with σ=
O (cid:16) log U log (cid:16) log Uγ (cid:17) /ρ
(cid:17) ensures ρ
-zCDP (Lemma 2.8).
Similar to Lemma A.1, define E ( γ, U ) =
O (cid:32)(cid:114)
[Equation]
, we can prove that with constant probability, all the Gaussian noises we add to all log U CountSketches are bounded by E (for simplicity, we use E to represent E ( γ, U ) ).
Conditioned on the high probability event above, we prove that for a fixed quantile, the estimated quantile will be accurate with high probability.
As has been proven in Theorem 3.2, the output estimated frequency is unbiased for any item.
Therefore, similar to [Wang et al., 2013], for any item x (corresponding to a fixed CountSketch), we have the output (cid:98) f ( x ) of that CountSketch satisfies
[Equation]
By a union bound, with probability 1 − log U
× O (cid:16) γ log U (cid:17) = 1 − O ( γ ) , for any item corresponding to this fixed quantile, the error of CountSketch is bounded by 1 w · N + E .
Conditioned upon this event, by Hoeffding’s inequality, with probability 1 − O (cid:16) γ log U (cid:17) , the sum of log U such independent errors is bounded by
[Equation]
where E ′ =
O (cid:32) log U log ( log Uγ ) √ ρ ·
[Equation]
log log U log ( log Uγ ) γ (cid:33) .
To sum up, for a fixed quantile, with probability 1 − O ( γ ) , the estimating error is bounded by γ · N + E ′ .
Finally, apply another union bound on the 1 γ different quantiles, with constant probability, all the quantiles are estimated accurately (within the error bound (2)).
Note that similar to [Wang et al., 2013], the failure probability here is a constant.
For any failure probability β , we can further increase d by a factor of log 1 β to reduce this failure probability to β .
Take-away of Private DCS.
First, our Private DCS has a same space complexity as the original
DCS.
In addition, according to (2), the additional error bound is proportional to log U log 1 γ √ ρ (ignoring log log terms), and independent to the size of database N .
In the most popular regime where the privacy budget ρ is a constant, the additional error bound only appears as lower order terms, which will become negligible as N goes large.
Evaluation
We have implemented DP linear sketches and DP DCS, and conducted extensive experiments to evaluated the privacy-utility trade-off of our proposed private sketches.
The implementations are written in Python with the advantage of fast prototyping and good readability. The code for the following experiments can be found on Github 3 .
5.1 Data Sets
The experimental evaluation is conducted using both synthetic and real world data sets.
We consider the synthetic Zipf dataset Zipf [2016] with universe size of 2 16 and the source IP addresses from
CAIDA Anonymized Internet Trace 2015 dataset pas with universe size of 2 32 .
For each independent run in the experiments, we use an input database size N = 10 5 .
5.2 Metrics
In all experiments, we average the various metrics over 5 independent runs to minimize the measurement variance.
The metrics used in the experiments are:
Average Relative Error : Let the set Ψ denotes all unique items in the database.
Average Relative Error (ARE) is computed based on Ψ in which 1Ψ (cid:80) e ∈ Ψ | f ( e ) − (cid:98)
f
( e ) | f
( e ) .
F1 Score : F1 score is the harmonic mean of the precision and recall ( 2 · precision · recall precision + recall ).
Average Rank Error : For each evenly spaced quantile and its associated item, we average the distance between the true rank and estimated rank.
We use ARE to evaluate sketch performance on frequency estimation and F1 score to evaluate the sketch’s performance in identifying the top 10 items.
For quantile approximation, we consider the m evenly spaced quantiles and items.
For instance, if m = 1 , we consider the rank error for the median item; if m = 2 , we consider then average rank error for the 33 rd and 67 th percentile items.
Lower
ARE and average rank error, and higher F1 score indicate better approximation.
5.3 Private Linear Sketches Experiments
To evaluate the utility of DP linear sketches, we compare the average relative error (ARE) and F score for frequency estimation and identify the top 10 items, respectively.
As shown in Figure 1, the x-axis represents the space budget for each sketch (from 9.2 KB to 147.3 KB), and the y-axis denotes
ARE or F1 score.
The DP linear sketches use ρ ∈ { 0 . 1 , 1 , 10 } in which lower ρ value indicates more noise need to be added, and all sketches assume β = 1% .
For frequency estimation, the performance of our private CountSketch with various privacy budgets is basically equivalent to the performance of the non-private CountSketch.
Under different space and privacy budgets, they have minimal difference in ARE for both Zipf and CAIDA datasets, meaning that, while providing strong privacy guarantee, the estimated frequencies are still very accurate.
The accurate estimation of private CountSketch is primarily due to the unbiased nature of CountSketch in which, by adding Gaussian noise, the private CountSketch still provides unbiased estimation for an item’s frequency as proved in Theorem 3.2.
As shown in both Figure 1(a) and Figure 1(b), the performance of private Count-Min degrades when the space budget increases or the privacy budget decreases.
This behavior is expected as the upper bound on the frequency error in Theorem 3.1 has a dependency on both γ and ρ .
In order to preserve the property of not underestimating an item’s frequency, the private Count-Min sketch needs to add larger noise to each counter when the number of counters increases.
As a result, the estimated frequencies for low-frequency items become inflated and this in turn decreases the overall accuracy.
[Figure]
For approximate top 10 items, private CountSketch has similar performance to CountSketch.
Since both non-private and private CountSketch are unbiased, they may underestimate the frequency of true top K items and decrease the recall.
On the other hand, the property of no underestimation is desirable for approximate top K items.
In particular, non-private and private Count-Min sketch score high F1 scores for all datasets.
While providing privacy guarantees, private Count-Min achieve [-1.
-]1.
F1 scores for all space and all privacy budgets in Zipf dataset, as shown in Figure 1(c).
5.4 Private Quantile Sketch Experiments
[Figure]
To evaluate the utility of DP DCS, we compare the average rank error.
As shown in Figure 2, the x-axis represents the number of evenly spaced quantiles, and the y-axis denotes the average rank error.
The DP DCS use privacy budget ρ ∈ { 0 . 1 , 1 , 10 } and all sketches assume γ = 1% .
For the quantile approximation, we observe that the increase in the number of evenly spaced quantiles does not impact the average rank error, as shown in both Figure 2(a) and Figure 2(b).
Since the
CAIDA dataset universe size ( 2 32 ) is larger than Zipf dataset universe size ( 2 16 ), the average rank error in the CAIDA dataset is larger than the average rank error in the Zipf dataset.
As shown in
Equation (2), the error bound has a term depending on the universe size in which a large universe size leads to more error.
When the privacy budget decreases, the average rank error increases as more noise needs to be added.
Comparing DP DCS with strong privacy ( ρ = 0 . 1 ) to DCS, the increase in rank error is relatively small compared to the database size of 10 5 .
In addition, the desired rank error upper bound is γ · N = 10 3 and all the rank errors are one order of magnitude lower.
Related Works
In this section, we discuss and compare our results to previous literature on Private Count-Min
Sketch [Mir et al., 2011, Melis et al., 2015, Ghazi et al., 2019], and the concurrent work on Private
CountSketch [Pagh and Thorup, 2022].
In fact, Pagh and Thorup [2022] and us both independently discovered the same algorithm for Private CountSketch with differences in the theoretical analysis.
To the best of our knowledge, we are the first to present a DP quantile sketch in the turnstile model.
Private Count-Min.
Mir et al.
[2011] proposed to add Laplace noise into the Count-Min Sketch estimator to derive the number of heavy hitters with Pan-Privacy [Dwork et al., 2010].
Similarly, Melis et al.
[2015] add independent Laplace noise to each counter of the sketch instead of the estimator.
However, adding Laplace noise breaks the nice property of never underestimation in Count-Min.
In contrast, our private Count-Min guarantees no underestimation with high probability.
[Ghazi et al., 2019] added one-sided binomial noise into each counter of the sketch to preserve the property of no underestimation.
However, using the Binomial mechanism inherently implies approximate differential privacy [Canonne et al., 2020].
In contrast, by using the Gaussian mechanism, our Private
Count-Min provides the stronger concentrated differential privacy guarantee.
Private CountSketch.
Pagh and Thorup [2022] and us both independently discovered the same algorithm for private CountSketch.
There is a major difference in the analysis and we believe both analyses are valuable, in which Pagh and Thorup [2022] focused on deriving a tight point-wise bound for | (cid:98) f ( x ) − f ( x ) | , while we focused on deriving a uniform bound for sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | .
Our uniform bound for sup x | (cid:98) f ( x ) − (cid:101) f
( x ) | can derive the point-wise bound for | (cid:98) f
( x ) − f ( x ) | , by combining our result with any point-wise bound for | (cid:101)
f ( x ) − f ( x ) | due to the triangle inequality.
[Pagh and Thorup, 2022] obtains a tighter point-wise bound by using concentration of median instead of triangle inequality. However, Pagh and Thorup [2022]’s analysis can not imply the point-wise bound for | (cid:98) f
( x ) − (cid:101) f
( x ) | .
More detailed comparisons are included in Appendix C.
Conclusion
In this work, we demonstrate that linear sketches can be made differentially private and provide useful information while maintaining their original properties by adding a small amount of Gaussian noise at initialization.
In addition, leveraging the private CountSketch, we propose the DP DCS for quantile approximation in the turnstile model.
DP DCS achieves low rank errors even for a large data universe.
Moreover, for all the proposed algorithms, when the privacy budget is constant, the additional error due to privacy is independent of the database size and the error will become negligible when the database grows larger.
Moreover, private linear sketches bring new opportunities for other statistical questions such as the private euclidean distance estimation [Stausholm, 2021] which can be calculate as the dot product of two private linear sketches.
As a result, we believe our proposed algorithms are efficient and practical for real-world systems and enable these systems to perform data analysis and machine learning tasks privately. Acknowledgments and Disclosure of Funding
This work is partially supported by gifts from Snowflake Inc, and NSF grants CNS-1703560, CNS1815733 and CNS-2048091.
The authors thank Rasmus Pagh for a helpful discussion regarding their concurrent work [Pagh and Thorup, 2022].
The authors also thank Adam Smith for clarifying the mergeability in the inherently private Flajolet-Martin Sketch [Smith et al., 2020].
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] Claims in abstract and introduction reflect our contributions.
(b) Did you describe the limitations of your work?
[Yes] In Theorem 3.1, the private CountMin frequency estimation’s additional error has a dependency of (cid:113) log 1 γ in order to keep the property of not underestimating item’s frequency.
Assume database size is fixed, decreasing gamma will increase the average error.
Note the error is independent from the database size, and when database size grows, the error will become negligible.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Apply our proposed algorithms to current systems will protect user privacy.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] We put some proves in Appendix A due to space limits 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will provide an url to our Github Repo once the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We provide the parameters used for the experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] We didn’t use random seed (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] we didn’t use any external resources beside a macbook pro. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use existing assets.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing or conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In this section, we present the missing proofs. Recall that for each item x , we perform query as in Algorithm 2. In the proof below, we use arr ′ to denote the arr in Algorithm 2 (with private initialization) while arr denotes the arr in Algorithm 2 under non-private initialization (all zero initialization) and the same set of hash functions.
In addition, (cid:98) f ( x ) is the output estimated frequency, f ( x ) is the actual frequency and (cid:101) f ( x ) is the non-private estimated frequency (the output of the original Count-Min and CountSketch with the same set of hash functions). We first present the following Lemma A.1, which gives a high probability bound for the Gaussian noises we add. Lemma A.1 (Utility analysis) .
If there are 1 γ log( 2 β ) independent Gaussian noises sampled from
N (0 , σ 2 ) (where σ = (cid:113) log(2 /β ) ρ ), denoted as σ ij where i ∈ [log(2 /β )] , j ∈ [1 /γ ] , then with probability 1 − β 2 , for any i ∈ [log(2 /β )] , j ∈ [1 /γ ] ,
[Equation]
Proof of Lemma A.1.
The lemma directly results from the concentration inequality of Gaussian distribution and a union bound.
Theorem A.2 (Restate Theorem 3.1) .
Private Count-Min satisfies ρ
-zCDP regardless of the number of queries.
Furthermore, with probability 1 − β , the output (cid:98) f ( x ) satisfies that
[Equation]
In addition, for each item x , with probability 1 − β ,
[Equation]
≤ γ · N + 2 E.
Proof of Theorem A.2.
Differential privacy directly results from the DP guarantee of Gaussian
Mechanism (Lemma 2.8) and post processing (Lemma 2.7).
According to Lemma A.1, we have with probability 1 − β 2 , for all noises E + σ ij , where σ ij ∼
N (0 , σ 2 ) , i ∈ [log(2 /β )] , j ∈ [1 /γ ] , it holds that
[Equation]
ij
[Equation]
Under the above case that will happen with probability 1 − β 2 , for all x and the corresponding arr , arr ′ , it holds that
[Equation]
Therefore, for all x ,
[Equation]
For the last conclusion (point-wise bound), by the property of Count-Min [Cormode and Muthukrishnan, 2005], we have for any item x , with probability 1 − β 2 ,
[Equation]
Therefore, for any item x , with probability 1 − β , we have
[Equation]
and
[Equation]
Then the proof is completed by plugging in the definition of E .
Theorem A.3 (Restate Theorem 3.2) .
Private CountSketch satisfies ρ
-zCDP regardless of the number of queries.
Furthermore, the frequency query from Private CountSketch is unbiased and with probability 1 − β ,
[Equation]
In addition, for each item x , with probability 1 − β ,
[Equation]
Proof of Theorem A.3.
First of all, differential privacy directly results from the DP guarantee of
Gaussian Mechanism (Lemma 2.8) and post processing (Lemma 2.7).
Next we claim that the conclusion that E (cid:98) f ( x ) =
f ( x ) arises from symmetry.
If we replace { h i } i ∈ [log(2 /β )] , { g i } i ∈ [log(2 /β )] , { σ ij } i ∈ [log(2 /β )] ,j ∈ [1 /γ ] with { h i } i ∈ [log(2 /β )] , { g ′ i } i ∈ [log(2 /β )] , {− σ ij } i ∈ [log(2 /β )] ,j ∈ [1 /γ ] where g ′ i ( x ) =
g i ( x ) and
g ′ i ( x ′ )
= − g i ( x ′ ) , ∀ x ′ ̸
= x , then the outputs under these two cases will be symmetric around f ( x ) and the probability distribution function at these two cases are identical.
Therefore, we have
[Equation]
According to Lemma A.1, we have with probability 1 − β 2 , for all noises σ ij ∼ N (0 , σ 2 ) , i ∈ [log(2 /β )] , j ∈ [1 /γ ] , it holds that
[Equation]
Under the above case that will happen with probability 1 − β 2 , we will prove that for all x ,
[Equation]
≤ E.
Without loss of generality, we can assume that log 2 β
= 2 k
+ 1 , where k is a positive integer
(we can choose k to be the minimum integer such that 2 k + 1 ≥ log 2 β ).
Suppose that median(arr ′ ) > mediam(arr)
+ E , then it holds that there are at least k + 1 elements in arr ′ that is larger than mediam(arr)
+ E due to the definition of median.
Because | σ ij | ≤ E, for all i, j , there are at least k + 1 elements in arr that is larger than mediam(arr) , which leads to contradiction.
As a result, we have
[Equation]
Similarly, (cid:98) f
( x ) − (cid:101) f
( x ) ≥ − E. Combining these two results, for all x , it holds that
[Equation]
Finally, we prove the point-wise bound.
By the property of CountSketch [Charikar et al., 2002], we have for any item x , with probability 1 − β 2 ,
[Equation]
Therefore, according to triangle inequality, for each item x , with probability 1 − β ,
[Equation]
Then the proof is completed by plugging in the definition of E .
B Missing Quantile Algorithms
Gilbert et al.
[2002] made the connection between frequency and quantiles to propose the first universe based RSS quantile sketch in the turnstile model.
Observe, that the relationship between frequency and rank is that one can sum up all items’ frequency in the range of 0 to the item itself to estimate the rank.
However, this naive approach requires summing all items’ frequencies in the range, and the error quickly escalates.
A better approach is to break the range from 0 to item x into at most log U dyadic intervals [Cormode et al., 2019] and then sum all frequencies for each dyadic interval to obtain the estimation of the rank( x ).
Cormode and Muthukrishnan [2005] proposed the Dyadic Count-Min sketch (DCM) which uses Count-Min sketches for estimating the frequencies of each dyadic interval with space complexity O ( 1 γ log 2 U log log Uγ ) and update time O (log U log log Uγ ) .
Later, Wang et al.
[2013] leveraged the unbiased property of CountSketch and proposed the Dyadic CountSketch (DCS) which replaces the Count-Min sketch with the CountSketch [Charikar et al., 2002] to further improve the space complexity to O ( 1 γ log 1 .
5 U log 1 .
5 ( log Uγ ))
while maintaining the same update time.
DCS and DCM share the same update and query algorithms as shown below.
DCM uses
O ( 1 γ log U log log Uγ ) space for each Count-Min sketch and DCS uses O ( 1 γ log 0 .
5 U log 1 .
5 ( log Uγ )) space for each CountSketch, where both of them use O (log log Uγ ) rows.
For more specific details, [Cormode and Yi, 2020] provide a comprehensive analysis of linear and quantile sketches.
Based on the observations, DCS and DCM quantile sketches keeps log U number of linear sketches, one for each dyadic interval.
As a result, to update an item x with value v ∈ {− 1 , +1 } , DCS and
DCM need to update log U levels: they first map item x to a dyadic interval for the level and then update the corresponding linear sketch, as shown in Algorithm 4.
To estimate the rank of an item,
DCS and DCM first break the range into at most log U dyadic intervals and then query the frequency for each interval from the corresponding linear sketch, as shown in Algorithm 5.
C Detailed Comparison for Private CountSketch
Recall that for some item x , (cid:98) f
( x ) is the output estimated frequency, f ( x ) is the actual frequency and (cid:101)
f ( x ) is the non-private estimated frequency (the output of the original Count-Min and CountSketch with the same set of hash functions).
Let k denote the number of rows in our counter, and we choose k = log 2 β to bound the failure probability by β .
We first state both Pagh and Thorup [2022]’s results and ours.
Our uniform bound (Theorem 3.2): sup x | (cid:98) f ( x ) − (cid:101) f
( x ) | ≤ E = (cid:101)
O ( (cid:113) kρ ) .
Our lower bound (Theorem E.1): sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | ≥ Ω( (cid:113) kρ ) .
Pagh and Thorup [2022]’s point-wise bound 4 : | (cid:98) f ( x ) − f ( x ) | ≤ non-private error bound + (cid:101) O
( (cid:113) 1 ρ ) .
There is a major difference between the analysis in [Pagh and Thorup, 2022] and ours.
While Pagh and Thorup [2022] focused on the point-wise bound for | (cid:98) f
( x ) − f ( x ) | , we focused on the uniform bound for sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | .
{+We are interested in the trade-off between privacy and accuracy for the 
+}
We
CountSketch.
In particular, we want to answer the question of what additional error will be imposed on the estimated frequency due to the Differential Privacy guarantee.
As a result, our result shows an uniform bound sup x | (cid:98) f ( x ) − (cid:101) f
( x ) | ≤ E where E is a function of the desired accuracy, failure probability, and privacy guarantee.
To derive the point-wise bound for | (cid:98) f ( x ) − f ( x ) | , we can simply combine our result with any point-wise bound for | (cid:101) f
( x ) − f ( x ) | (like the one in our work or [Pagh and Thorup, 2022]) due to triangular inequality.
However, Pagh and Thorup [2022]’s analysis can not imply even point-wise bound for | (cid:98) f
( x ) − (cid:101) f
( x ) | .
We agree that Pagh and Thorup [2022] has a tight point-wise bound for | (cid:98) f ( x ) − f ( x ) | by using the concentration of median.
By comparing our analysis and [Pagh and Thorup, 2022] for | (cid:98) f ( x ) − f ( x ) | ,
Pagh and Thorup [2022]’s point-wise bound removes the (cid:113) log 2 β ( β denotes failure probability) in the lower order term E which is added to the original estimation error | (cid:101) f
( x ) − f ( x ) | .
However, the difference may not be substantial.
When the database is large (which is the usual case for why we need to perform approximations), E is small compared to the | (cid:101)
f ( x ) − f ( x ) | .
Even for the extreme case of setting β
= 1 e − 10 , the amplification factor for calculating E is (cid:113) log 2 β ≈ 5 . 8 , which E is still very likely to be small compare to | (cid:101)
f ( x ) − f ( x ) | .
We believe that none of the two results dominate each other.
Both Pagh and Thorup [2022]’s pointwise bound for | (cid:98) f
( x ) − f ( x ) | and our uniform bound for | (cid:98) f ( x ) − (cid:101) f
( x ) | are useful analysis for understanding the Differentially Private CountSketch with Gaussian noise.
D Extension to the Data Stream Setting
Our Differentially Private Linear Sketches only guarantee Differential Privacy for the query of database, i.e., the adversary is only allowed to query after the whole database passes our algorithm.
However, queries for data stream is also quite practical in real-life applications, i.e., the adversary can query at any time.
Take reinforcement learning (RL) as an instance, we can use differentially private linear sketches to estimate the visitation number of all (state,action) pairs while preserving privacy.
If we only have linear sketches for database, we can only handle offline RL [Qiao and Wang, 2022], while with linear sketches for data stream, we can deal with the more challenging online RL [Qiao et al., 2022].
Our algorithms can be extended to the data stream setting with moderate modifications.
Different from our approach of adding noise at the beginning (Algorithm 3), we need to add noise after each item passes our algorithm.
To guarantee Differential Privacy under data stream, we can apply the tree-based algorithm (as shown in Kairouz et al.
[2021]) to add Gaussian noises to continuous data.
In this way, the algorithm is Differentially Private no matter how many times the adversary queries the data stream and the additional error bound is the same scale as E in our main theorems, with some extra multiplicative logarithmic terms.
E Lower Bound for the Additional Error due to Privacy
In this section, we provide a lower bound for CountSketch (the counterpart for Count-Min is similar and we omit it here) showing that our analysis of sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | is tight.
For simplicity, we assume the counter arrays
C
we use has
shape k × d and k is odd, which means our upper bound E for sup x | (cid:98) f
( x ) − (cid:101) f
( x ) | is (cid:101) O
( (cid:113) k ρ ) .
Then according to our Algorithm 3, the noise we add has scale (standard variance) (cid:113) kρ .
The following theorem shows that if the universe is large enough, with constant probability, sup x | (cid:98) f ( x ) − (cid:101) f ( x ) | ≥ Ω( (cid:113) kρ ) , which matches our upper bound in Theorem 3.2 up to logarithmic terms.
Theorem E.1.
There exists constants c, p , such that if the size of universe satisfies U ≥ ckd (1 +d − 1 ) k − 1 , for our Private CountSketch, there exists a database, with probability at least p ,
[Equation]
Proof of Theorem E.1.
Fix some item x with its corresponding { h i ( x ) , g i ( x ) } i ∈ [ k ] .
For each i ∈ [ k ] , we aim to find y i in universe such that h i ( y i )
=
h i ( x ) , h j ( y i ) ̸
= h j ( x ) ,
∀ j ̸
= i and g i ( y i )
=
g i ( x ) (this is replaced with g i ( y i ) =
− g i ( x ) if i ≥ k +12 ).
For any item in the universe, due to the uniform randomness of its hash functions, the probability it satisfies such conditions is Pr = 12 d (1 − 1 d ) k − 1 .
Therefore, when U ≥ ckd (1 + 1 d − 1 ) k − 1 for some constant c , with constant probability, we can find { y i } i ∈
[ k ] satisfying the previous conditions.
Next we can construct a database with only { x, y 1 , · · · , y k − 1 } , where the frequency of x is some large n x , and the frequencies of all
y
i ’s are n y that satisfies that n y is much larger than E .
Therefore, the arr of x (without adding noise) consists of one n x , k − 12 numbers much larger than n x + E and k − 12 numbers much smaller than n x − E .
Finally, with high probability, the change from (cid:101) f
( x ) to (cid:98) f
( x ) is from n x to n x + N (0 , kρ ) , which finishes our proof.