Private and Communication-Efﬁcient Algorithms for Entropy Estimation
Abstract
Modern statistical estimation is often performed in a distributed setting where each sample belongs to a single user who shares their data with a central server.
Users are typically concerned with preserving the privacy of their samples, and also with minimizing the amount of data they must transmit to the server.
We give improved private and communication-efﬁcient algorithms for estimating several popular measures of the entropy of a distribution.
All of our algorithms have constant communication cost and satisfy local differential privacy.
For a joint distribution over several variables whose conditional independence given by a tree, we describe algorithms for estimating the Shannon entropy that require a number of samples that is linear in the number of variables, compared to the quadratic sample complexity of prior work.
We also describe an algorithm for estimating the Gini entropy whose sample complexity has no dependence on the support size of the distribution and can be implemented using a single round of concurrent communication between the users and the server.
In contrast, the previously bestknown algorithm has high communication cost and requires the server to facilitate interaction between the users.
Finally, we describe an algorithm for estimating the collision entropy that matches the space and sample complexity of the best known algorithm but generalizes it to the private and communication-efﬁcient setting.
Introduction
Statistical estimation has traditionally focused on minimizing the number of samples needed to estimate properties of a distribution.
In the ‘big data’ era, statisticians and computer scientists have also tried to minimize the space complexity of estimation algorithms, particularly in the streaming setting.
More recently, the increasing prevalence of mobile computing has led to a focus on the privacy and communication costs of statistical estimation.
In this paper, we focus on the following setting: a set of users each draw one sample from a distribution each and share information about their sample with a central server.
The central server then uses the collected data to estimate a property of the distribution.
Users are concerned with preserving the privacy of their sample, and also with minimizing the amount of data that is transmitted to the server.
For example, consider the problem of detecting ﬁngerprinting on the web [30].
Many websites track users across the web without their consent by recording (enough) information about their devices (e.g., installed fonts, operating system, timezone, etc.), a practice known as “browser ﬁngerprinting”.
Entropy is the standard metric used to quantify the identiﬁability of the collected ﬁngerprints.
So a private and distributed method for estimating entropy can be used by a browser to warn users that this covert tracking could occur, without ever storing the ﬁngerprints themselves.
The study of entropies has an extensive and rich history in mathematics and sciences.
Related quantities called “entropy” appear in many contexts (thermodynamics, information theory, dynamic systems [34], category theory [10, 6], etc.).
These may be broadly thought as measures of information of a system or process obeying certain properties, which, in turn, lead to natural measures of disorder, randomness, outcome diversity, information content, uniformity, etc.
In this paper, we study private and communication-efﬁcient algorithms for estimating certain entropies of a distribution.
Speciﬁcally, we give algorithms for estimating the following entropies, which are widely-used in many scientiﬁc ﬁelds to quantify the uncertainty, diversity and spread of a discrete distribution:
• Shannon entropy [32], a fundamental quantity in information theory.
• Gini entropy (also known as Tsallis entropy [37] of order 2, or (one minus the) second frequency moment).
Some of its applications include measuring ecological diversity [33, 27], market competition among ﬁrms [22], effective size of political parties [26], and suitability of features to split on during decision tree learning [29].
• Collision entropy (also known as Rényi entropy [31] of order 2).
Some of its applications include measuring the quality of random number generators [35], and determining the number of reads needed to reconstruct a DNA sequence [28].
Our algorithms are implemented in either the non-interactive model (for the Gini and collision entropies), in which all users simultaneously exchange information with the server during a single round of communication, or the (stronger) sequentially interactive model (for the Shannon entropy), in which the server queries users one at a time, possibly in an adaptive manner [23].
When analyzing the communication complexity of an algorithm, we prove bounds on the number of bits that each user transmits to the server.
However, the server is allowed to broadcast an arbitrary amount of information to the users (this is also called the blackboard model [17]), including shared random bits (also known as the public coin model [1, 2, 24]).
When analyzing the privacy of our algorithms, we use the framework of local differential privacy [18] which ensures that the server learns very little about each user’s data.
Our contributions:
• A sequentially interactive ↵ -local differentially private algorithm for estimating the Shannon entropy of a joint distribution on d variables within ✏ d error using ˜ O ( d/ ↵ 2 ✏ 3 ) samples and O (1) bits per sample.
Our analysis assumes that each of the d variables has a constant support size and that their conditional independence graph is a tree.
We also describe algorithms that have better dependence on 1 / ✏ in certain special cases, such as when the tree has low diameter or is a chain.
Our algorithms achieve O (1) communication complexity by observing only two or three of the d variables in any single sample; we call these pair and triplet observations .
The only previously known algorithm for estimating the Shannon entropy of a tree-structured distribution from pair observations is a non-interactive algorithm due to Chow and Liu [13].
We also prove that any non-interactive algorithm requires ⌦ ( d 2 ) pair observations to achieve O ( d ) error.
We also prove that, for any sequentially interactive algorithm, ⌦ ( d/ ✏ ) pair observations are necessary to achieve O ( ✏ d ) error.
• A non-interactive ↵ -local differentially private algorithm for estimating the Gini entropy of a distribution within ✏ error using O ( b 2 max { 1  g, 2  b } / ( ↵ 2 ✏ 2 )) samples, b bits per sample, and ˜ O ( b ) space, where g 2 [0 , 1] is the Gini entropy of the distribution.
The best previous algorithm [11] has the same sample complexity, even when b = 1 , but is sequentially interactive, and also requires ⌦ ( k ) bits per sample and ⌦ ( k ) space, where k is the support size of the distribution.
Also our error bound also holds with high probability instead of only in expectation.
• A non-interactive ↵ -local differentially private algorithm for estimating the collision entropy of a distribution with support size k within ✏ error using ˜ O
( b 2 k 2 / ( ↵ 2 ✏ 2 min { k, 2 b } )) samples , b bits per sample, and ˜ O ( b ) space.
Setting b = log k and ↵
=
O (1) recovers the sample and space complexity guarantees of the non-interactive algorithm from [35] up to logarithmic factors, and thus, our algorithm generalizes the previously best known algorithm to the private and communication-efﬁcient setting.
Related Work
There is a very extensive literature on distributed statistical estimation under communication constraints (see [39] for the paper that appears to have started this thread).
Variations on the problem include whether communication is allowed between users, whether communication happens in one or multiple rounds, whether there is a shared source of randomness among the users, and whether communication is limited per-user or only cumulatively across all users.
Many previous results in this area bound the sample and communication complexity of estimating the parameters of a distribution P ✓ , where ✓ 2 ⇥ (see e.g. [21]).
This problem class includes discrete distribution estimation, where the guarantees are usually stated as bounds on the relative entropy or total variation distance between the estimated and true distribution (see e.g. [5]).
Other problems of interest are mean estimation [36] and heavy hitter estimation [4].
There has also been signiﬁcant interest in differentially private statistical estimation, and of particular relevance is work by [3], who gave private algorithms for estimating certain functionals of a distribution, including the Shannon entropy.
However, they used the central model of differential privacy, while in this paper we prove guarantees using the (stronger) local model.
Entropy Measures
The Shannon , Tsallis , and Rényi entropy of a discrete random variable X are deﬁned as (Shannon) [Equation] (1) (Tsallis) [Equation]
Pr
[Equation]
(2) (Rényi)
[Equation]
[Equation]
(Pr[
[Equation]  [Equation] (3) where  in (2) and (3) is a free parameter satisfying  > 0 and  6 = 1 .
Both Tsallis and Rényi entropy are generalizations of Shannon entropy in the sense that lim  !
1 T  ( X ) =
lim  !
1 R  ( X ) = H ( X ) .
In this paper, we describe algorithms for estimating the Shannon entropy and special cases of the
Tsallis and the Rényi entropy that are widely used in many scientiﬁc ﬁelds: T 2 ( X ) , also called the
Gini entropy , and R 2 ( X ) , also called the collision entropy .
Substituting  = 2 into the deﬁnitions above and using the abbreviations G ( X ) ⌘ T 2 ( X ) and C ( X ) ⌘ R 2 ( X ) we have
[Equation]
Gini entropy is so-called because it is equivalent to Gini diversity index [20], a statistics proposed by Corrado Gini in 1912 [20] to measure income and wealth inequality.
Collision entropy takes its name from the observation that if X and X 0 are independent and identically distributed then
C ( X ) =
log Pr
[ X = X 0 ] .
For the problem of estimating the Shannon entropy, we specialize to a high-dimensional setting, where we only observe a pair (or triplet ) of the dimensions at a time.
That is, X is a random-vector of d discrete variables, where d is large, but each X i has a constant support size (e.g., the are binary), and we only observe two (or three) dimensions per sample.
Without making any assumption about this joint distribution, the problem is intractable.
One of the most common assumptions, which we also adopt in this work, is that the joint distribution is tree-structured.
In this case, the distribution can be estimated by the celebrated [13] (and optimal [8]) Chow-Liu algorithm.
While the ChowLiu algorithm requires ⌦ ( d 2 ) pairs observations to estimate the Shannon entropy, our sequential algorithm requires only O ( d ) pairs observations (see Section 5.2 for more details).
The joint Shannon entropy H ( X 1 , . . .
, X d ) of a set of random variables X 1 , . . .
X d is the Shannon entropy H ( X ) of the random variable X =
( X 1 , . . .
, X d ) .
We write the abbreviated term joint entropy
The mutual information between two random variables X and Y and their conditional mutual information given another random variable Z , which are deﬁned as
[Equation]
Estimation Algorithms and Evaluation Criteria
A set of n users and a central server cooperate according to the following protocol to estimate the entropy of a random variable X :
Each user i 2 [ n ] draws an independent sample x i according to the distribution of X . 2.
For r rounds: (a) The server sends information to a subset of the users.
(b) Those users send (partial) information about their sample back to the server. 3.
The server outputs an estimate of the Shannon entropy (Algorithm 1,2 and 4) or the Gini or collision (Algorithm 5) entropies of X .
An estimation algorithm speciﬁes the steps that each user and the server perform to implement the above protocol.
The algorithm is non-interactive if the protocol consists of a single round in which all users participate.
In a non-interactive algorithm the server cannot adapt its queries to users based on responses from other users, since the server communicates with all the users concurrently.
An algorithm is sequentially interactive if each round consists of communication with a single user, who is never contacted again.
Sequential interactivity enables the server to query users adaptively [23].
We evaluate estimation algorithms according to the following criteria:
• Sample complexity : The number of users from whom the server requests data.
• Space complexity : The space used by the server when executing the algorithm.
• Communication complexity : The maximum number of bits transmitted by any single user to the server.
Note that the amount of information sent by the server to the users is not counted when determining communication complexity.
Privacy : Let x i be the sample belonging to user
i and o i be the data observed by the server from user i .
We say that an algorithm satisﬁes ↵ -local differential privacy if
[Equation]
0 ] for any user i , measurable set O , and possible sample values x, x 0 .
• Error :
The absolute difference between the true entropy of the distribution and the estimate output by the server.
Estimating the Shannon Entropy of Tree-structured Distributions
In this section we assume that X
=
( X 1 , . . .
, X d ) is a vector of d discrete variables, and that the support size of each variable X i is a constant ( e.g. , each variable is Boolean).
We also assume that X has a tree-structured distribution, which means that there exists a rooted tree T with d nodes such that for any node i
[ d ] we have Pr[ X i |
X  i ] =
Pr[ X i | X pa T ( i ) ] , where X  i =
( X 1 , . . .
X i  1 , X i +1 , . . .
, X d ) and the node pa T ( i ) is the parent of node i in tree T .
If i is the root node, then we deﬁne Pr[ X i | X pa T ( i ) ]
=
Pr[ X i ] .
Equivalently, a tree-structured distribution is a
Markov random ﬁeld with a tree as the underlying undirected graph.
Essentially, the tree-structured assumption implies that the only correlations among the X i ’s are pairwise correlations.
If T is a chain or a star we say that X is chain-structured and star-structured , respectively.
We will treat these two special cases at the end of this section (Algorithms 2 and 4, respectively).
5.1 Breaking-up the problem: Entropy Estimation When the Support Size is Small
Before proceeding to describe algorithms for estimating the Shannon entropy of tree-structured distributions, we use existing results for private distribution estimation to devise a local differentially private estimator for the Shannon entropy that is sample and communication efﬁcient when the support size of the distribution is small (as is the case for the marginals).
The server will repeatedly invoke this algorithm as a subroutine in the sections below.
First we recall that the Shannon entropy can be upper bounded according to Theorem 17.3.3 of [14] as
[Equation]
where X p and X p 0 are two discrete random variables with support size c and distributions p and p 0 .
Next, we apply a local differentially private learning algorithm for discrete distribution due to [4] that learns the parameters of a discrete distribution with small L 1 error.
The following theorem combines these two results by using the fact
that x/ log(1 /x )
 1 whenever 0 < x  1 / 2 .
Theorem 5.1.
For any discrete distribution X with support size c and for any 1 / 2  ✏ > 0 , there exists an estimator satisfying ↵ -local differential privacy that estimates H ( X ) within ✏ error using n =
O ( c 2 log 1  / ( ✏ 2 ↵ 2 )) samples with probability 1   when ↵ 2 (0 , 1) .
This algorithm resulting from Theorem 5.1 can be used to privately estimate the entropy H ( X i ) , mutual information
I ( X i ; X j ) , and conditional mutual information I ( X i ;
X j | X k ) of any variables
X i , X j and X k by using O ⇣
[Equation]
⌘ samples per estimate and O (1) bits per sample, since each of these variables has constant support size, and both mutual information and conditional mutual information can be expressed in terms of entropies ((4) and (5) in Section 3).
We call such an estimate ( ↵ , ✏ ,  ) -good .
5.2 Algorithm for General Trees
Note that the support size of X can be exponential in d .
In the worst case, estimating the entropy of a distribution with support size k within constant error requires ˜ ⇥ ( k ) samples [19].
However the tree-structure of X can be exploited to signiﬁcantly reduce the sample complexity.
In their seminal paper, Chow and Liu [13] proved the identity
[Equation]
for any tree-structured random variable X , where the maximization is taken over all possible trees connecting the d variables.
Eq.
(6) suggests a communication-efﬁcient algorithm for estimating the entropy of X , which is known as the Chow-Liu
algorithm :
First, estimate each marginal entropy H ( X i ) and each mutual information I ( X i ; X j ) .
Next, compute a maximum spanning tree on the d variables, where the weight of each edge ( X i , X j ) is the estimate of the mutual information
I ( X i ; X j ) .
Finally, plug all of the H ( X i ) estimates, the maximum spanning tree, and the corresponding I ( X i ; X j ) estimates, into Eq.
The Chow-Liu algorithm requires ⌦ ( d 2 ) samples, since it computes the mutual information between every pair of variables in order to compute a maximum spanning tree.
However, estimating the right-hand side of Eq.
(6) only requires estimating the weight of the maximum spanning tree, which is signiﬁcantly easier than ﬁnding the tree itself.
Algorithm 1 adapts a technique from [12] that estimates the weight of the maximum spanning tree of a graph in time that is sublinear in the number of edges in the graph.
The basic idea is to select nodes of the graph at random and use breadth-ﬁrst search to determine the size of each of their connected components if we were to drop edges that do not meet a weight threshold, short-circuiting the search when the size becomes too large.
These quantities are combined to estimate the weight of the maximum spanning tree.
In our case, an edge weight is a mutual information between a pair of variables, which we estimate from pair observations.
Theorem 5.2. Algorithm 1 is ↵ -locally differentially private and has O (1) communication complexity. The number of samples requested by the server is O [Equation]
⌘ .
Let ˆ H be the output of the algorithm. If X is tree-structured then | ˆ H  H ( X ) |  ✏ d with probability 1   . 1: Let M = ⌃ 2 ✏ ⌥ and R = ⌃ 1 ✏⌥ 2: for m = 1 , . . . , M do 3: for r = 1 , . . .
, R do 4:
Choose positive integer Z randomly according to Pr[ Z  z ] = 1 /z .
Choose i
⇤ uniformly at random from [ d ] .
6: if Z  2 ✏ then  mr 0 .
7: else .
Breadth-ﬁrst search 8:
Initialize queue to contain i ⇤ and a set V
= {
i ⇤ } .
9: while queue length is non-zero and shorter than Z do 10:
Remove i from front of queue and V =
V [ { i } .
11: for j = [ d ] \ V do 12:
Server computes ( ↵ , ✏ ,  ) -good estimate
ˆ I ij of I ( X i ; X j ) .
13: if ˆ I ij  ✏ m then add j to back of queue.
14: if queue length is zero then  mr 0 else  mr 1 .
15: ˆ c m dR
P
Rr =1  mr .
16: ˆ W ✏ Md  ✏
P
Mm =1 ˆ c m 17: Server computes ( ↵ , ✏ ,  ) -good estimate of each entropy in the ﬁrst sum in Eq.
18: Let ˆ
S
be the sum of the entropy estimates.
19: Return ˆ H =
ˆ S  ˆ W .
5.3 Algorithm for a Chain
Verma and Pearl [38] observed that if X is chain-structured with chain T then for any triplet ( X i ,
X j , X k ) , if X k is on the unique path in T between X i and X j , then I ( X i ; X j | X k )
= 0 .
This observation alone does not help to recover the chain, since the conditional mutual information
I ( X i ; X j | X k ) can also be zero for X i , X j and X k when X k is not on the path between X i and X j in the chain T .
Nevertheless, under the mild assumption that the mutual information I ( X i , X j ) between every pair of variables is distinct, we can recover the chain T by estimating the conditional mutual information of triplets of variables.
The algorithm is similar to a sorting algorithm, such as “merge sort” which requires O ( d log 2 d ) pairwise comparisons over d items.
While we cannot compare pairs explicitly like in a sorting problem, for any triplets ( X i , X j , X k ) , we can decide locally which “item” is between the other two: i.e. , X i $ X j $ X k , X i $ X k $ X j or X k $ X i $ X j in the chain T , by estimating conditional mutual information.
As the observation fro, Verma and Pearl implies that if X is chainstructured then one of I ( X i ; X j | X k ) or I ( X i ; X k | X j ) or I ( X j ; X k | X i ) has to be zero (due to the observation from Verma and Pearl).
This suggests our Algorithm 2, which inserts the variables in a chain one by one in a sequential manner.
Algorithm 3 is called by Algorithm 2 as a subroutine which seeks to ﬁnd the position where to insert.
Theorem 5.3.
Algorithm 2 is ↵ -locally differentially private and has O (1) communication complexity.
The number of samples requested by the server is O
[Equation]
⌘ .
Let ˆ H be the output of the algorithm.
If X is chain-structured and | I ( X i ; X j )  I ( X j ; X k ) |  ✏ then | ˆ H  H ( X ) |  ✏ d with probability  .
5.4 Algorithm for a Star
If X is star-structured then recovering the star T is a matter of identifying the center of the star, which can be done by computing the mutual information between only ˜ O ( d ) pairs of variables.
The algorithm picks a random marginal X i and takes a “Prim’s step”, i.e., choosing the neighboring node (say X k ) that has the largest mutual information with X i .
The edge between X i and X k is in the maximal spanning tree, when we assume that each edges weight is different.
Next, the algorithm estimates
P j 6 = i I
( X i , X j ) and
P j 6 = k I ( X k , X j ) to decide whether X i or X k is the center node of the star. Algorithm 4 presents the procedure, and Theorem 5.4 gives its sample complexity.
Algorithm
[Equation]
=
[Equation]
and returns their sum ˆ H .
Theorem 5.4.
Algorithm 4 is ↵ -locally differentially private and has O (1) communication complexity.
The number of samples requested by the server is O ⇣
[Equation]
⌘ .
Let ˆ H be the output of the algorithm.
If X is star-structured and | I ( X i ; X j )  I ( X j ; X k ) |  ✏ then | ˆ H  H ( X ) |  ✏ d with probability  .
5.5 Lower Bounds
We prove sample complexity lower bounds for estimating the Shannon entropy of a tree-structured distribution from pair observations.
Our ﬁrst lower bound focuses on the non-interactive case, when the algorithm must select all the pairs in advance.
The second claim is more general, and holds for all sequentially interactive algorithms.
Theorem 5.5.
For any non-interactive algorithm that uses o ( d 2 ) pair observations to estimate
Shannon entropy, there exists a tree-structured distribution over { 0 , 1 } d such that the error of the algorithm is ⌦ ( d ) with constant probability.
Theorem 5.6.
For any ✏ > 0 and for any sequentially interactive algorithm that uses o ( d/ ✏ ) pair observations to estimate the Shannon entropy, there exists a tree-structured distribution on { 0 , 1 } d such that the error of the algorithm is ⌦ ( ✏ · d ) with constant probability.
The lower bound given in Theorem 5.5 is based on Turán’s theorem [9], which we use to show that for any algorithm with sub-quadratic sample complexity and for any constant C 2 (0 , 1) , there is a graph with d nodes containing C · d -clique – when d is large enough – such that the algorithm does not observe any edge of that clique.
This implies that the additive error of the algorithm is linear in d .
The lower bound for sequentially interactive algorithms in Theorem 5.6 is based on a information theoretical approach.
Interestingly, our construction of problem instances for which we applied Le Cam’s theorem is fairly simple, since it contains d independent random variables in every case.
Nevertheless, this lower bound shows that Algorithm 1 is optimal in d .
Moreover, note that
Theorem 5.6 also holds for non-interactive algorithms.
5.6 Comparison to Prior Work
To the best of our knowledge, the Chow-Liu algorithm is the only published method for estimating the entropy of a distribution that takes advantage of its tree structure.
Since the algorithm is noninteractive, the lower bound in Theorem 5.5 shows that our algorithms have provably better sample complexity when the number of variables
d is large (note that the dependence on d in each of
Theorems 5.2, 5.3 and 5.4 is sub-quadratic).
The Chow-Liu algorithm can also be used to estimate the distribution itself, not just its entropy, and it has recently been shown [8, 16] that the algorithm has optimal sample complexity when given full observations ( i.e. , samples of the entire vector ( X 1 , . . .
, X d ) and not just pairs or triplets of the variables).
So the Chow-Liu algorithm is optimal for estimating a tree-structured distribution, but suboptimal for estimating the entropy of a tree-structured distribution.
The root cause of this difference appears to be the fact that it is signiﬁcantly easier to estimate the weight of the maximum spanning tree than ﬁnding the tree itself.
Estimating Gini and Collision Entropy
Algorithm 5 below estimates both the Gini entropy and collision entropy of a random variable X using samples from the distribution of X while observing only b bits per sample.
Here, we do not require extra restrictions on the discrete distribution of X .
In the algorithm, the server ﬁrst partitions all users into pairs (we assume for simplicity that the number of users is even).
The server then distributes a b -bit hash function to each user, along with a distinct cryptographic salt to each user pair.
Each user then hashes their sample along with their salt, and returns the hash value to the server.
The server computes entropy estimates based on the number of observed hash collisions across all pairs.
In Algorithm 5 we let h x
y
i denotes a binary string that encodes x , followed by a delimiter, and by y .
1: Each user i 2 [ n ] draws sample x i independently from the distribution of X . 2: Server partitions the n users into n 2 disjoint pairs.
3: Let q i⇥ n⇤ be the index of the pair containing user i .
4: Server transmits q i and hash function h : { 0 , 1 } ⇤ 7! { 0 , 1 } b to each user i .
Each user i generates a b -bit hash value h
i
=
h ( h q i , x i i ) for their sample x i . 6: Each user i lets ˆ h
i
= h
i with probability  =
e ↵ 2 b + e ↵ and otherwise draws ˆ h
i uniformly from [2 b ] .
7: Server receives ˆ h i from each user i .
8: If pair q contains users i and j then let c q = 1 [ˆ h
i
=
ˆ h j ] indicate whether a hash collision was observed for pair q . 9: Server computes ¯ c = ⇣ 2 (1   ) n P q c q ⌘   2 b . 10: Server outputs ˆ G = 2 b 2 b  1 ¯ c  1 2 b  1 and ˆ C =
log 1  ˆ G
The analysis of Algorithm 5 is based on the observation that if X and X 0 are independent and identically distributed then the Gini entropy is equal to 1  Pr[ X = X 0 ] and the collision entropy is equal to  log Pr[ X = X 0 ] .
If the server observed each sample directly then it could estimate
Pr[ X = X 0 ] using the collision frequency, i.e. , the fraction of sample pairs ( x i , x j ) such that x i = x j .
However, the server only observes a b -bit hash of each sample.
Among sample pairs in which there is a true collision, all of them also produce a hash collision.
Among samples pairs in which there is not a true collision, about a 12 b fraction of them produce a hash collision.
Therefore the true collision frequency can be estimated using an appropriately bias-corrected hash collision frequency, and the server uses this estimate to approximate the Gini entropy and collision entropy.
The analysis of Algorithm 5 is given in Theorem 6.1 below.
As is customary, for the analysis we assume that the hash function h is constructed by assigning each element of its domain to a uniform random element of its range [7].
See the Appendix for the proof of the theorem.
Theorem 6.1.
Algorithm 5 is ↵ -locally differentially private and has ˜ O ( b ) communication complexity and ˜ O ( b ) space complexity.
Let ˆ G and ˆ C be the outputs of the algorithm.
Let ↵ , ✏ ,  2 (0 , 1) .
If n = ⌦
[Equation]
⌘ then | ˆ G  G ( X ) |  ✏ with probability at least 1   .
Also, if X has support size k and n =
⌦
[Equation]
⌘ then | ˆ C  C ( X ) |  ✏ with probability at least 1   .
6.1 Comparison to Prior Work
Recall that Gini entropy is one minus the second frequency moment (up to a constant).
Local differentially private algorithms for estimating frequency moments were recently studied in [11].
Letting b = 1 in Algorithm 5 yields a sample complexity of ˜ O
(1 / ↵ 2 ✏ 2 ) , which matches the sample complexity of the sequentially interactive algorithm for estimating the second frequency moment from [11].
However our algorithm is non-interactive, which is a much weaker communication model.
Also it only uses 1 bit per sample and ˜ O (1) space, while the previous algorithm uses ⌦ ( k ) bits per sample and ⌦ ( k ) space, where k is the support size of the distribution.
The authors in [11] asked whether there is a non-interactive algorithm for privately estimating frequency moments with a sample complexity that is independent of the distribution’s support size.
Here we afﬁrmatively answer this open question for the second frequency moment.
The best known algorithm for estimating collision entropy using ˜ O (1) space is due to [35].
The sample complexity of their algorithm is ˜ O  k/ ✏ and its communication complexity is O (log k ) communication per user, where k is the support size of the distribution.
Letting b = log k and ↵
=
O (1) in Algorithm 5 recovers these results (up to logarithmic factors), and using smaller values for b or ↵ generalizes the previous algorithm to the private and communication-efﬁcient setting.
Also, it was shown in [15] that (conditioned on a plausible conjecture) any algorithm that estimates collision entropy to within O (1) error using O (1) space requires ⌦ ( k ) samples.
Therefore our algorithm is likely to be Pareto optimal with respect to the sample complexity-space complexity trade-off.
Experiments
In this section we present two sets of experiments to support our theoretical ﬁndings.
First, we justify that Algorithm 1 is indeed amenable to estimate the Shannon entropy of tree-based distributions with linear sample complexity in d .
Thus it has a superior sample complexity comparing to the state-of-the-art non-interactive method [13, 8] which has a quadratic sample complexity in d .
The sample complexity is deﬁned here in terms of the number of observation from pairs of marginals.
In the second set of experiments, we test Algorithm 5 to estimate the collision entropy of discrete distributions, and we compare its performance to the best known communication efﬁcient, non-private algorithm, to our best knowledge.
We refer to this algorithm as Skorski’s algorithm [35].
Estimating Shannon entropy: To estimate the Shannon entropy as it is given in (6), the marginal entropy values and the mutual information between certain or all pairs of marginals have to be estimated.
More concretely, Chow-Liu algorithm estimates the mutual information between all pairs of marginals which results in quadratic sample complexity, whereas Algorithm 1 estimates the mutual information only for linear fraction of pairs.
What is common in these two algorithms is that they both estimates the marginal entropy values by sampling the each marginals independently from the mutual information estimations.
In addition to this, both algorithm estimates the mutual information between pairs of marginals with the same ✏ additive error and with the same privacy budget ↵ .
Therefore it is fair to compare the performance of these two algorithms in terms of number pairs for which the mutual information is estimated by them.
For Chow-Liu algorithm this is d 2 whereas Algorithm 1 is a randomized algorithm, thus we evaluate it over 100 repetitions and report the average.
In this experiments, we used tree-structured multivariate distributions over { 0 , 1 } d .
The tree structure is picked randomly by taking the maximum spanning tree in a full graph with edge weights that obey standard normal distribution.
Then we chose parameters for each marginals uniformly at random from
The tree structure is achieved by inducing dependence between marginal distribution while preserving the marginals.
More concretely, for two marginals
X i and X j with parameters p i and p j , we set P (( X i , X j ) =
= (1  p i ) · p j + d , P (( X i , X j ) =
= (1  p i ) · (1  p j )  d , P (( X i , X j ) =
= p i · (1  p j )  d and P (( X i , X j ) =
=
p i · p j + d where selected d uniformly at random so as each probability stays positive. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] The results are shown in Figure 1a. One can clearly seen that the sample size, i.e. number of mutual information estimate, is close to linear however Chow-Liu algorithm requires a higher sample size.
Note that sample size means only the pairs of marginals for which the algorithms requires to estimate the mutual information.
Estimating collision entropy: In this set of experiment, we focus on estimating collision entropy.
We drew samples from a discrete exponential distribution p i /
e  i with support size k = 1000 .
We used the previously best-known algorithm [35], which requires O (log k ) bits per sample and is not private, to estimate the collision entropy of the distribution.
We also used our algorithm, which requires only 1 bit per sample and can be made differentially private.
The results is viewed in Figure 1b.
Our experiment shows that the previous algorithm has 5% estimation error after observingbits, while our algorithm has less than 3.5% estimation error.
Thus our algorithm has lower error for the same communication cost, and it is also (local) differentially private.
Conclusion and future work
Estimating entropy is of importance in many practical applications.
In this paper, we studied three widely used entropy measures: Shannon, Gini and collision entropy.
We described estimation algorithms for each entropy that require minimal communication and satisfy local differential privacy.
Our sequentially interactive algorithm for estimating the Shannon entropy of high-dimensional tree-structured distributions observes only two of these dimensions per sample, and has a sample complexity O ( d/ ✏ 3 ) .
Our approach relies on the celebrated Chow-Liu approximation [13].
It provides an improvement of the sample complexity of the original non-interactive Chow-Liu algorithm whose sample complexity is ⌦ ( d 2 ) .
We also identiﬁed some special cases when the underlying graphical model of the joint distribution is either a chain or star graph.
In these cases, the proposed algorithms have a sample complexity of ˜ O
( d log d/ ✏ 2 ) .
Our algorithm for estimating the Gini and collision entropy estimation also improved on the state-of-the-art, either by matching the sample complexity of previous work but in a weaker communication model and with signiﬁcantly better communication complexity, or by generalizing the best known algorithm to the private and communication-efﬁcient setting.
Lastly, we demonstrated the versatility of our methods on synthetic data, and showed that the proposed methods for Shannon entropy and collision entropy are superior when comparing them to the performance to the state-of-the-art methods.
A natural extension of our work on Shannon entropy estimation is to consider higher-order correlations in the Chow-Liu decomposition [25].
In this case, discovering the underlying structure of the joint distribution is already computationally challenging, unlike in the second order case when it reduces to a maximum spanning tree problem.
However, efﬁciently estimating the entropy of the resulting distribution might still be possible.