F ED - χ 2 : S ECURE F EDERATED C ORRELATION T EST
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose the ﬁrst secure federated χ 2 -test protocol, F ED - χ 2 .
We recast χ 2 -test as a second frequency moment estimation problem and use stable projection to encode the local information in a short vector.
As such encodings can be aggregated with summation, secure aggregation can be applied to conceal the individual updates.
We formally establish the security guarantee of F ED - χby demonstrating that the joint distribution is hidden in a subspace containing exponentially possible distributions.
Our evaluation results show that F ED - χachieves good accuracy with small client-side computation overhead.
F ED - χperforms comparably to the centralized χ 2 -test in several real-world case studies.
The code for evaluation is in the supplementary material.
I NTRODUCTION
Correlation test, as the name implies, is the process of examining the correlation between two random variables using observational data.
It is a fundamental building block in a wide variety of real-world applications, including feature selection (Zheng et al., 2004), cryptanalysis (Nyberg, 2001), causal graph discovery (Spirtes et al., 2000), empirical ﬁnance (Ledoit & Wolf, 2008; Kim & Ji, 2015), medical studies (Kassirer, 1983) and genomics (Wilson et al., 1999; Dudoit et al., 2003).
Because the observational data used in correlation tests may contain sensitive information such as genomic information, centralizing the data collection is risky.
To address, we resort to a federated setting in which each client maintains its own data and communicates with a centralized server to calculate a function.
Note that the communication should contain as little information as feasible.
Otherwise, the server may be able to infer sensitive information from the communication transcript.
In the present work, we study a representative correlation test, namely χ 2 -test, under the federated setting.
There are two straightforward methods for conducting χ 2 -test in such a context.
First, clients can upload their raw data to the centralized server and delegate the test to it.
While this method is effective in terms of communication, it entirely exposes the clients’ private information.
Second, clients may run secure multiparty computation (MPC) under the server’s coordination.
Thus, clients can jointly run χ 2 -test without disclosing their data to the server.
However, general-purpose MPC imposes signiﬁcant computation and communication overhead, which is typically intolerable in a federated setting with computationally limited clients, e.g. , mobile devices.
To address the dilemma, we present a federated protocol optimized for χ 2 -test that is computationally and communicationally efﬁcient and discloses limited information to the server .
We begin by recasting χ 2 -test as a second frequency moment estimation problem.
To approximate the second frequency moment in a federated setting, each client encodes its raw data into a low-dimensional vector via stable random projection (Indyk, 2006; Vempala, 2005; Li, 2008).
Such encodings can be aggregated with only summation, allowing clients to leverage secure aggregation (Bonawitz et al., 2017; Bell et al., 2020) to aggregate the encodings and the server to decode them to approximate the second frequency moment.
Because secure aggregation conceals each client’s individual update within the aggregated global update, the server learns only limited information about the clients’ data.
Our evaluation on four synthetic datasets and 16 real-world datasets shows that F ED - χ 2 can replace centralized χ 2 -test with good accuracy and low computation overhead.
Additionally, we analyze
F ED - χ 2 in three real-world use cases: feature selection, cryptanalysis, and online false discovery rate control.
The results show that F ED - χ 2 can achieve comparable performance with centralized χ 2 -test and can withstand up to 20% of clients dropping out with minor inﬂuence on the accuracy.
In summary, we make the following contributions:
We propose F ED - χ 2 , the ﬁrst secure federated χ 2 -test protocol.
F ED - χ 2 is computation- and communication-efﬁcient and leaks much less information than trivially deploying secure aggregation.
F ED - χ 2 decomposes χ 2 -test into frequency moments estimation that can easily be encoded/decoded using stable projection and secure aggregation techniques.
We give formal security proof and utility analysis of F ED - χ 2 .
We evaluate F ED - χ 2 in real-world use cases, and the ﬁndings suggest that F ED - χ 2 can substitute centralized χ 2 -test with comparable accuracy, and F ED - χ 2 can tolerate up to 20% of clients dropout with minor accuracy drop.
R ELATED W ORK
Bonawitz et al.
(2017) proposed the well-celebrated secure aggregation protocol as a low-cost way to calculate linear functions in a federated setting.
It has seen many variants and improvements since then.
For instance, Truex et al.
(2019) and Xu et al.
(2019) employed advanced crypto tools for secure aggregation, such as threshold homomorphic encryption and functional encryption.
So et al.
(2021) proposed T URBO AGG, which combines secure sharing with erasure codes for better dropout tolerance.
To improve communication efﬁciency, Bell et al.
(2020) and Choi et al.
(2020) replaced the complete graph in secure aggregation with either a sparse random graph or a low-degree graph.
Secure aggregation is deployed in a variety of applications.
Agarwal et al.
added binomial noise to local gradients, resulting in both differential privacy and communication efﬁciency.
Wang et al.
(2020) replaced the binomial noise with discrete Gaussian noise, which is shown to exhibit better composability.
Kairouz et al.
(2021) proved that the sum of discrete Gaussian is close to discrete
Gaussian, thus discarding the common random seed assumption from Wang et al.
The above three works all incorporate secure aggregation in their protocols to lower the noise scale required for differential privacy.
Chen et al.
(2020) added an extra public parameter to each client to force them to train in the same way, allowing for the detection of malicious clients during aggregation.
Nevertheless, designing secure federated correlation tests, despite its importance in real-world scenarios, is not explored by existing research in this ﬁeld.
On the other end of the spectrum, Wang et al.
(2021) proved that stable projection is differentially private if the projection matrix is secret.
In our protocol, the projection matrix is public information; hence F ED - χ 2 does not consider the differential privacy guarantee.
F EDERATED C ORRELATION T EST WITH M INIMAL L EAKAGE
In this section, we elaborate on the design of F ED - χ 2 , a secure federated protocol for χ 2 -test.
Sec.
ﬁrst formalizes the problem, establishes the notation system, and introduces the threat model.
In
Sec. 3.2, we recast χ 2 -test as a second frequency moment estimation problem in the federated setting, and consequently, we are able to leverage stable projection to encode each client’s local information (Sec. 3.3), and then aggregate them using secure aggregation (Sec. 3.4).
Sec. 3.5, 3.6, and 3.7 present security proof, utility analysis, communication analysis, and computation analysis of F ED - χ 2 .
3. P ROBLEM S ETUP
We now formulate the problem of the federated correlation test and establish the notation system.
We use [ n ] to denote { 1 , · · · , n } .
We denote vectors with bold lower-case letters ( e.g. , a , b , c ) and matrices with bold upper-case letters ( e.g. , A , B , C ).
We consider a population of n clients C
= { c i } i ∈ [ n ] .
Each client has one share of local data composed of the triplets D i
= { ( x, y, v ( i ) xy ) } , x ∈ X ,
y ∈ Y , v ( i ) xy ∈ {− M, · · · , M } , where x and y are categories of the contingency table, v ( i ) xy is the observed counting of the categories x and y in the local contingency table of the i th client, |X| =
m x and |Y| = m y are ﬁnite domains, and
M is the maximum value | v ( i ) xy | can be.
The global dataset is deﬁned as D =
{ ( x, y, v xy ) : v xy = (cid:80) i ∈ [ n ] v ( i ) xy } .
We focus on federated χ 2 -test
and the data in contingency table is discrete.
For the ease of presentation, we deﬁne the marginal statistics v x = (cid:80) y ∈
[ |Y| ] v xy , v y =
(cid:80) x ∈ [ |X| ] v xy , and v = (cid:80) x ∈ [ |X| ] ,
y ∈
[ |Y| ] v xy .
Besides, we deﬁne ¯ v xy = v x × v y v , denoting the expectation of v xy if x and y are uncorrelated.
We deﬁne m = m x m y and use an indexing function I : [ m x ] × [ m y ]
→ [ m ] to obtain a uniform indexing given the indexing of each variable.
A centralized server S calculates the statistics for χ 2 -test s χ
( D ) =
(cid:80) x ∈ [ |X| ] ,
[Equation]
xy on the global dataset to decide whether
X and Y are correlated without collecting the raw data from clients.
Overall, using MPC to conduct secure correlation tests in a federated scenario is highly expensive and impractical (Boyle et al., 2015; Damgård et al., 2012).
Hence, in the present work, we trade off accuracy for efﬁciency, as long as the estimation error is small with a high probability.
Formally, if
F ED - χ 2 outputs
ˆ s χ
, whose corresponding standard centralized χ 2 -test output is s χ 2 , the following accuracy requirement should be satisﬁed with small (cid:15) and δ .
P
[Equation]
Threat Model.
We assume that the centralized server S is honest but curious.
It honestly follows the protocol due to regulatory or reputational pressure but is curious to discover extra private information from clients’ legitimate updates for proﬁt or surveillance purposes.
As a result, client updates should contain as little sensitive information as feasible.
We want to emphasize that, while the server may explore the privacy of clients, the server will honestly follow the protocol due to regulation or reputational pressure.
The server won’t provide adversarial vectors to the clients.
On the other hand, we assume honest clients.
Speciﬁcally, we do not consider client-side adversarial attacks ( e.g. , data poisoning attacks (Bagdasaryan et al., 2020;
Bhagoji et al., 2019)).
However, we allow a small portion of clients to drop out during the execution.
Also, as we mentioned in Sec. 2, we do not consider the differential privacy guarantee; see further clariﬁcation in Appendix A.
3. F ROM C ORRELATION T EST TO F REQUENCY M OMENTS E STIMATION
We ﬁrst recast correlation test to a second frequency moments estimation problem as deﬁned below.
Given a set of key-value pairs S =
{ k i , v i } i ∈ [ n ] , we re-organize it into a histogram H =
{ k j , v j = (cid:80) k
i =
k j ,i ∈ [ n ] v i } , and estimate the α th frequency moments as F α = (cid:80) j v αj .
χ 2 -test can thus be recast to a 2 nd frequency moments estimation problem as follows:
[Equation]
In federated setting, each client c i holds a local dataset D i
= { ( x, y, v ( i ) xy ) }
and computes a vector u i , where u i [ I ( x, y )]
= v ( i )
xy − ¯ v xy /n √ ¯ v xy and u i has m elements.
Thus, the challenge in federated χ 2 -test becomes calculating the following equation:
[Equation]
3. E NCODING BY S TABLE P ROJECTION & D ECODING BY G EOMETRIC M EAN E STIMATOR
To address the aforementioned challenges and to easily integrate the algorithm into secure aggregation protocols, we use stable random projection (Indyk, 2006; Vempala, 2005) and geometric mean estimator (Li, 2008) to approximate the data’s second frequency moment efﬁciently.
We begin by discussing stable distributions, followed by the encoding and decoding techniques.
Deﬁnition 1 ( α -stable distribution) .
A random variable X follows an α -stable distribution Q α,β,F if its characteristic function is as follows.
[Equation]
, where F is the scale to the α th power and β is the skewness.
α -stable distribution is named due to its property called α -stability.
Brieﬂy, the sum of independent α -stable variables still follows an α -stable distribution with a different scale.
Deﬁnition 2 ( α -stability) .
If random variables X ∼ Q α,β, 1 , Y ∼ Q α,β, 1 and X
and Y are indepen- dent, then C
[Equation]
We borrow the genius idea from Indyk’s well-celebrated paper (Indyk, 2006) to encode the frequency moments in the scale parameter of a stable distribution deﬁned in Deﬁnition 1.
To encode the local dataset D i
= { ( x, y, v ( i ) xy ) } , each client c i , i ∈ [ n ] , is given by the server an (cid:96)
× m projection matrix P whose values are drawn independently from an α -stable distribution Q α,β, 1 , where (cid:96) is the encoding
Note that the encoding and decoding themselves do not provide any security guarantee.
1 Function E NCODE ( P , u i ) : return P × u i 3 Function G EOMETRIC M EAN E STIMATOR ( e ) :
ˆ d (2) ,gm ← (cid:81) (cid:96)k =1 | e k | 2 /(cid:96)
(π Γ( 2 (cid:96) )
Γ(1 − 1 (cid:96) )sin( π(cid:96) ))
(cid:96) //
(cid:96) is the encoding size.
return ˆ d (2) ,gm 6 Function D ECODE ( e ) : size and m = m x m y .
The client re-organizes the data into a vector u i , where u i [ I ( x, y )]
= v ( i ) xy .
Then, the client projects u i to e i
=
P
×
u
i as the encoding (line 2 in Alg. 1).
To decode, the server ﬁrst sums the encodings up e =
(cid:80) i ∈ [ n ] e i and estimates the scale of the variables in the aggregated encoding with an unbiased geometric mean estimator (Li, 2008) in lineof Alg. 1.
According to the α -stability deﬁned in Deﬁnition 2, every element e k in e , k ∈ [ (cid:96) ] , follows this stable distribution:
e k ∼ Q 2 , 0 , || (cid:80) i ∈ [ n ] u i ||
Thus, the l 2 norm can be estimated by calculating the scale of the distribution Q 2 , 0 , || (cid:80) i ∈
[ n ] u i || 22 with e containing (cid:96) elements.
During the above processes, u i and (cid:80) i ∈ [ n ] u
i are not leaked and the aggregation among clients is calculated with only summation.
Thus, secure aggregation protocols can be naturally applied with quantization, as will be discussed in Sec.
3.4 and 3.5.
Furthermore, (cid:96) =
c(cid:15) 2 log(1 /δ )
sufﬁces to guarantee that the second frequency moment can be approximated with a 1 ± (cid:15) factor and a probability no less than 1 − δ .
We will further analyze the utility of F ED - χ 2 in Sec. 3.6.
3. S ECURE F EDERATED C ORRELATION T EST
The complete protocol for F ED - χ 2 is presented in Alg. 2.
Firstly, the marginal statistics v x , v y and v are calculated with secure aggregation and broadcasted to all clients (lines 1–6 in Alg. 2).
This step can be omitted if the marginal statistics are already known.
Then, on the server side, a projection matrix P is sampled from an α -stable distribution Q (cid:96) × m 2 , 0 , 1 .
The projection matrix is broadcasted to all clients (lines 8–10 in Alg. 2).
For each client c i , the local data is re-organized into u i and projected to e i as encoding (lines 11–14 in Alg. 2).
Then, the encoding results will be quantized and aggregated with secure aggregation (line 15 in Alg. 2).
As we have already known the marginal statistics in the ﬁrst round, the quantization bound can be set accordingly.
Additionally, we can use high precision for quantization, such as 64 bits, since the size of the contingency table is normally moderate rather than enormous.
Thus, the precision of the quantized ﬂoat number is comparable to or even better than that of ﬂoat64, and hence we disregard the effect of quantization on accuracy.
Finally, the server gets the χ 2 -test result using the decoding algorithm described in Alg.
1 (line 17 in Alg. 2).
Dropouts in the ﬁrst round have no effect on the test’s accuracy because they can be recovered inside secure aggregation (Bonawitz et al., 2017; Bell et al., 2020).
On the other hand, dropouts in the second round will affect the accuracy of the test.
Still, because the χ 2 value is typically far from the decision threshold, F ED - χ 2 is intrinsically robust to a small portion of clients dropping out (see
Section 4 for empirical assessment).
3. S ECURITY A NALYSIS
As discussed in Sec. 2, the secure aggregation protocol is well studied, and many well-celebrated secure aggregation protocols have been proposed (Bonawitz et al., 2017; Truex et al., 2019; Xu et al., 2019;
So et al., 2021; Bell et al., 2020; Choi et al., 2020).
In this paper, we choose the state-of-the-art secure aggregation protocol by Bell et al.
(Bell et al., 2020), which replaces the complete graph with a sparse random graph to enhance communication efﬁciency.
We clarify that F ED - χ 2 can incorporate other popular secure aggregation protocols.
We now prove the security enforced by Alg. 2 via a standard simulation proof process (Lindell, 2017) on the basis of Theorem 1.
Theorem 1 (Security) .
Let Π be an instantiation of Alg. 2 with the secure aggregation protocol in
Alg. 4 of Appendix G with cryprographic security parameter λ .
There exists a PPT simulator S IM such that for all clients C , the number of clients n , all the marginal distributions { v x } , { v y } , and all the encodings { e i } , the output of S IM is indistinguishable from the view of the real server Π C in that execution, i.e. , Π C ≈ λ S IM ( (cid:80) e i , n ) .
Intuitively, Theorem 1 illustrates that no more information about the clients except the averaged updates is revealed to the centralized server.
Thus, each client’s update is hidden by the rest clients in secure aggregation.
We now present the formal proof for Theorem 1.
Proof for Theorem 1.
To prove Theorem 1, we need the following lemma.
Lemma 1 (Security of secure aggregation protocol) .
Let S ECURE A GG be the secure aggregation protocol in Alg.
4 of Appendix G instantiated with cryprographic security parameter λ .
There exists a probabilistic polynomial-time (PPT) simulator S IM SA such that for all clients C , the number of clients n , and all inputs X = { e i } i ∈
[ n ] , the output of S IM SA is perfectly indistinguishable from the view of the real server, i.e. , S ECURE A GG C ≈ λ S IM SA ( (cid:80) i ∈ [ n ] e i , n ) .
Lemma 1 is derived from the security analysis of our employed secure aggregation protocol (Theorem 3.6 in Bell et al.
(2020)), which establishes that the secure aggregation protocol securely conceals the individual information in the aggregated result.
With this lemma, we are able to prove the theorem for federated χ 2 -test by presenting a sequence of hybrids that begin with real protocol execution and end with simulated protocol execution.
We demonstrate that every two consecutive hybrids are indistinguishable, illustrating that the hybrids are indistinguishable according to transitivity.
H YB
This is the view of the server in the real protocol execution, R EAL C .
H YB
In this hybrid, we replace the view during the execution of each S ECURE A GG ( { v ( i ) x } i ∈ [ n ] ) in line 3 of Alg.
2 with the output of S IM SA ( v x , n ) one by one.
According to Lemma 1, each replacement does not change the indistinguishability.
Hence, H YB 2 is indistinguishable from H YB 1 .
H YB 3 Similar to
H YB 2 , we replace the view during the execution of each
S ECURE A GG ( { v ( i ) y
} i ∈ [ n ] ) in line 4 of Alg.
2 with the output of S IM SA ( v y , n ) one by one.
According to Lemma 1, H YB 3 is indistinguishable from H YB 2 .
H YB
In this hybrid, we replace the view during the execution of S ECURE A GG ( { e i } i ∈ [ n ] ) in line 15 of Alg.
2 with the S IM SA ( (cid:80) e i , n ) .
This hybrid is the output of S IM .
According to Lemma 1, H YB 4 is indistinguishable from H YB 3 .
Remark:
what does Alg.
2 leak?
By Theorem 1, we show that individual updates of clients are perfectly hidden in the aggregated results and F ED - χ 2 leaks no more than a linear equation system:
[Equation]
Algorithm 2: F ED - χ 2 : secure federated χ 2 -test.
S ECURE
A GG is a remote procedure that receives inputs from the clients and returns the summation to the server.
I NIT S ECURE
A GG is the corresponding setup protocol deciding the communication graph and other hyper-parameters. is
Server
Sample the projection matrix P from Q (cid:96) × m 2 , 0 ,
Broadcast the projection matrices to the clients
Client c i , i ∈ [ n ]
Calculate ¯ v xy
= v x v y v
Prepare u i s.t.
u i
[ I ( x, y )]
= v ( i )
xy − ¯ v xy /n √ ¯ v xy
Calculate e i = E
NCODE ( P , u i ) e =
S ECURE A GG (Q UANTIZE ( { e i } i ∈ [ n ] ))
Server ˆ s
2 = D ECODE ( e )
, where J 1 ,m x and J 1 ,m y are 1 × m x and 1 × m y unit matrices, V is an m x × m y matrix whose elements are { v xy } , and v is a vector ﬂattened by V .
To understand this, information leaked by F EDχ 2 includes the estimation e and marginal statistics v x and v y .
The following theorem establishes an important fact: the above equation system has an exponentially large solution space, which effectively conceals the real joint distribution.
We thus believe that Alg. 2 practically ensures privacy due to the solution space’s vastness.
Theorem 2.
Given a projection matrix P ∈ Z (cid:96) × m q , { v x } , { v y } and e , there are at least q m − (cid:96) − m x − m y feasible choices of { v xy } .
Proof sketch for Theorem 2.
As demonstrated above, the given information forms a linear equation with m x + m y + (cid:96) equations.
Given m > m x + m y + (cid:96) , the rank of the coefﬁcient matrix is no more than m x + m y + (cid:96) .
Solving the equations with Smith normal form, we know that the solution space is at least ( m − m x − m y − (cid:96) )
-dimensional.
With the following lemma, we manage to prove
Theorem 2.
Lemma 2.
There are q r × c vectors in the subspace of Z r × c q .
3. U TILITY A NALYSIS
In this section, we conduct the utility analysis in terms of multiplicative error.
We show that the output of F ED - χ 2 , ˆ s χ 2 , is a fairly accurate approximation (parameterized by (cid:15) ) to the correlation test output s χ 2 in the standard centralized setting with high probability parameterized by δ .
Theorem 3 (Utility) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G. Π is parameterized with (cid:96) =
c(cid:15) 2 log(1 /δ )
for some constant c .
After executing Π C on all clients C , the server yields ˆ s χ 2 , whose distance to the accurate correlation test output s χ 2 is bounded with high probability as follows:
[Equation]
Proof sketch for Theorem 3.
First, we introduce the following lemma from Li (2008).
Lemma 3 (Tail bounds of geometric mean estimator (Li, 2008)) .
The right tail bound of geometric mean estimator is:
[Equation]
where (cid:15) 2 G R = C 1 log(1 + (cid:15) ) − C 1 γ e ( α − 1) − log( 2 π Γ( αC 1 )Γ(1 − C 1 ) sin( παC 1 2 )) , α = 2 in our setting, C 1 =
π tan − 1 ( log(1+ (cid:15) )
(2+ α 2 ) π/ 6 ) , and γ e = 0 .
577215665 ... is the Euler’s constant.
The left tail bound of the geometric mean estimator is:
[Equation]
where (cid:96) > (cid:96) 0 ,
[Equation]
−
[Equation]
0 C [Equation] log
[Equation]
Γ
[Equation] [Equation]
) sin(
[Equation]
and C 2 = 12 π(cid:15) [Equation] With Lemma 3, Taking c ≥ max( G R , G L ) and δ = exp( − (cid:96)(cid:15) 2 c ) , we are able to prove P [ˆ s χ 2 < (1 − (cid:15) )
s χ
2 ∨ ˆ s χ 2 > (1 + (cid:15) ) s χ 2 ] ≤ δ , and the above bound holds when (cid:96) =
c(cid:15) 2 log(1 /δ ) .
3. C OMMUNICATION & C OMPUTATION A NALYSIS
In this section, we present the communication and computation cost of Alg. 2.
Theorem 4 (Communication Cost) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G, then (1) the client-side communication cost is O (log n + m x + m y + (cid:96) ) ; (2) the server-side communication cost is O ( n log n + nm x + nm y + n(cid:96) ) .
Theorem 5 (Computation Cost) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G, then (1) the client-side computation cost is O (log 2 n +( (cid:96) + m x + m y ) log n + m(cid:96) ) ; (2) the server-side computation cost is O ( n log 2 n + n ( (cid:96) + m x + m y ) log n + (cid:96) ) .
Note that compared with the original computation cost presented in (Bell et al., 2020), the client-side overhead has an extra O ( m(cid:96) ) term.
This term is incurred by the encoding overhead.
We also give an empirical evaluation on the client-side computation overhead in Sec. 4.1.
Please refer to Appendix H for the detailed proof of Theorem 4 and Theorem 5.
E VALUATION
Experiment Setup.
To assess F ED - χ 2 ’s accuracy, we simulate it on four synthetic datasets and 16 real-world datasets.
We compare the multiplicative errors of F ED - χ 2 with that of the standard centralized χ 2 -test.
The four synthetic datasets are independent, linearly correlated, quadratically correlated, and logistically correlated.
As the real-world datasets, we report the details in Appendix I.
Additionally, we evaluate F ED - χ 2 ’s utility in three real-world application scenarios: 1) feature selection, 2) cryptanalysis, and 3) online false discovery rate (FDR) control.
For feature selection, we report the model accuracy trained on the selected features.
For cryptanalysis, we report the success rate of cracking ciphertexts.
For Online FDR control, we report the average false discovery rate.
We compare the performance of F ED - χ 2 with that of the centralized χ 2 -test in each of the three experiments.
Unless otherwise speciﬁed, experiments are launched on an Ubuntu 18.04 LTS server equipped with 32 AMD Opteron(TM)
Processor 6212 and 512GB RAM.
4. E VALUATION R ESULTS
[Figure]
lti p
[Figure]
ca ti v e E rr o r (cid:15) /
A
CC
[Figure]
lti
[Figure]
E rr o r (cid:15) /
A
CC (i)
[Figure]
lti
[Figure]
li ca ti v e E rr o r (cid:15) /
A
CC
[Figure]
M u lti p li ca ti v e E
rr o r (cid:15) /
[Figure]
CC
[Figure]
Accuracy.
We begin by evaluating the accuracy of F ED - χ 2 , as illustrated in Fig.
Each point represents the mean of 100 independent runs with 100 clients, while the error bars indicate the standard deviation.
We choose m x = m y = 20 in this experiment.
Note that the accuracy drop is independent of the number of clients.
From Fig.
1, we observe that the larger the encoding size (cid:96) , the smaller the multiplicative error.
When (cid:96) = 50 , the multiplicative error (cid:15)
≈ 0 .
This conforms with Theorem 3, in which the multiplicative error (cid:15) =
(cid:112) c(cid:96)
log(1 /δ ) decreases as (cid:96) increases.
We also evaluate the power (Cohen, 2013) of F ED - χ 2 .
We set the p-value threshold as 0 . 05 , which determines whether or not to reject the null hypothesis.
From The dashed lines in Fig.
1, we can tell that the power of F ED - χ 2 is high.
This conforms with our observation on the multiplicative errors.
Speciﬁcally, since the χ 2 values are typically far from the decision threshold, a multiplicative error of 0 . 2 rarely ﬂips the ﬁnal decision.
We also present the results when 5% of clients drop out in the second round of F ED - χ 2 in Fig.
The results show that F ED - χ 2 is robust to a small portion of dropouts.
In Appendix F, we present the results in terms of 10%, 15%, and 20% dropout rates.
[Figure]
Client-side Computation Overhead.
To assess extra computation overhead incurred by F ED - χ 2 on the client side, we measure the execution time of the encoding scheme on an Android 10 mobile device equipped with a Snapdragon865 CPU and 12GB RAM.
We use
PyDroid (Sandeep Nandal, 2020) to run the client-side computation of F ED - χ 2 on the Android device.
The results are shown in Fig.
Each point represents the average of 100 separate runs, with accompanying error bars.
The overhead is generally negligible.
For example, for a 500 × 500 contingency table, the encoding takes less than 30ms.
The overhead grows linearly in relation to m x ( m y ) and consequently quadratically in Fig.
2, where m x equals m y .
4. D OWNSTREAM U SE C ASE S TUDY
[Figure]
Feature Selection.
Our ﬁrst case study explores secure federated feature selection using F ED - χ 2 .
The setting is that each client holds data with a large feature space and wants to collaborate with other clients to rule out unimportant features and retain features with topk highest
χ 2 scores.
We use Reuters-21578 (Hayes & Weinstein, 1990), a standard text categorization dataset (Yang, 1999; Yang &
Pedersen, 1997; Zhang & Yang, 2003), and pick the top-20 most frequent categories using 17,262 training and 4,316 test documents.
These documents are distributed randomly to 100 clients, each of whom receives the same number of training documents.
After removing all numbers and stop-words, we obtain 167,135 indexing terms.
After performing feature selection using F ED - χ 2 , we select the top 40,000 terms with the highest χ 2 scores.
When compared with the centralized χ 2 -test, 38,012 (95.03%) of the selected terms are identical, indicating that
F ED - χ 2 produces highly consistent results with the standard χ 2 -test.
We then train logistic regression models using the terms selected by F ED - χ 2 and the centralized χ 2 -test, respectively.
All hyper-parameters are the same.
The details of these models are reported in
Appendix J. The results in Fig.
3 further demonstrate that F ED - χ 2 exhibits comparable performance with the centralized χ 2 -test.
When 10% and 20% of clients dropout in the second round of F ED - χ 2 , the accuracy of the trained model using the features selected by F ED - χ 2 does not drop much.
We also examine performance without feature selection, and as expected, model accuracy is signiﬁcantly greater after feature selection.
Note that the model without feature selection has 2,542,700 more parameters than the model with feature selection.
Hence, feature selection effectively improves model accuracy while reducing model size and computational cost.
Cryptanalysis.
In the second case study, we explore federated cryptanalysis with F ED - χ 2 .
We break Caesar cipher (Luciano & Prichett, 1987), a classic substitution cipher, with F ED - χ 2 .
In a
Caesar cipher, each letter in the plaintext is replaced by another letter with some ﬁxed number of positions down the alphabet.
For instance, each English letter can be right-shifted by three, converting the plaintext “good” to the ciphertext “jrrg”.
There are 26 possible shifts when given 26 English letters.
The plaintext can be cracked in a shortcut by performing a correlation test on the ciphertext in relation to normal English text.
[Figure]
In our setting, each client is assumed to possess a segment of the
Caesar ciphertext.
To collaboratively crack the ciphertext, these clients run 26 χ 2 -tests to determine the correlation level between each ciphertext letter and the letters in normal English text.
The χ 2 -test yielding the highest correlation level elucidates how English letters are encrypted into Caesar ciphertexts.
We take Shakespeare’s lines as the plaintext and encrypt it into a
Caesar ciphertext with a length of 1000 characters.
We initiate the cracking process on ten non-overlapping ciphertexts to compute the average success rate (see Fig. 4).
In general, the larger the encoding size, the more precise the χ 2 -statistics is, and consequently the higher the success rate.
Again, according to Theorem 3, the multiplicative error (cid:15) decreases as the encoding size increases.
In Fig.
4, we also report the success rate when 10% and 20% of clients dropout in Round two of F ED - χ 2 , respectively.
Even if 20% of clients dropout, the success rate can still be 100% as long as the encoding size (cid:96) is larger than 20.
[Figure]
Online False Discovery Rate Control.
In the third case study, we explore federated online false discovery rate (FDR) control (Foster & Stine, 2008) with F ED - χ 2 .
In an online FDR control problem, a data analyst receives a stream of hypotheses on the database, or equivalently, a stream of p -values: p 1 , p 2 , · · · .
At each time t , the data analyst should pick a threshold α t to reject the hypothesis when p t < α t .
The error metric is the false discovery rate, and the objective of online FDR control is to ensure that for any time t , the
FDR up to time t is smaller than a pre-determined quantity.
We use the SAFFRON procedure (Ramdas et al., 2018), the state-ofthe-art online FDR control, for multiple hypothesis testing.
The χresults and corresponding p -values are calculated by F ED - χ 2 .
We present the detailed algorithm of SAFFRON and the hyper-parameters used in our evaluation in
Appendix K. The size of the randomly synthesized contingency table is 20 × 20 .
Each time, there are 100 independent hypotheses, with a probability of 0 .
5 that each hypothesis is either independent or correlated.
The time sequence length is 100 , and the number of clients is 10 .
The data are synthesized from a multivariate Gaussian distribution.
For the correlated data, the covariance matrix is randomly sampled from a uniform distribution.
For the independent data, the covariance matrix is diagonal, and its entries are randomly sampled from a uniform distribution.
At time t , we use F ED - χ 2 to calculate the p -values p t of all the hypotheses, and then use the
SAFFRON procedure to estimate the reject threshold α t using p t .
The relationship between the average FDR and encoding size (cid:96) is shown in Fig.
We observe that the variance of independent runs is very small, so we omit the error bars.
The results indicate that by increasing the encoding size (cid:96) , F ED - χ 2 can achieve a low FDR of less than 5 . 0% .
We also observe that dropouts improve the performance of F ED - χ 2 in this case study.
The reason for this is because dropouts reduce the estimated χ 2 value, which increases the probability of accepting the null hypothesis in F ED - χ 2 .
As a larger portion of queries follows the null hypothesis in online FDR control, the accuracy also increases.
The results further demonstrate that F ED - χ 2 can be employed in practice to facilitate online FDR control using a secure federated correlation test.
C ONCLUSION
This paper takes an important step towards designing non-linear secure aggregation protocols in the federated setting.
Speciﬁcally, we propose a universal secure protocol to evaluate frequency moments in the federated setting.
We focus on an important application of the protocol: χ 2 -test.
We give formal security proof and utility analysis on our proposed secure federated learning χ 2 -test protocol
F ED - χ 2 and validate them with empirical evaluations and case studies.
[-i ∈ A (cid:48) 
-]
[-= 
-]