F ED - χ 2 : S ECURE F EDERATED C ORRELATION T EST
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose the ﬁrst secure federated χ 2 -test protocol, F ED - χ 2 .
We recast χ 2 -test as a second frequency moment estimation problem and use stable projection to encode the local information in a short vector.
As such encodings can be aggregated with summation, secure aggregation can be applied to conceal the individual updates.
We formally establish the security guarantee of F ED - χby demonstrating that the joint distribution is hidden in a subspace containing exponentially possible distributions.
Our evaluation results show that F ED - χachieves good accuracy with small client-side computation overhead.
F ED - χperforms comparably to the centralized χ 2 -test in several real-world case studies.
The code for evaluation is in the supplementary material.
I NTRODUCTION
Correlation test, as the name implies, is the process of examining the correlation between two random variables using observational data.
It is a fundamental building block in a wide variety of real-world applications, including feature selection (Zheng et al., 2004), cryptanalysis (Nyberg, 2001), causal graph discovery (Spirtes et al., 2000), empirical ﬁnance (Ledoit & Wolf, 2008; Kim & Ji, 2015), medical studies (Kassirer, 1983) and genomics (Wilson et al., 1999; Dudoit et al., 2003).
Because the observational data used in correlation tests may contain sensitive information such as genomic information, centralizing the data collection is risky.
To address, we resort to a federated setting in which each client maintains its own data and communicates with a centralized server to calculate a function.
Note that the communication should contain as little information as feasible.
Otherwise, the server may be able to infer sensitive information from the communication transcript.
In the present work, we study a representative correlation test, namely χ 2 -test, under the federated setting.
There are two straightforward methods for conducting χ 2 -test in such a context.
First, clients can upload their raw data to the centralized server and delegate the test to it.
While this method is effective in terms of communication, it entirely exposes the clients’ private information.
Second, clients may run secure multiparty computation (MPC) under the server’s coordination.
Thus, clients can jointly run χ 2 -test without disclosing their data to the server.
However, general-purpose MPC imposes signiﬁcant computation and communication overhead, which is typically intolerable in a federated setting with computationally limited clients, e.g. , mobile devices.
To address the dilemma, we present a federated protocol optimized for χ 2 -test that is computationally and communicationally efﬁcient and discloses limited information to the server .
We begin by recasting χ 2 -test as a second frequency moment estimation problem.
To approximate the second frequency moment in a federated setting, each client encodes its raw data into a low-dimensional vector via stable random projection (Indyk, 2006; Vempala, 2005; Li, 2008).
Such encodings can be aggregated with only summation, allowing clients to leverage secure aggregation (Bonawitz et al., 2017; Bell et al., 2020) to aggregate the encodings and the server to decode them to approximate the second frequency moment.
Because secure aggregation conceals each client’s individual update within the aggregated global update, the server learns only limited information about the clients’ data.
Our evaluation on four synthetic datasets and 16 real-world datasets shows that F ED - χ 2 can replace centralized χ 2 -test with good accuracy and low computation overhead.
Additionally, we analyze
F ED - χ 2 in three real-world use cases: feature selection, cryptanalysis, and online false discovery rate control.
The results show that F ED - χ 2 can achieve comparable performance with centralized χ 2 -test and can withstand up to 20% of clients dropping out with minor inﬂuence on the accuracy.
In summary, we make the following contributions:
We propose F ED - χ 2 , the ﬁrst secure federated χ 2 -test protocol.
F ED - χ 2 is computation- and communication-efﬁcient and leaks much less information than trivially deploying secure aggregation.
F ED - χ 2 decomposes χ 2 -test into frequency moments estimation that can easily be encoded/decoded using stable projection and secure aggregation techniques.
We give formal security proof and utility analysis of F ED - χ 2 .
We evaluate F ED - χ 2 in real-world use cases, and the ﬁndings suggest that F ED - χ 2 can substitute centralized χ 2 -test with comparable accuracy, and F ED - χ 2 can tolerate up to 20% of clients dropout with minor accuracy drop.
R ELATED W ORK
Bonawitz et al.
(2017) proposed the well-celebrated secure aggregation protocol as a low-cost way to calculate linear functions in a federated setting.
It has seen many variants and improvements since then.
For instance, Truex et al.
(2019) and Xu et al.
(2019) employed advanced crypto tools for secure aggregation, such as threshold homomorphic encryption and functional encryption.
So et al.
(2021) proposed T URBO AGG, which combines secure sharing with erasure codes for better dropout tolerance.
To improve communication efﬁciency, Bell et al.
(2020) and Choi et al.
(2020) replaced the complete graph in secure aggregation with either a sparse random graph or a low-degree graph.
Secure aggregation is deployed in a variety of applications.
Agarwal et al.
added binomial noise to local gradients, resulting in both differential privacy and communication efﬁciency.
Wang et al.
(2020) replaced the binomial noise with discrete Gaussian noise, which is shown to exhibit better composability.
Kairouz et al.
(2021) proved that the sum of discrete Gaussian is close to discrete
Gaussian, thus discarding the common random seed assumption from Wang et al.
The above three works all incorporate secure aggregation in their protocols to lower the noise scale required for differential privacy.
Chen et al.
(2020) added an extra public parameter to each client to force them to train in the same way, allowing for the detection of malicious clients during aggregation.
Nevertheless, designing secure federated correlation tests, despite its importance in real-world scenarios, is not explored by existing research in this ﬁeld.
On the other end of the spectrum, Wang et al.
(2021) proved that stable projection is differentially private if the projection matrix is secret.
In our protocol, the projection matrix is public information; hence F ED - χ 2 does not consider the differential privacy guarantee.
F EDERATED C ORRELATION T EST WITH M INIMAL L EAKAGE
In this section, we elaborate on the design of F ED - χ 2 , a secure federated protocol for χ 2 -test.
Sec.
ﬁrst formalizes the problem, establishes the notation system, and introduces the threat model.
In
Sec. 3.2, we recast χ 2 -test as a second frequency moment estimation problem in the federated setting, and consequently, we are able to leverage stable projection to encode each client’s local information (Sec. 3.3), and then aggregate them using secure aggregation (Sec. 3.4).
Sec. 3.5, 3.6, and 3.7 present security proof, utility analysis, communication analysis, and computation analysis of F ED - χ 2 .
3. P ROBLEM S ETUP
We now formulate the problem of the federated correlation test and establish the notation system.
We use [ n ] to denote { 1 , · · · , n } .
We denote vectors with bold lower-case letters ( e.g. , a , b , c ) and matrices with bold upper-case letters ( e.g. , A , B , C ).
We consider a population of n clients C
= { c i } i ∈ [ n ] .
Each client has one share of local data composed of the triplets D i
= { ( x, y, v ( i ) xy ) } , x ∈ X ,
y ∈ Y , v ( i ) xy ∈ {− M, · · · , M } , where x and y are categories of the contingency table, v ( i ) xy is the observed counting of the categories x and y in the local contingency table of the i th client, |X| =
m x and |Y| = m y are ﬁnite domains, and
M is the maximum value | v ( i ) xy | can be.
The global dataset is deﬁned as D =
{ ( x, y, v xy ) : v xy = (cid:80) i ∈ [ n ] v ( i ) xy } .
We focus on federated χ 2 -test
and the data in contingency table is discrete.
For the ease of presentation, we deﬁne the marginal statistics v x = (cid:80) y ∈
[ |Y| ] v xy , v y =
(cid:80) x ∈ [ |X| ] v xy , and v = (cid:80) x ∈ [ |X| ] ,
y ∈
[ |Y| ] v xy .
Besides, we deﬁne ¯ v xy = v x × v y v , denoting the expectation of v xy if x and y are uncorrelated.
We deﬁne m = m x m y and use an indexing function I : [ m x ] × [ m y ]
→ [ m ] to obtain a uniform indexing given the indexing of each variable.
A centralized server S calculates the statistics for χ 2 -test s χ
( D ) =
(cid:80) x ∈ [ |X| ] ,
[Equation]
xy on the global dataset to decide whether
X and Y are correlated without collecting the raw data from clients.
Overall, using MPC to conduct secure correlation tests in a federated scenario is highly expensive and impractical (Boyle et al., 2015; Damgård et al., 2012).
Hence, in the present work, we trade off accuracy for efﬁciency, as long as the estimation error is small with a high probability.
Formally, if
F ED - χ 2 outputs
ˆ s χ
, whose corresponding standard centralized χ 2 -test output is s χ 2 , the following accuracy requirement should be satisﬁed with small (cid:15) and δ .
P
[Equation]
Threat Model.
We assume that the centralized server S is honest but curious.
It honestly follows the protocol due to regulatory or reputational pressure but is curious to discover extra private information from clients’ legitimate updates for proﬁt or surveillance purposes.
As a result, client updates should contain as little sensitive information as feasible.
We want to emphasize that, while the server may explore the privacy of clients, the server will honestly follow the protocol due to regulation or reputational pressure.
The server won’t provide adversarial vectors to the clients.
On the other hand, we assume honest clients.
Speciﬁcally, we do not consider client-side adversarial attacks ( e.g. , data poisoning attacks (Bagdasaryan et al., 2020;
Bhagoji et al., 2019)).
However, we allow a small portion of clients to drop out during the execution.
Also, as we mentioned in Sec. 2, we do not consider the differential privacy guarantee; see further clariﬁcation in Appendix A.
3. F ROM C ORRELATION T EST TO F REQUENCY M OMENTS E STIMATION
We ﬁrst recast correlation test to a second frequency moments estimation problem as deﬁned below.
Given a set of key-value pairs
S =
{ k i , v i } i ∈ [ n ] , we re-organize it into a histogram H =
{ k j , v j = (cid:80) k
i =
k j ,i ∈ [ n ] v i } , and estimate the α th frequency moments as F α = (cid:80) j v αj .
χ 2 -test can thus be recast to a 2 nd frequency moments estimation problem as follows:
[Equation]
In federated setting, each client c i holds a local dataset D i
= { ( x, y, v ( i ) xy ) }
and computes a vector u i , where u i [ I ( x, y )]
= v ( i )
xy − ¯ v xy /n √ ¯ v xy and u i has m elements.
Thus, the challenge in federated χ 2 -test becomes calculating the following equation:
[Equation]
3. E NCODING BY S TABLE P ROJECTION & D ECODING BY G EOMETRIC M EAN E STIMATOR
To address the aforementioned challenges and to easily integrate the algorithm into secure aggregation protocols, we use stable random projection (Indyk, 2006; Vempala, 2005) and geometric mean estimator (Li, 2008) to approximate the data’s second frequency moment efﬁciently.
We begin by discussing stable distributions, followed by the encoding and decoding techniques.
Deﬁnition 1 ( α -stable distribution) .
A random variable X follows an α -stable distribution Q α,β,F if its characteristic function is as follows.
[Equation]
, where F is the scale to the α th power and β is the skewness.
α -stable distribution is named due to its property called α -stability.
Brieﬂy, the sum of independent α -stable variables still follows an α -stable distribution with a different scale.
Deﬁnition 2 ( α -stability) .
If random variables X ∼ Q α,β, 1 , Y ∼ Q α,β, 1 and X
and Y are indepen- dent, then C
[Equation]
We borrow the genius idea from Indyk’s well-celebrated paper (Indyk, 2006) to encode the frequency moments in the scale parameter of a stable distribution deﬁned in Deﬁnition 1.
To encode the local dataset D i
= { ( x, y, v ( i ) xy ) } , each client c i , i ∈ [ n ] , is given by the server an (cid:96)
× m projection matrix P whose values are drawn independently from an α -stable distribution Q α,β, 1 , where (cid:96) is the encoding
Note that the encoding and decoding themselves do not provide any security guarantee.
1 Function E NCODE ( P , u i ) : return P × u i 3 Function G EOMETRIC M EAN E STIMATOR ( e ) :
ˆ d (2) ,gm ← (cid:81) (cid:96)k =1 | e k | 2 /(cid:96)
(π Γ( 2 (cid:96) )
Γ(1 − 1 (cid:96) )sin( π(cid:96) ))
(cid:96) //
(cid:96) is the encoding size.
return ˆ d (2) ,gm 6 Function D ECODE ( e ) : size and m = m x m y .
The client re-organizes the data into a vector u i , where u i [ I ( x, y )]
= v ( i ) xy .
Then, the client projects u i to e i
=
P
×
u
i as the encoding (line 2 in Alg. 1).
To decode, the server ﬁrst sums the encodings up e =
(cid:80) i ∈ [ n ] e i and estimates the scale of the variables in the aggregated encoding with an unbiased geometric mean estimator (Li, 2008) in lineof Alg. 1.
According to the α -stability deﬁned in Deﬁnition 2, every element e k in e , k ∈ [ (cid:96) ] , follows this stable distribution:
e k ∼ Q 2 , 0 , || (cid:80) i ∈ [ n ] u i ||
Thus, the l 2 norm can be estimated by calculating the scale of the distribution Q 2 , 0 , || (cid:80) i ∈
[ n ] u i || 22 with e containing (cid:96) elements.
During the above processes, u i and (cid:80) i ∈ [ n ] u
i are not leaked and the aggregation among clients is calculated with only summation.
Thus, secure aggregation protocols can be naturally applied with quantization, as will be discussed in Sec.
3.4 and 3.5.
Furthermore, (cid:96) =
c(cid:15) 2 log(1 /δ )
sufﬁces to guarantee that the second frequency moment can be approximated with a 1 ± (cid:15) factor and a probability no less than 1 − δ .
We will further analyze the utility of F ED - χ 2 in Sec. 3.6.
3. S ECURE F EDERATED C ORRELATION T EST
The complete protocol for F ED - χ 2 is presented in Alg. 2.
Firstly, the marginal statistics v x , v y and v are calculated with secure aggregation and broadcasted to all clients (lines 1–6 in Alg. 2).
This step can be omitted if the marginal statistics are already known.
Then, on the server side, a projection matrix P is sampled from an α -stable distribution Q (cid:96) × m 2 , 0 , 1 .
The projection matrix is broadcasted to all clients (lines 8–10 in Alg. 2).
For each client c i , the local data is re-organized into u i and projected to e i as encoding (lines 11–14 in Alg. 2).
Then, the encoding results will be quantized and aggregated with secure aggregation (line 15 in Alg. 2).
As we have already known the marginal statistics in the ﬁrst round, the quantization bound can be set accordingly.
Additionally, we can use high precision for quantization, such as 64 bits, since the size of the contingency table is normally moderate rather than enormous.
Thus, the precision of the quantized ﬂoat number is comparable to or even better than that of ﬂoat64, and hence we disregard the effect of quantization on accuracy.
Finally, the server gets the χ 2 -test result using the decoding algorithm described in Alg.
1 (line 17 in Alg. 2).
Dropouts in the ﬁrst round have no effect on the test’s accuracy because they can be recovered inside secure aggregation (Bonawitz et al., 2017; Bell et al., 2020).
On the other hand, dropouts in the second round will affect the accuracy of the test.
Still, because the χ 2 value is typically far from the decision threshold, F ED - χ 2 is intrinsically robust to a small portion of clients dropping out (see
Section 4 for empirical assessment).
3. S ECURITY A NALYSIS
As discussed in Sec. 2, the secure aggregation protocol is well studied, and many well-celebrated secure aggregation protocols have been proposed (Bonawitz et al., 2017; Truex et al., 2019; Xu et al., 2019;
So et al., 2021; Bell et al., 2020; Choi et al., 2020).
In this paper, we choose the state-of-the-art secure aggregation protocol by Bell et al.
(Bell et al., 2020), which replaces the complete graph with a sparse random graph to enhance communication efﬁciency.
We clarify that F ED - χ 2 can incorporate other popular secure aggregation protocols.
We now prove the security enforced by Alg. 2 via a standard simulation proof process (Lindell, 2017) on the basis of Theorem 1.
Theorem 1 (Security) .
Let Π be an instantiation of Alg. 2 with the secure aggregation protocol in
Alg. 4 of Appendix G with cryprographic security parameter λ .
There exists a PPT simulator S IM such that for all clients C , the number of clients n , all the marginal distributions { v x } , { v y } , and all the encodings { e i } , the output of S IM is indistinguishable from the view of the real server Π C in that execution, i.e. , Π C ≈ λ S IM ( (cid:80) e i , n ) .
Intuitively, Theorem 1 illustrates that no more information about the clients except the averaged updates is revealed to the centralized server.
Thus, each client’s update is hidden by the rest clients in secure aggregation.
We now present the formal proof for Theorem 1.
Proof for Theorem 1.
To prove Theorem 1, we need the following lemma.
Lemma 1 (Security of secure aggregation protocol) .
Let S ECURE A GG be the secure aggregation protocol in Alg.
4 of Appendix G instantiated with cryprographic security parameter λ .
There exists a probabilistic polynomial-time (PPT) simulator S IM SA such that for all clients C , the number of clients n , and all inputs X = { e i } i ∈
[ n ] , the output of S IM SA is perfectly indistinguishable from the view of the real server, i.e. , S ECURE A GG C ≈ λ S IM SA ( (cid:80) i ∈ [ n ] e i , n ) .
Lemma 1 is derived from the security analysis of our employed secure aggregation protocol (Theorem 3.6 in Bell et al.
(2020)), which establishes that the secure aggregation protocol securely conceals the individual information in the aggregated result.
With this lemma, we are able to prove the theorem for federated χ 2 -test by presenting a sequence of hybrids that begin with real protocol execution and end with simulated protocol execution.
We demonstrate that every two consecutive hybrids are indistinguishable, illustrating that the hybrids are indistinguishable according to transitivity.
H YB
This is the view of the server in the real protocol execution, R EAL C .
H YB
In this hybrid, we replace the view during the execution of each S ECURE A GG ( { v ( i ) x } i ∈ [ n ] ) in line 3 of Alg.
2 with the output of S IM SA ( v x , n ) one by one.
According to Lemma 1, each replacement does not change the indistinguishability.
Hence, H YB 2 is indistinguishable from H YB 1 .
H YB 3 Similar to
H YB 2 , we replace the view during the execution of each
S ECURE A GG ( { v ( i ) y
} i ∈ [ n ] ) in line 4 of Alg.
2 with the output of S IM SA ( v y , n ) one by one.
According to Lemma 1, H YB 3 is indistinguishable from H YB 2 .
H YB
In this hybrid, we replace the view during the execution of S ECURE A GG ( { e i } i ∈ [ n ] ) in line 15 of Alg.
2 with the S IM SA ( (cid:80) e i , n ) .
This hybrid is the output of S IM .
According to Lemma 1, H YB 4 is indistinguishable from H YB 3 .
Remark:
what does Alg.
2 leak?
By Theorem 1, we show that individual updates of clients are perfectly hidden in the aggregated results and F ED - χ 2 leaks no more than a linear equation system:
[Equation]
Algorithm 2: F ED - χ 2 : secure federated χ 2 -test.
S ECURE
A GG is a remote procedure that receives inputs from the clients and returns the summation to the server.
I NIT S ECURE
A GG is the corresponding setup protocol deciding the communication graph and other hyper-parameters.
is
Server
Sample the projection matrix P from Q (cid:96) × m 2 , 0 ,
Broadcast the projection matrices to the clients
Client c i , i ∈ [ n ]
Calculate ¯ v xy
= v x v y v
Prepare u i s.t.
u i
[ I ( x, y )]
= v ( i )
xy − ¯ v xy /n √ ¯ v xy
Calculate e i = E
NCODE ( P , u i ) e =
S ECURE A GG (Q UANTIZE ( { e i } i ∈ [ n ] ))
Server ˆ s
2 = D ECODE ( e )
, where J 1 ,m x and J 1 ,m y are 1 × m x and 1 × m y unit matrices, V is an m x × m y matrix whose elements are { v xy } , and v is a vector ﬂattened by V .
To understand this, information leaked by F EDχ 2 includes the estimation e and marginal statistics v x and v y .
The following theorem establishes an important fact: the above equation system has an exponentially large solution space, which effectively conceals the real joint distribution.
We thus believe that Alg. 2 practically ensures privacy due to the solution space’s vastness.
Theorem 2.
Given a projection matrix P ∈ Z (cid:96) × m q , { v x } , { v y } and e , there are at least q m − (cid:96) − m x − m y feasible choices of { v xy } .
Proof sketch for Theorem 2.
As demonstrated above, the given information forms a linear equation with m x + m y + (cid:96) equations.
Given m > m x + m y + (cid:96) , the rank of the coefﬁcient matrix is no more than m x + m y + (cid:96) .
Solving the equations with Smith normal form, we know that the solution space is at least ( m − m x − m y − (cid:96) )
-dimensional.
With the following lemma, we manage to prove
Theorem 2.
Lemma 2.
There are q r × c vectors in the subspace of Z r × c q .
3. U TILITY A NALYSIS
In this section, we conduct the utility analysis in terms of multiplicative error.
We show that the output of F ED - χ 2 , ˆ s χ 2 , is a fairly accurate approximation (parameterized by (cid:15) ) to the correlation test output s χ 2 in the standard centralized setting with high probability parameterized by δ .
Theorem 3 (Utility) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G. Π is parameterized with (cid:96) =
c(cid:15) 2 log(1 /δ )
for some constant c .
After executing Π C on all clients C , the server yields ˆ s χ 2 , whose distance to the accurate correlation test output s χ 2 is bounded with high probability as follows:
[Equation]
Proof sketch for Theorem 3.
First, we introduce the following lemma from Li (2008).
Lemma 3 (Tail bounds of geometric mean estimator (Li, 2008)) .
The right tail bound of geometric mean estimator is:
[Equation]
where (cid:15) 2 G R = C 1 log(1 + (cid:15) ) − C 1 γ e ( α − 1) − log( 2 π Γ( αC 1 )Γ(1 − C 1 ) sin( παC 1 2 )) , α = 2 in our setting, C 1 =
π tan − 1 ( log(1+ (cid:15) )
(2+ α 2 ) π/ 6 ) , and γ e = 0 .
577215665 ... is the Euler’s constant.
The left tail bound of the geometric mean estimator is:
[Equation]
where (cid:96) > (cid:96) 0 ,
[Equation]
−
[Equation]
0 C
[Equation]
log
[Equation]
Γ
[Equation] [Equation]
) sin(
[Equation]
and C 2 = 12 π(cid:15)
[Equation]
With Lemma 3, Taking c ≥ max( G R , G L ) and δ = exp( − (cid:96)(cid:15) 2 c ) , we are able to prove P [ˆ s χ 2 < (1 − (cid:15) )
s χ
2 ∨ ˆ s χ 2 > (1 + (cid:15) ) s χ 2 ] ≤ δ , and the above bound holds when (cid:96) =
c(cid:15) 2 log(1 /δ ) .
3. C OMMUNICATION & C OMPUTATION A NALYSIS
In this section, we present the communication and computation cost of Alg. 2.
Theorem 4 (Communication Cost) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G, then (1) the client-side communication cost is O (log n + m x + m y + (cid:96) ) ; (2) the server-side communication cost is O ( n log n + nm x + nm y + n(cid:96) ) .
Theorem 5 (Computation Cost) .
Let Π be an instantiation of Alg. 2 with secure aggregation protocol in Alg.
4 of Appendix G, then (1) the client-side computation cost is O (log 2 n +( (cid:96) + m x + m y ) log n + m(cid:96) ) ; (2) the server-side computation cost is O ( n log 2 n + n ( (cid:96) + m x + m y ) log n + (cid:96) ) .
Note that compared with the original computation cost presented in (Bell et al., 2020), the client-side overhead has an extra O ( m(cid:96) ) term.
This term is incurred by the encoding overhead.
We also give an empirical evaluation on the client-side computation overhead in Sec. 4.1.
Please refer to Appendix H for the detailed proof of Theorem 4 and Theorem 5.
E VALUATION
Experiment Setup.
To assess F ED - χ 2 ’s accuracy, we simulate it on four synthetic datasets and 16 real-world datasets.
We compare the multiplicative errors of F ED - χ 2 with that of the standard centralized χ 2 -test.
The four synthetic datasets are independent, linearly correlated, quadratically correlated, and logistically correlated.
As the real-world datasets, we report the details in Appendix I.
Additionally, we evaluate F ED - χ 2 ’s utility in three real-world application scenarios: 1) feature selection, 2) cryptanalysis, and 3) online false discovery rate (FDR) control.
For feature selection, we report the model accuracy trained on the selected features.
For cryptanalysis, we report the success rate of cracking ciphertexts.
For Online FDR control, we report the average false discovery rate.
We compare the performance of F ED - χ 2 with that of the centralized χ 2 -test in each of the three experiments.
Unless otherwise speciﬁed, experiments are launched on an Ubuntu 18.04 LTS server equipped with 32 AMD Opteron(TM)
Processor 6212 and 512GB RAM.
4. E VALUATION R ESULTS
[Figure]
lti p
[Figure]
ca ti v e E rr o r (cid:15) /
A
CC
[Figure]
lti
[Figure]
E rr o r (cid:15) /
A
CC (i)
[Figure]
lti
[Figure]
li ca ti v e E rr o r (cid:15) /
A
CC
[Figure]
M u lti p li ca ti v e E
rr o r (cid:15) /
[Figure]
CC
[Figure]
Accuracy.
We begin by evaluating the accuracy of F ED - χ 2 , as illustrated in Fig.
Each point represents the mean of 100 independent runs with 100 clients, while the error bars indicate the standard deviation.
We choose m x = m y = 20 in this experiment.
Note that the accuracy drop is independent of the number of clients.
From Fig.
1, we observe that the larger the encoding size (cid:96) , the smaller the multiplicative error.
When (cid:96) = 50 , the multiplicative error (cid:15)
≈ 0 .
This conforms with Theorem 3, in which the multiplicative error (cid:15) =
(cid:112) c(cid:96)
log(1 /δ ) decreases as (cid:96) increases.
We also evaluate the power (Cohen, 2013) of F ED - χ 2 .
We set the p-value threshold as 0 . 05 , which determines whether or not to reject the null hypothesis.
From The dashed lines in Fig.
1, we can tell that the power of F ED - χ 2 is high.
This conforms with our observation on the multiplicative errors.
Speciﬁcally, since the χ 2 values are typically far from the decision threshold, a multiplicative error of 0 . 2 rarely ﬂips the ﬁnal decision.
We also present the results when 5% of clients drop out in the second round of F ED - χ 2 in Fig.
The results show that F ED - χ 2 is robust to a small portion of dropouts.
In Appendix F, we present the results in terms of 10%, 15%, and 20% dropout rates.
[Figure]
Client-side Computation Overhead.
To assess extra computation overhead incurred by F ED - χ 2 on the client side, we measure the execution time of the encoding scheme on an Android 10 mobile device equipped with a Snapdragon865 CPU and 12GB RAM.
We use
PyDroid (Sandeep Nandal, 2020) to run the client-side computation of F ED - χ 2 on the Android device.
The results are shown in Fig.
Each point represents the average of 100 separate runs, with accompanying error bars.
The overhead is generally negligible.
For example, for a 500 × 500 contingency table, the encoding takes less than 30ms.
The overhead grows linearly in relation to m x ( m y ) and consequently quadratically in Fig.
2, where m x equals m y .
4. D OWNSTREAM U SE C ASE S TUDY
[Figure]
Feature Selection.
Our ﬁrst case study explores secure federated feature selection using F ED - χ 2 .
The setting is that each client holds data with a large feature space and wants to collaborate with other clients to rule out unimportant features and retain features with topk highest
χ 2 scores.
We use Reuters-21578 (Hayes & Weinstein, 1990), a standard text categorization dataset (Yang, 1999; Yang &
Pedersen, 1997; Zhang & Yang, 2003), and pick the top-20 most frequent categories using 17,262 training and 4,316 test documents.
These documents are distributed randomly to 100 clients, each of whom receives the same number of training documents.
After removing all numbers and stop-words, we obtain 167,135 indexing terms.
After performing feature selection using F ED - χ 2 , we select the top 40,000 terms with the highest χ 2 scores.
When compared with the centralized χ 2 -test, 38,012 (95.03%) of the selected terms are identical, indicating that
F ED - χ 2 produces highly consistent results with the standard χ 2 -test.
We then train logistic regression models using the terms selected by F ED - χ 2 and the centralized χ 2 -test, respectively.
All hyper-parameters are the same.
The details of these models are reported in
Appendix J. The results in Fig.
3 further demonstrate that F ED - χ 2 exhibits comparable performance with the centralized χ 2 -test.
When 10% and 20% of clients dropout in the second round of F ED - χ 2 , the accuracy of the trained model using the features selected by F ED - χ 2 does not drop much.
We also examine performance without feature selection, and as expected, model accuracy is signiﬁcantly greater after feature selection.
Note that the model without feature selection has 2,542,700 more parameters than the model with feature selection.
Hence, feature selection effectively improves model accuracy while reducing model size and computational cost.
Cryptanalysis.
In the second case study, we explore federated cryptanalysis with F ED - χ 2 .
We break Caesar cipher (Luciano & Prichett, 1987), a classic substitution cipher, with F ED - χ 2 .
In a
Caesar cipher, each letter in the plaintext is replaced by another letter with some ﬁxed number of positions down the alphabet.
For instance, each English letter can be right-shifted by three, converting the plaintext “good” to the ciphertext “jrrg”.
There are 26 possible shifts when given 26 English letters.
The plaintext can be cracked in a shortcut by performing a correlation test on the ciphertext in relation to normal English text.
[Figure]
In our setting, each client is assumed to possess a segment of the
Caesar ciphertext.
To collaboratively crack the ciphertext, these clients run 26 χ 2 -tests to determine the correlation level between each ciphertext letter and the letters in normal English text.
The χ 2 -test yielding the highest correlation level elucidates how English letters are encrypted into Caesar ciphertexts.
We take Shakespeare’s lines as the plaintext and encrypt it into a
Caesar ciphertext with a length of 1000 characters.
We initiate the cracking process on ten non-overlapping ciphertexts to compute the average success rate (see Fig. 4).
In general, the larger the encoding size, the more precise the χ 2 -statistics is, and consequently the higher the success rate.
Again, according to Theorem 3, the multiplicative error (cid:15) decreases as the encoding size increases.
In Fig.
4, we also report the success rate when 10% and 20% of clients dropout in Round two of F ED - χ 2 , respectively.
Even if 20% of clients dropout, the success rate can still be 100% as long as the encoding size (cid:96) is larger than 20.
[Figure]
Online False Discovery Rate Control.
In the third case study, we explore federated online false discovery rate (FDR) control (Foster & Stine, 2008) with F ED - χ 2 .
In an online FDR control problem, a data analyst receives a stream of hypotheses on the database, or equivalently, a stream of p -values: p 1 , p 2 , · · · .
At each time t , the data analyst should pick a threshold α t to reject the hypothesis when p t < α t .
The error metric is the false discovery rate, and the objective of online FDR control is to ensure that for any time t , the
FDR up to time t is smaller than a pre-determined quantity.
We use the SAFFRON procedure (Ramdas et al., 2018), the state-ofthe-art online FDR control, for multiple hypothesis testing.
The χresults and corresponding p -values are calculated by F ED - χ 2 .
We present the detailed algorithm of SAFFRON and the hyper-parameters used in our evaluation in
Appendix K.
The size of the randomly synthesized contingency table is 20 × 20 .
Each time, there are 100 independent hypotheses, with a probability of 0 .
5 that each hypothesis is either independent or correlated.
The time sequence length is 100 , and the number of clients is 10 .
The data are synthesized from a multivariate Gaussian distribution.
For the correlated data, the covariance matrix is randomly sampled from a uniform distribution.
For the independent data, the covariance matrix is diagonal, and its entries are randomly sampled from a uniform distribution.
At time t , we use F ED - χ 2 to calculate the p -values p t of all the hypotheses, and then use the
SAFFRON procedure to estimate the reject threshold α t using p t .
The relationship between the average FDR and encoding size (cid:96) is shown in Fig.
We observe that the variance of independent runs is very small, so we omit the error bars.
The results indicate that by increasing the encoding size (cid:96) , F ED - χ 2 can achieve a low FDR of less than 5 . 0% .
We also observe that dropouts improve the performance of F ED - χ 2 in this case study.
The reason for this is because dropouts reduce the estimated χ 2 value, which increases the probability of accepting the null hypothesis in F ED - χ 2 .
As a larger portion of queries follows the null hypothesis in online FDR control, the accuracy also increases.
The results further demonstrate that F ED - χ 2 can be employed in practice to facilitate online FDR control using a secure federated correlation test.
C ONCLUSION
This paper takes an important step towards designing non-linear secure aggregation protocols in the federated setting.
Speciﬁcally, we propose a universal secure protocol to evaluate frequency moments in the federated setting.
We focus on an important application of the protocol: χ 2 -test.
We give formal security proof and utility analysis on our proposed secure federated learning χ 2 -test protocol
F ED - χ 2 and validate them with empirical evaluations and case studies.
A PPENDIX A C LARIFICATION ON P RIVACY
As indicated in Sec. 2, the projection matrix is public information, and hence F ED - χ 2 does not take the differential privacy guarantee into account.
We would want to provide more information in order to eliminate any potential misunderstandings.
To begin, we would want to emphasize that “privacy” in our paper refers to MPC-style privacy, not
DP-style privacy.
In general, MPC-style privacy is orthogonal to DP-style privacy: in MPC, privacy is obtained against a semi-honest server in such a way that the server cannot witness individual client’s updates but only an aggregate of them, e.g. SecAgg (Bonawitz et al., 2017).
In DP, privacy is accomplished by including random noise in each client’s update, such that the distribution of the output result does not reveal the clients’ private information and the server cannot infer the clients’ identiﬁcation from the output result.
Second, we would like to emphasize that our work proposes a novel secure aggregation scheme particularly for the χ 2 -test.
Existing standard secure aggregation schemes are inapplicable to the χ 2 -test, which will reveal much more information than F ED - χ 2 , as we have clariﬁed in Sec. 1.
Again, this work requires guaranteeing MPC-style privacy, not DP.
Third, to quantify MPC-style privacy, we prove in Theorem 2 that the clients’ updates in F ED - χ 2 are hidden inside a space with exponential size.
This is weaker than hiding users’ updates in the whole space, but still gives meaningful privacy guarantees (consider attempting to guess the output of an exponential-sided dice, which is practically infeasible).
Finally, while DP is orthogonal to this research, we would want to emphasize that our protocol can achieve DP by introducing calibrated discrete Gaussian noise to the users’ local updates.
B C OMPARISON WITH P OOLING χ 2 - TEST
We also compare the performance of F ED - χ 2 with pooling χ 2 -test.
That is, the clients compute the χ 2 -test with their local observations and then they aggregate their test results by pooling.
The result of the pooling χ 2 -test is determined by the majority of the clients’ local results.
Fig.
6 shows the result of pooling χ 2 -test on the real-world datasets.
The details of the datasets are presented in Appendix I.
We observe that pooling χ 2 -test cannot give meaningful results and it tends to give judgement that the data is independent since that the numbers of the local observations are not sufﬁcient to make correct judgement.
The results further demonstrate the effectiveness and the necessity of F ED - χ 2 .
C P ERFORMANCE OF F ED - χ 2 W HEN O RIGINAL χ 2 - TEST A CHIEVES L OW P OWER
We have computed the multiplicative error and accuracy of F ED - χ 2 with the original centralized χ 2 -test as the baseline.
To further demonstrate the effectiveness of F ED - χ 2 , we also set up the experiment to evaluate the performance of F ED - χ 2 on the synthesized dataset when the accuracy of the original centralized χ 2 -test is lower.
More speciﬁcally, we synthesize the random independent, linearly correlated, quadratically correlated, and logistically correlated datasets with the same hyperparameters for 20 times.
We then compute the accuracy of the original centralized
χ 2 -test over these 20 datasets.
We report the results in Fig.
The results show that F ED - χ 2 can still achieve good performance on the datasets when the original centralized χ 2 -test’s accuracy is lower.
Consistent with our results in Sec.
4.1, the multiplicative error becomes lower with the increase of the encoding size l .
Thus, we conclude that F ED - χ 2 ’s performance is comparable to that of the original centralized χ 2 -test with an appropriate encoding size l and is not dependent on the datasets.
D F URTHER R ESULTS FOR O NLINE FDR C ONTROL
[Figure]
[Figure] [Figure]
In this section
[Figure]
we provide further results for online FDR control.
As we have shown in Fig.
5, F ED - χ 2 achieves good performance (FDR lower than 5%) when the encoding size l is larger than 200.
In
Fig.
8, we provide the FDR result of the original χ 2 -test as well as the true discovery rate (TDR, i.e., #correct reject / #should reject).
In addition, we provide statistics for each encoding size l that was evaluated in Table 1.
These results demonstrate that F ED - χ 2 performs well and is comparable to the centralized χ 2 -test when the encoding size l is increased.
More importantly, as we have shown in Sec. 4.
and Appendix C, the performance of F ED - χ 2 is independent from the data.
[Table]
E I NCORPORATE G AUSSIAN M ECHANISM IN F ED - χ
As we have mentioned in Appendix A, F ED - χ 2 can achieve differential privacy easily by incorporating well-studied differentially private mechanisms.
To further demonstrate this point, we utilize Gaussian
Mechanism to provide ( (cid:15), δ ) -DP guarantee.
We clipped the local clients’ data such that the encoding function’s sensitivity, which is the L2-norm of the clients’ local data is bounded by ∆ f .
Before encoding their local data, each client add Gaussian noise N d (0 , σ 2 ) to their local data vector µ i .
We can calculate σ 2 = 2ln(1 .
25 /δ )
(∆ f ) 2 n ( (cid:15)/l ).
After encoding and decoding with the computed vector, F ED - χprovides ( (cid:15), δ ) − DP guarantee.
We evaluate the performance of differentially private F ED - χ 2 on the ﬁrst four real-world datasets adopted in this research as shown in Fig. We choose encoding size l = 10 and δ = 0 .
The results show that the performance our algorithm becomes better with the increase of privacy budget (cid:15) .
However, when the privacy budget (cid:15) is small, the protocol tends to give the judgement that the data is independent because that the independent noise is too large and it dominates the test.
When the privacy budget (cid:15) is large enough (i.e., (cid:15) = 100 ), our protocol can achieve 0.92, 0.68, 0.70, and 0.
accuracy on these datasets accordingly, and its performance is comparable to the original F ED - χ 2 .
Again, our work is orthogonal to differential privacy, thus we leave it as future work to further study saving privacy budget, and boosting the algorithm’s performance on lower (cid:15) .
[Figure]
F F URTHER R ESULTS ON F ED - χ 2 WITH D ROPOUTS
We present the results of 10%, 15%, and 20% clients dropout in Fig.
The results further show that
F ED - χ 2 can tolerate a considerable portion of clients dropout in Round 2 of Alg. 2.
G S ECURE A GGREGATION
The secure aggregation protocol from Bell et al.
(2020) is presented in Alg. 4.
The ﬁrst step of the protocol is to generate a k -regular graph G , where the n vertices are the clients participating in the protocol.
The server runs a randomized graph generation algorithm I NIT S ECURE A GG presented in Alg. 3 that takes the number of clients n and samples output ( G, t, k ) from a distribution D .
In
Alg. 3, we uniformly rename the nodes of a graph known as a Harary graph deﬁned in Deﬁnitionwith n nodes and k degrees.
The graph G is constructed by sampling k neighbours uniformly and without replacement from the set of remaining n − 1 clients.
We choose k
= O ( log ( n )) , which is large enough to hide the updates inside the masks.
t is the threshold of the Shamir’s Secret Sharing.
[Figure]
CC
[Figure]
M u lti p li ca ti v e E rr o r (cid:15) /
A
CC
[Figure]
i)
[Figure]
u lti
[Figure]
ca
[Figure]
e E rr o r (cid:15) /
A
CC
[Figure]
In the second step, the edges of the graph determine pairs of clients, each of which runs key agreement protocols to share random keys.
The random keys will be used by each party to derive a mask for her input and enable dropouts.
In the third step, each client c i , i ∈
A 1 sends secret share to its neighbors.
In the fourth step, the server checks whether the clients dropout exceeds the threshold δ , and lets the clients know their neighbors who didn’t dropout.
In the ﬁfth step, each pair ( i, j ) of connected clients in G runs a λ -secure key agreement protocol s i,j = KA .Agree
( sk 1 i , pk 1 j ) which uses the key exchange in the previous step to derive a shared random key s i,j .
The pairwise masks m i,j = F ( s i,j ) can be computed, where F is the pseudorandom generator (PRG).
If the semi-honest server announces dropouts and later some masked inputs of the claimed dropouts arrive, the server can recover the inputs.
To prevent this happening, another level of masks, called self masks, r i is added to the input.
Thus, the input of client c i is: y i =
e i
+ r i − (cid:80) j ∈ N G
( i ) ,j<
i m i,j + (cid:80) j ∈ N G
( i ) ,j>i m i,j .
Steps 6–8 deal with the clients dropout by recovering the self masks r i of clients who are still active and pairwise masks m i,j of the clients who have dropped out.
Finally, the server can cancel out the pairwise masks and subtract the self masks in the ﬁnal sum: (cid:80) i ∈ A (cid:48) 2 ( y i − r i + (cid:80) j ∈ NG ( i ) ∩
( A (cid:48) 1 \ A (cid:48) 2 ) , 0 <j<i m i,j − (cid:80) j ∈ NG ( i ) ∩
( A (cid:48) 1 \ A (cid:48) 2 ) ,i<j ≤ n m i,j ) .
Deﬁnition 3 (H ARARY ( n, k ) Graph) .
Let H ARARY ( n, k ) denotes a graph with n nodes and degree k .
This graph has vertices V =
[ n ] and an edge between two distinct vertices i and j if and only if j − i (mod n ) ≤ ( k + 1) / 2 or j − i (mod n ) ≥ n − k/ 2 .
Algorithm
1 Function I NIT S ECURE A GG ( n ) : (cid:46) n : Number of nodes.
(cid:46) t : Threshold of Shamir’s Secret Sharing.
k
= O ( log ( n )) .
Let H
=
H ARARY ( n, k ) .
Sample a random permutation π : [ n ] → [ n ] .
Let G be the set of edges { ( π ( i ) , π ( j ))
| ( i, j ) ∈ H } .
return ( G, t, k )
H P ROOF FOR C OMMUNICATION & C OMPUTATION C OST
We provide the proof for Theorem 4 and Theorem 5 in the following.
Theorem 4 (Communication Cost).
Let Π be an instantiation of Alg. 2 with secure aggregation protocol from Bell et al.
(2020), then (1) the client-side communication cost is O (log n + m x + m y + (cid:96) ) ; (2) the server-side communication cost O ( n log n + nm x + nm y + n(cid:96) ) .
Proof sketch for Theorem 4.
Each client performs k key agreements ( O ( k ) messages, line 9 in Alg. 4) and sends 3 masked inputs ( O ( m x + m y + (cid:96) ) complexity, lines 3, 4, 15 in Alg. 2 and line 10 in
Alg. 4).
Thus, the client communication cost is O (log n + m x + m y + (cid:96) ) .
The server receives or sends O (log n + m x + m y + (cid:96) ) messages to each client, so the server communication cost is O ( n log n + nm x + nm y + n(cid:96) ) .
Theorem 5 (Computation Cost).
Let Π be an instantiation of Alg. 2 with secure aggregation protocol from Bell et al.
(2020), then (1) the client-side computation cost is O ( m x log n + m y log n + (cid:96) log n + m(cid:96) ) ; (2) the server-side computation cost is O ( m x + m y + (cid:96) ) .
Proof sketch for Theorem 5.
Each client computation can be broken up as k key agreements ( O ( k ) complexity, line 9 in Alg. 4), generating masks m i,j for all neighbors
c j ( O ( k ( m x + m y + (cid:96) )) complexity, lines 3, 4, 15 in Alg. 2 and line 10 in Alg. 4), and encoding computation cost O ( m(cid:96) )
(line 14 in Alg. 2).
Thus, the client computation cost is O ( m x log n + m y log n + (cid:96) log n + m(cid:96) ) .
The server-side follows directly from the semi-honest computation analysis in Bell et al.
The extra O ( (cid:96) ) term is the complexity of the geometric mean estimator.
I D ETAILS OF D ATASETS
The details for the real-world datasets used in Sec.
4.1 are provided in Table 2.
The license of
Credit Risk Classiﬁcation (Govindaraj, Praveen) is CC BY-SA 4.0, the license of German Trafﬁc
Sign (Houben et al., 2013) is CC0: Public Domain.
Other datasets without a license are from UCI
Machine Learning Repository (Dua & Graff, 2017).
J D ETAILS OF R EGRESSION M ODELS
The details of the regression models trained in feature selection in Sec.
4.2 is reported in Table 3.
The training and testing splits are the same for F ED - χ 2 , centralized χ 2 -test and model without feature
(cid:46) We denote by A 1 , A 2 , A 3 the sets of clients that reach certain points without dropping out.
Speciﬁcally A 1 consists of the clients who ﬁnish step (3), A 2 those who ﬁnish step (5), and
A 3 those who ﬁnish step (7).
For each A i , A (cid:48)
i is the set of clients for which the server sees they have completed that step on time.
(1) The server runs ( G, t, k )
=
I NIT S ECURE A GG ( n ) , where G is a regular degree- k undirected graph with n nodes.
By N G ( i ) we denote the set of k nodes adjacent to c
i (its neighbors).
generates key pairs ( sk 1 i , pk 1 i ) , ( sk 2 i , pk 2 i ) and sends ( pk 1 i , pk 2 i ) to the server who forwards the message to N G ( i ) .
Client c i
i
A 1 do • Generates a random PRG seed b i .
• Computes two sets of shares:
[Equation]
• Sends to the server a message m =
( j, c i,j ) , where c i,j = E auth .Enc
( k i,j , ( i ||
j || h
bi,j || h si,j )) and k i,j = KA .Agree
( sk 2 i , pk 2 j ) , for each j ∈ N G ( i ) .
(4) The server aborts if | A (cid:48) 1 | < (1 − δ ) n and otherwise forwards ( j, c i,j ) to client c j who deduces A (cid:48) 1 ∩ N G ( j ) .
(5) for each Client c
i , i ∈
A 2 do • Computes a shared random PRG seed s i,j as s i,j = KA .Agree
( sk 1 i , pk 1 j ) .
• Computes masks m i,j = F ( s i,j ) and r i = F ( b i ) .
• Sends to the server their masked input
[Equation]
(6) The server collects masked inputs.
It aborts if | A (cid:48) 2 | < (1 − δ ) n and otherwise sends ( A (cid:48) 2 ∪ N G ( i ) , ( A 1 \ A (cid:48) 2 ) ∪ N G ( i )) to every client c
i , i ∈
A (cid:48) 2 .
(7) Client c j , j ∈
A 3 receives ( R 1 , R 2 ) from the server and sends { ( i,
h b
i,j ) } i ∈ R 1 ∪ { ( i,
h
s i,j ) } i ∈ R 2 obtained by decrypting the c i,j received in Step (3).
server aborts if | A (cid:48) 3 | < (1 − δ ) n and otherwise: • Collects, for each client c
i , i ∈
A (cid:48) 2 , the set B i of all shares in H bi sent by clients in A 3 .
Then aborts if | B i | < t and otherwise recovers b i and r
i using the t shares received which came from the lowest client IDs.
• Collects, for each client c
i , i ∈ ( A 1 \ A (cid:48) 2 ) , the set S i of all shares in H si sent by clients in A 3 .
Then aborts if | S i | < t and otherwise recovers sk 1 i and m i,j . • return (cid:80)
i ∈ A (cid:48)
( y i − r
i + (cid:80) j ∈ NG ( i ) ∩
( A (cid:48) 1 \ A (cid:48) 2 ) , 0 <j<i m i,j − (cid:80) j ∈ NG ( i ) ∩
( A (cid:48) 1 \ A (cid:48) 2 ) ,i<j ≤ n m i,j ) .
selection (i.e. there are 17,262 training and 4,316 test documents).
We use the same learning rate; random seed and all other settings are also the same to make the comparison fair.
We get the result of
Fig.
3 and the models are all trained on NVIDIA GeForce RTX 3090.
[Table] [Table]
K SAFFRON P ROCEDURE
In Sec.
4.2, we adopt the SAFFRON procedure (Ramdas et al., 2018) to perform online FDR control.
SAFFRON procedure is currently the state of the arts for multiple hypothesis testing.
In Alg. 5, we formally present the SAFFRON algorithm.
Algorithm
[Table] [Table]
// Set rejection number.
[Table]
for each p -value p t ∈ { p 1 , p 2 , · · · } doλ t ← g t ( R 1: t − 1 , C 1: t − 1 ) C t ← I ( p t < λ t )
// Set the indicator for candidacy C t .
j + ← (cid:80) t − 1 i =
τ j +1 C
i
// Set the candidates after the j th rejection.
if t =
then
− λ
γ 1 W
t
(1 − λ t )( W 0 γ t − C 0+ + ( α − W 0 ) γ t − τ 1 − C 1+ + (cid:80) j ≥ 2 αγ t − τ j − C j + )
R t ← I ( p t ≤ α t )
// Output R
t .
=
// Update rejection number.
τ i ← t // Set the i th rejection time.
return { R 0 , R 1 , · · · }
The initial error budget for SAFFRON is (1 − λ 1 W 0 ) <
(1 − λ 1 α ) , and this will be allocated to different tests over time.
The sequence { λ j } ∞ j =1 is deﬁned by g t and λ j serves as a weak estimation of α j .
g t can be any coordinate wise non-decreasing function (line 8 in Alg. 5).
R j :=
I ( p j < α j ) is the indicator for rejection, while C j :=
I ( p j < λ j ) is the indicator for candidacy.
τ j is the j th rejection time.
For each p t , if p t < λ t , SAFFRON adds it to the candidate set C t and sets the candidates after the j th rejection (lines 9-10 in Alg. 5).
Further, the α t is updated by several parameters like current wealth, current total rejection numbers, the current size of the candidate set, and so on (lines 11-14 in
Alg. 5).
Then, the decision R t is made according to the updated α t (line 15 in Alg. 5).
The hyper-parameters we use for the SAFFRON procedure in online false discovery rate control of Sec. 4 are aligned with the setting in Ramdas et al.
In particular, the target FDR level is α = 0 .
05 , the initial wealth is W 0 = 0 . 0125 , and γ j is calculated in the following way: γ j =
[Equation]