SecureFedYJ: a safe feature Gaussianization protocol for Federated Learning
Abstract
The Yeo-Johnson (YJ) transformation is a standard parametrized per-feature unidimensional transformation often used to Gaussianize features in machine learning.
In this paper, we investigate the problem of applying the YJ transformation in a cross-silo Federated Learning setting under privacy constraints.
For the ﬁrst time, we prove that the YJ negative log-likelihood is in fact convex, which allows us to optimize it with exponential search.
We numerically show that the resulting algorithm is more stable than the state-of-the-art approach based on the Brent minimization method.
Building on this simple algorithm and Secure Multiparty
Computation routines, we propose S ECURE F ED YJ, a federated algorithm that performs a pooled-equivalent YJ transformation without leaking more information than the ﬁnal ﬁtted parameters do.
Quantitative experiments on real data demonstrate that, in addition to being secure, our approach reliably normalizes features across silos as well as if data were pooled, making it a viable approach for safe federated feature Gaussianization.
Introduction
Federated Learning (FL) [45, 32] is an approach that was recently proposed to train machine learning (ML) models across multiple data holders or clients , without centralizing data points, notably for privacy reasons.
While many FL applications have been proposed, two main settings have emerged [23]: cross-device FL, involving a large number of small edge devices and cross-silo FL, dealing with a smaller number of clients, with larger computational capabilities.
Due to the sensitivity and relative local scarcity of medical data, healthcare is a promising application of cross-silo FL [40], e.g. to train a biomedical ML model between different hospitals as if all the datasets were pooled in a central server.
In this paper, we focus on the cross-silo setting.
The constraints of cross-silo FL
Although cross-silo FL resembles standard distributed learning, it faces at least two important distinct challenges: privacy and heterogeneity.
Due to data sensitivity, clients might impose stringent security and privacy constraints on FL collaborations.
This arises in coopetitive FL projects, where models are jointly trained on industrial competitors’ datasets [55], as well as medical FL applications, where conservative data regulations might apply.
In this setting, using standard FL algorithms such as F ED A VG [32] might not provide enough privacy guarantees, as privacy attacks such as data reconstruction can be carried out based on the clients’ gradients [56, 54].
Various protocols based on Secure Multiparty Computation (SMC) (see Section 2 for more details), such as Secure Aggregation [4], can mitigate this shortcoming by disclosing only the sum of the gradients from all clients to the server, without disclosing each gradient individually.
An additional constraint is that data might present statistical heterogeneity across clients, i.e. the local clients’ data distributions may not be identical.
In the case of medical applications, such heterogeneity may be caused e.g. by environmental variations or differences in the material that was used for acquisition [43, 47, 2].
While different ways of adapting federated training algorithms have been proposed to automatically tackle heterogeneity [28, 29, 24], these solutions do not address data harmonization and normalization prior to FL training.
Preprocessing in ML Data preprocessing is a crucial step in many ML applications, leading to important performance gains.
Among others, common preprocessing methods include data whitening, principal component analysis (PCA) [22] or zero component analysis [27, 20, 46].
However, linear normalization methods might not sufﬁce when the original data distribution is highly non-Gaussian.
For tabular and time series data, a popular approach to Gaussianize the marginal distributions is to apply feature-wise non-linear transformations.
Two commonly-used parametric methods are the
Box-Cox [5] transformation and its extension, the Yeo-Johnson (YJ) transformation [52].
Both have been used in multiple applications, such as climate and weather forecast [53, 50, 51], economics [13] and genomic studies [7, 58, 9].
Problem and contributions In this paper, we investigate the problem of data normalization in the cross-silo FL setting, by exploring how to apply the YJ transformation to a distributed dataset.
This problem arises frequently in medical cross-silo FL, e.g. when trying to jointly train models on genetic data (see e.g. [19, 57]).
Due to data heterogeneity, no single client can act as a reference client: indeed, there is no guarantee that transformation parameters ﬁtted on a single client would be relevant for other clients’ data.
Hence, it is necessary to ﬁt normalization methods on the full federated dataset.
Moreover, in this setting, data privacy is of paramount importance, and therefore
FL protocols should be carefully designed.
Our main contributions to this problem are as follows:
We prove that the negative YJ log-likelihood is convex (Section 3), which is a novel result, to the best of our knowledge.
Building on this property, we introduce E XP YJ, a method to ﬁt the YJ transformation based on exponential search (Section 3).
We numerically show that this method is more stable than standard approaches for ﬁtting the YJ transformation based on the Brent minimization method [6].
We propose S ECURE F ED YJ (Section 4), a secure way to extend E XP YJ in the cross-silo FL setting using SMC.
We show that S ECURE F ED YJ does not leak any information on the datasets apart from what is leaked by the parameters minimizing the YJ negative loglikelihood (Section 4 and Proposition 4.1).
By construction, S ECURE F ED YJ provides the same results as the pooled-equivalent E XP YJ, regardless of how the data is split across the clients.
We check this property in numerical experiments (Section 4).
The core ideas behind the resulting algorithm, S ECURE F ED YJ, are summarised in Figure 7.
Finally, we illustrate our contributions in numerical applications on synthetic and genomic data in
Section 5.
Background
The Yeo-Johnson transformation The YJ transformation [52] was introduced in order to Gaussianize data that can be either positive or negative.
It was proposed as a generalization of the Box-Cox transformation [5], that only applies to non-negative data.
The YJ transformation consists in applying to each feature a monotonic function Ψ( λ, · ) parametrized by a scalar λ , independently of the other features.
Thus, there are as many λ ’s as there are features.
For a real number x , Ψ( λ, x ) is deﬁned as:
- -   4 x - -    ( , x ) = 0.
=0.0=0.
=1.0=1.
[Figure] 3 x 0.
3 ( , x ) 0.
Yeo-Johnson
[Figure]
[Equation]
Figure 1a shows the shape of the YJ function for various values of λ .
The Yeo-Johnson likelihood Let us consider real-valued samples { x i } i =1 , ··· ,n , and let us apply the YJ transformation Ψ( λ, · ) to these samples to Gaussianize their distribution.
The log-likelihood that { Ψ( λ, x i ) } i =1 , ··· ,n comes from a Gaussian with mean µ and variance σ 2 is given by (derivation details are provided in Appendix A.1):
[Equation]
For a given λ , the log-likelihood is maximized for µ ∗ = 1 n (cid:80) ni
=1 Ψ( λ, x i ) and σ 2 ∗ = n (cid:80) ni
=1 (Ψ( x i , λ ) − µ ∗ ) 2 .
Once we replace µ and σ 2 by µ ∗ and σ 2 ∗ , it becomes:
[Equation]
see [52].
Maximizing the YJ log-likelihood is therefore a 1-dimensional problem for each feature.
Once the optimal λ ∗ is found, the transformed data Ψ( λ, x i ) is usually renormalized by subtracting its empirical mean µ ∗ and dividing by the square root of its empirical variance σ 2 ∗ .
Figure 1b shows an example of the YJ transformation applied to a skew-normal distribution.
Note that in a typical application, the triplet ( λ ∗ , µ ∗ , σ 2 ∗ ) is ﬁtted on the training data only, and is then used to Gaussianize the test dataset during inference.
Minimization methods in dimension 1 As seen above, ﬁtting a YJ transformation can be reduced to a 1D optimization problem.
To tackle this problem, we introduce two standard 1D minimization methods: (i) Brent minimization [6] and (ii) exponential search [3].
Brent minimization [6] (not to be confused with the Brent-Dekker method, see [6], chapters 3 and 4) is a widely used method for 1D optimization.
It is based on golden section search and successive parabolic interpolations, and does not require evaluating any derivatives.
This algorithm is guaranteed to converge to a local minimum with superlinear convergence of order at least 1.3247.
Standard implementations of the YJ transformation, in particular the scikit-learn implementation [36], are based on the Brent minimization method to minimize the negative log-likelihood provided by Eq.
Exponential search [3] is a dichotomic algorithm designed for unbounded search spaces.
The idea is to ﬁrst ﬁnd bounds, and then to perform a classic binary search within these bounds.
This algorithm can be used to ﬁnd the minimum of convex differentiable functions with linear convergence, as explained in Appendix B.
In this work, we build on exponential search to propose a federated version of the YJ transform, for two main reasons: (i) it is more numerically stable than Brent minimization, as shown in Section 3 and Figure 2, (ii), it may conveniently be adapted to a federated setting, as shown in Section 4, and (iii), this latter federated adaptation offers strong privacy garantees, as shown by Proposition 4.1.
Secure Multiparty Computation As illustrated by various privacy gradient attacks [56, 54], sensitive information on the clients’ datasets can be leaked to the central server during an FL training.
One way to mitigate this risk is to use Secure Multiparty Computation (SMC) protocols to hide individual contributions to the server.
SMC enables one to evaluate functions with inputs distributed across different users without revealing intermediate results and is often based on secret sharing.
SMC protocols tailored for ML use-cases have been recently proposed [12, 14, 34, 39, 48, 33, 49, 41].
These protocols are either designed to enhance the privacy of FL trainings, or to perform secure inference, i.e. to enable the evaluation of model trained privately on a server without revealing the data nor the model.
A popular FL algorithm relying on SMC is Secure Aggregation (SA) [4].
Schematically, in SA each client adds a random mask to their model update before sending it to the central server.
These masks have been tailored in such a way that they all together sum to zero.
Therefore, the central server cannot see the individual updates of the clients, but it can recover the sum of these updates by adding all the masked quantities sent from them.
More generally, an SMC routine schematically works as follows (we refer to Appendix D for further details).
Let us consider the setting where K parties k
= 1 , . .
, K want to compute g = f
( h (1) , . . .
, h ( K ) ) for a known function f , where ( h (1) , . . .
, h ( K ) ) denote private inputs.
Each party k knows h ( k ) and is not willing to share it.
During the ﬁrst step, secret sharing , each party splits its private input h ( k ) into K secret shares h ( k ) 1 , . .
, h ( k ) K , and sends the shares h ( k ) k (cid:48) to the party k (cid:48) .
These secret shares are constructed in such a way that (i) knowing h ( k ) k (cid:48) does not provide any information on the value of h ( k ) , and (ii) h ( k ) can be reconstructed from the vector ( h ( k ) 1 , . .
, h ( k ) K ) .
For simplicity, we denote (cid:74) h ( k ) (cid:75)
=
( h ( k ) 1 , . . .
, h ( k ) K ) the vector of share secrets.
In a second step, the computation , each party k (cid:48) computes the quantity denoted g k (cid:48) using the secret shares they know along with intermediate quantities exchanged with the other parties.
The way to compute g k (cid:48) depends on f and on the SMC protocol that is used, and is chosen so that g = f ( h (1) , . .
, h ( K ) ) can be reconstructed from ( g 1 , . . .
g K ) .
Said otherwise, g k (cid:48) are secret shares of g : (cid:74) g (cid:75)
=
( g 1 , . . .
g K ) .
Finally, during the reveal step, each party k reveals g k to all other parties, and each party can reconstruct g from ( g 1 , . . .
g K ) .
Threat model
In this work, we consider an honest-but-curious setting [35].
Neither the clients nor the server will deviate from the agreed protocol, but each party can potentially try to infer as much information as possible using data they see during the protocol.
This setting is relevant for cross-silo
FL, where participants are often large institutions whose reputation could be ternished by a more malicious behaviour.
A novel method to optimize the Yeo-Johnson log-likelihood: E XP YJ
Input: data x i , total data size n , number of steps t max
Initialize λ t =0 ← 0 , λ + t =0 ← ∞ , λ − t =0 ← −∞
Compute S ϕ for t = 1 to t max for g ∈ { Ψ( λ, · ) , Ψ( λ, · ) 2 , ∂ λ Ψ( λ, · ) , ∂ λ Ψ( λ, · ) 2 }
Compute S g end for ∆ t = sgn (cid:104) nS ∂ Ψ 2 − 2 S Ψ S
∂ Ψ − 2 S ϕ (cid:16) S Ψ 2 − S 2Ψ n (cid:17)(cid:105) λ t [Table] end for
[Table]
Output: The ﬁtted triplet [Table] λ ∗ , µ [Table]
[Table]
∗ )
In this section, we leverage the convexity of the negative log-likelihood of the
YJ transformation (see Proposition 3.1) to propose a new method to ﬁnd the optimal λ ∗ using exponential search.
While this method only offers linear convergence, compared to the super-linear convergence of Brent minimization method, we demonstrate two of its advantages: (i) it is more numerically stable, and (ii) it is easily amenable to an FL setting with strong privacy guarantees.
The method proposed in this section is based on the following result.
Proposition 3.1.
The negative loglikehood λ (cid:55)→ − log L YJ ( λ ) (2) is strictly convex.
The proof of Proposition 3.1 builds upon the work of [26] which shows that the negative log-likelihood of the Box-Cox transformation [5] is convex.
The complete proof is deferred to Appendix C.
The exponential YJ algorithm
The pseudo-code of the proposed algorithm is presented in Algorithm 1, and relies on the exponential search presented in Algorithm 2 (cf Appendix B for more details on exponential search).
An illustration of E XP YJ is shown in Figure 6 in Appendix B. Due to the strict convexity of the negative log-likelihood of the YJ transformation, we may perform the exponential search described in Section 2 and Appendix B.
To do so, it is enough to obtain the sign of the derivative.
Let ∂ λ Ψ( λ, · ) 2 = 2Ψ( λ, · )
∂ λ Ψ( λ, · ) and ϕ ( x ) = sgn
( x ) + log( | x | + 1) .
Further, for g ∈ { Ψ( λ, · ) , ∂ λ Ψ( λ, · ) , Ψ( λ, · ) 2 , ∂ λ Ψ( λ, · ) 2 , ϕ } , let us deﬁne S g def = (cid:80) ni
=1 g ( x i ) .
The derivative of the log-likelihood is available in closed form (see Appendix A.3):
[Equation]
Notice that S Ψ 2 − S 2Ψ /n can be expressed as a variance, hence is non-negative.
We may therefore obtain sgn [ ∂ λ log L YJ ] while avoiding performing division by computing
[Equation]
Avoiding this division is crucial to make the overall procedure more numerical stable, as explained below, and eases the use of SMC routines.
[Figure]
Algorithm
Input: λ , λ + , λ − , ∆ ∈ {− 1 , 1 } if ∆ = 1 then λ − ← λ λ ← ( λ + + λ ) / 2 if λ + < ∞ else λ ← max(2 λ, 1) else λ + ← λ λ ← ( λ − + λ ) / 2 if λ − >
−∞ else λ ← min(2 λ, − 1) end if
Output: Updated λ, λ
+ , λ −
Accuracy of E XP YJ
We check the accuracy of E XP YJ on the datasets presented in Appendix E.1.
In particular, we compare the results provided by E XP YJ with the outputs of the scikitlearn algorithm based on Brent minimization.
For 2 of the 108 features present in the datasets, the scikitlearn implementation leads to numerical instabilities discussed hereafter.
Therefore, we focus our comparison on theremaining features, that we aggregated regardless of the dataset.
Figure 2 reports the relative difference δλ between the results obtained by E XP YJ and by the scikit-learn implementation as a function of the number of iteration t max (as deﬁned in
Algorithm 1).
These results show that this relative difference
Numerical stability of E XP YJ
Our experiments demonstrate that E XP YJ is numerically more stable than Brent minimization.
Indeed, for some values of λ and some datasets { x i } , the transformation Ψ( λ, · ) concentrates all data points in a small interval such that the values of Ψ( λ, { x i } ) are all equal up to machine precision.
In that case, the log-likelihood is not well-deﬁned and the term log σ 2Ψ λ takes the value −∞ , which prevents Brent minimization from converging.
This phenomenon does not affect the E XP YJ routine as we do not compute directly the sign of ∂ λ L = ∂ λ
σ 2Ψ λ /σ
2Ψ λ − (cid:80) i ϕ ( x i ) , but rather the sign of σ 2Ψ λ
∂
λ L = − ∂ λ
σ 2Ψ λ − σ 2Ψ λ (cid:80) i ϕ ( x i ) , see Eq.
Figure 2 illustrates this in the case of a feature of the Breast Cancer Dataset .
The λ ∗ returned by the Brent minimization method of scikit-learn is − 14 .
53 while the minimizer of the negative log-likelihood found by the E XP YJ is − 0 .
In particular, Figure 2 shows the values of the negative log-likelihood as a function of λ computed using 64-bit ﬂoat precision.
The orange vertical full bands correspond to values for which σ 2Ψ λ is zero within the machine precision, resulting to a negative log-likelihood of ∞ .
This instability happens for 2 of the 108 features used in numerical experiments, where blindly applying the Brent-based YJ transformation leads to all data points collapsing to zero, while E XP YJ succeeds in transforming the data distributions to more Gaussian-like ones.
Appendix E.4 shows that this issue also arises in other real-life datasets.
Applying the Yeo-Johnson transformation in FL
So far, we only considered the centralized setting, where data is accessible from a single server.
Yet, as mentioned in Sections 1 and 2, many real-world situations require working with heterogeneous data split between different centers, and to take privacy constraints into account.
When the data is split across centers k = 1 , . .
, K and the function to optimize is separable, i.e. of the form
F ( λ ) =
(cid:80)
Kk =
1 f k ( λ ) where each f k can be computed from data present in the center k exclusively,
Federated Learning techniques were recently proposed.
In short, they consist in repeatedly performing a few rounds of local optimization in each center, before aggregating local parameters in the server.
We refer to [23] for an overview of recent advances in FL.
In our case, however, the YJ negative log-likelihood (2) is not separable, due to the log-variance term.
Indeed, turning the variance into a separable term would require sharing the global YJ mean µ λ to all centers at each iteration.
Compared to the method we propose in this section, this would lead to more privacy leakage.
We now introduce S ECURE F ED YJ, a secure federated algorithm that builds upon E XP YJ to apply YJ transformations.
This algorithm satisﬁes the two following properties: (i) it is pooled-equivalent , i.e. it yields the same results as if the data were freely accessible from a single server, and (ii) it leaks as little information as possible about the underlying datasets, as shown by Proposition 4.1.
Finally, it converges in a limited number of iterations, thanks to the linear convergence of the underlying exponential search.
[Equation] S ECURE F ED YJ is a federated adaptation of E XP YJ presented in Section 3 to ﬁnd the best parameters ( λ ∗ , µ ∗ , σ 2 ∗ ) of the YJ transformation when training datasets are split across different clients.
It relies on SMC to ensure that only the ﬁnal triplet ( λ ∗ , µ ∗ , σ 2 ∗ ) ﬁtted on the training datasets is revealed, without leaking any other information apart from the overall total number of training samples n .
Indeed, at each intermediate step, only the sign of ∂ λ log L YJ is revealed, and the mean and variance of the transformed data is only revealed at the last step.
The pseudo-code of the resulting algorithm is presented in Algorithm 3, and relies on the exponential search presented in
Algorithm 2.
A functional representation of S ECURE F ED YJ is displayed Figure 7.
In Algorithm 3,we label the clients by k = 1 , . . .
, K and each client k holds data { x k,i : i = 1 , . . . , n k } .
We suppose that the total number of samples n = (cid:80) Kk =1 n k is public and shared to all clients.
For a given function g , we denote S k,g the sum S k,g def = (cid:80) n k i =1 g ( x k,i ) .
As introduced in Section 2, we use double brackets (cid:74) · (cid:75) to indicate an SMC secret shared across the clients (see
Appendix D for more details).
Privacy leakage In Proposition 4.1 we show that Algorithm 3 only reveals information already contained in the ﬁtted triplet ( λ ∗ , µ ∗ , σ 2 ∗ ) .
In comparison, turning the YJ negative log-likelihood (2) into a log-separable function before using off-the-shelf FL methods would require sharing µ and its gradient and centrally computing σ 2 for intermediate values of λ at each iteration.
This could potentially lead to uncontrolled privacy leakage.
[-Indeed, ∂ λ log L YJ < 0 for λ > λ ∗ and ∂ λ log L YJ > 
-]
Input: Data { x k,i } , total data size n , number of steps t max
Notations: (cid:74) · (cid:75) indicates a SMC secret shared across the clients.
Any operation such as (cid:74) · (cid:75) =
f ( (cid:74) · (cid:75) , (cid:74) · (cid:75) , · · · ) where f can be the sum, product, or the sign, designs an SMC routine across the clients as described in Appendix D.5.
Initialize λ t =0 ← 0 , λ + ← ∞ , λ − ←
−∞ independently on each client
Clients compute in SMC (cid:74) S ϕ (cid:75)
=
(cid:80) k
(cid:74)
S k,ϕ (cid:75) for t = 1 to t max for g ∈ { Ψ( λ, · ) , Ψ( λ, · ) 2 , ∂ λ Ψ( λ, · ) , ∂ Ψ( λ, · ) 2 }
Clients compute in SMC (cid:74) S g (cid:75) =
(cid:80) k
(cid:74) S k,g (cid:75) , end for
Clients compute in SMC (cid:74) ∆ t (cid:75) =
sgn [ n (cid:74) S
∂ Ψ 2 (cid:75) − 2 (cid:
) S Ψ (cid:75)(cid:74) S
∂ Ψ (cid:75) − 2 (cid:74) S ϕ (cid:75)
( (cid:74) S Ψ 2 (cid:75) − (cid:74) S Ψ (cid:75) 2 /n )
(cid:3) Clients reveal ∆ t λ t , λ − t , λ + t ← E XP U PDATE ( λ t − 1 , λ − t − 1 , λ + t − 1 , ∆ t ) independently on each client end for λ ∗ ← λ t max Clients compute in SMC (cid:74) µ (cid:75) =
(cid:80) k
(cid:74) S k, Ψ (cid:75) /n
and (cid:74) σ 2 (cid:75) =
(cid:80) k
(cid:74)
S k, Ψ 2 (cid:75) /n
− (cid:74) µ 2 (cid:75)
Clients reveal µ ∗ ← µ and σ 2 ∗ ← σ
Output: The ﬁtted triplet ( λ ∗ , µ ∗ , σ 2 ∗ )
[Figure]
Proposition 4.1.
The ﬁtted parameter λ ∗ contains all the information revealed during the intermediate steps of S ECURE F ED YJ .
More precisely, there exists a deterministic function F such that for any set of datasets { x k,i } , if λ ∗ ( { x k,i } ) is the result of S ECURE F ED YJ on { x k,i } , then { λ t , λ + t , λ − t , ∆ t } t =1 , ··· ,t max = F [ λ ∗ ( { x k,i } )] Proof. This proposition comes from the fact that all gradient signs ∆ t revealed during the algorithm can be retrospectively inferred from λ ∗ . Indeed, ∂ λ log L YJ < 0 for λ > λ ∗ and ∂ λ log L YJ >for λ < λ ∗ .
Besides, the successive values of λ t explored at each step t can be deterministically inferred from the initial value λ t =
and and the ﬁnal ﬁtted value λ ∗ .
We construct such a function F and numerical verify this proposition in Appendix F. (cid:4)
Performance of S ECURE F ED YJ
We implement S ECURE F ED YJ in Python, using the MPyC library [42] based on Shamir Secret Sharing [44].
We refer to Appendix D for more details on our implementation.
To represent signed real-valued numbers in an SMC protocol, we use a ﬁxed-point representation (see Appendix D.2) using l bits, among which f bits are used for the decimal parts.
This means that we consider ﬂoats ranging from − 2 l − f to 2 l − f and that we have an absolution precision of 2 − f in our computations.
In order to ensure the accuracy of S ECURE F ED YJ results, we need to make sure that l and f are large enough.
Figure 3 shows the accuracy of S ECURE F ED YJ when compared to E XP YJ for various values of f .
According to these numerical experiments, taking f = 50 and l = 100 provides reasonably accurate results.
Moreover, by construction, the outputs of S ECURE F ED YJ do not depend on how the data is split across the clients, up to rounding numerical errors.
Therefore this algorithm is resilient to data heterogeneity, as long as the numerical decimal precision f is large enough, as shown in
Figure 3.
Performing S ECURE F ED YJ with t max = 40 takes 726 rounds of communication (see Appendix D.6).
During these communication rounds, each client sends overall about 8 Mb per feature to every other client (see also Appendix D.6).
S ECURE F ED YJ can be applied independently and in parallel to each feature.
Therefore, the overall number of rounds does not depend on the number of features being considered, and the communication costs grow proportionally to the number of features.
In a realistic cross-silo FL setting as described in [19], the bandwidth of the network is 1 Gb per second with a delay of 20 ms between every two clients.
In this context, the execution of S ECURE F ED YJ with t max =on p features would take about 726 × 20 ms (cid:39) 15 s due to the communication overhead, in addition to p × 8 Mb / 1 Gbps (cid:39) 8 p ms due to the bandwidth.
This shows that S ECURE F ED YJ is indeed a viable algorithm in a real-world scenario.
As pointed out in Appendix B, the binary search in the exponential search can be replaced by a k -ary search.
In such a setting, the sign of the negative log-likelihood of the YJ transformation is computed for k − 1 different values of λ at each round.
Such a modiﬁcation would reduce the number of communication rounds required to obtain a given accuracy, while increasing the size of the data exchanged over the network at each round.
Applications
[Figure]
Genomic data: TCGA We start by showing the beneﬁts of YJ preprocessing in survival analysis experiments on lung (LUAD+LUSC), pancreas (PAAD), and colorectal (CRC) cancers.
Given gene expression raw counts (features) and censored survival data (responses) from patients having either of those three cancers, we aim to ﬁt a Cox Proportional Hazards (CoxPH) model [10] with the highest possible concordance index (C-index) [25], which measures how well patients are ranked with respect to their survival times.
We refer to [25] for a more thorough introduction to survival analysis.
In Figure 4, we compare three different preprocessing methods: (i) whitening, (ii) log normalization, and (iii)
YJ, each followed by a PCA dimensionality reduction step.
More precisely, whitening (i) consists in centering and reducing to unit variance the total read counts of all genes across all samples, log normalization consists in applying u (cid:55)→ log(1 + u ) to raw read counts before applying global whitening, and YJ is a global YJ transform on the total read counts.
We then evaluate each strategy using 5-fold cross-validation with 5 different seeds.
We refer to Appendix E.2 for experimental details.
While this experiment is performed in a pooled environment, note that, importantly, each step has a federated pooled-equivalent version: apart from the proposed S ECURE F ED YJ for YJ, see e.g. [21] for PCA, and Webdisco [31] for Cox model ﬁtting.
This simpliﬁed setting allows us to understand the importance of the Yeo-Johnson transformation in an ideal setting, independently of other potential downstream federated learning artifacts.
In Figure 4, we see that YJ is better or on par with the best method for each cancer:
YJ improves prediction results for colorectal cancer, while yielding results which are on par with the best results for lung and pancreas cancers, with a smaller variance.
[Figure]
Synthetic data We show how applying YJ may help improving performance compared to no or basic preprocessing, and how S ECURE F ED YJ yields improvements compared to local YJ transforms in federated linear regression.
To do so, we generate covariates (cid:101) X and responses y as (cid:101)
X
= (exp( x 1 ) , exp( x 2 + 2) , σ ( x 3 )) with X = ( x 1 , x 2 , x 3 ) ∼ N (0 , I 3 ) , y = β T X + ε with ε ∼ N (0 , 0 . 1) , (4) where σ ( · ) is the sigmoid function and β = ( − 1 . 3 , 2 . 4 , 0 . 87) was randomly chosen.
The goal is then to ﬁt a linear model from i.i.d.
samples ( (cid:101)
X
i , y i ) , i = 1 , . .
. , n following (4), after an optional preprocessing step.
Similarly to the previous example, we simulate a cross-device FL setting only for the preprocessing steps, and the linear model is then ﬁtted in a pooled setting for simplicity.
All the details of the numerical experiments are provided in Appendix E.3.
We suppose that the samples ( (cid:101) X i , y i ) , i = 1 , . .
, n are homogeneously split across 10 centers.
The responses y
i have a highly nonlinear dependency on the covariates (cid:101)
X i , but depend linearly on the X
i ’s (up to Gaussian noise), which are not observed.
Hence, we expect that applying a suitable preprocessing step before training a linear model will transform back the (cid:101)
X i ’s into the normally-distributed X
i ’s and lead to a high performance, compared to no transformation.
The results of our experiments are summarized in Figure 5.
The left ﬁgure shows that the YJ transformation is indeed capable of roughly inverting the (cid:101)
X i ’s into the X
i ’s, yielding a major improvement compared to no preprocessing or standard centering and reduction to unit variance.
Besides, the right ﬁgure shows that even in this homogeneous setting where the data is i.i.d.
across centers, using a federated version of YJ compared to a local version of YJ leads to better average performance, and reduced variance.
Conclusion
Summary of our contributions In this work, we introduce S ECURE F ED YJ, a method to ﬁt a
YJ transformation on data shared by different clients in a cross-silo setting.
S ECURE F ED YJ is an SMC version of its pooled equivalent E XP YJ which builds upon the convexity of the negative log-likelihood of the YJ transformation, a novel result introduced by this work, and on the fact that the sign of its derivative can be computed in a stable way.
We show that S ECURE F ED YJ has the same accuracy as a standard YJ transformation on pooled data.
In particular, the results do not depend on how the data is split across the clients, making S ECURE F ED YJ resilient to data heterogeneity.
Besides, the quantities disclosed by S ECURE F ED YJ during the training to the central server do not leak any other information than what is contained in the ﬁnal parameters ( µ ∗ , λ ∗ , σ 2 ∗ ) .
Limitations and future work While Brent minimization has a super-linear convergence, our approach only has a linear convergence, as it relies on exponential search.
This can be an issue if the communication costs between the clients and the server are high.
Acceleration could be achieved by either adapting Brent minimization to a cross-silo setting, or applying a second-order method.
We leave the development of a faster SMC methods using either of those two approaches to future work.
Another limitation is that even if our approach reveals only information that would be contained in the ﬁnal ﬁtted parameters, such parameters themselves might leak information about individual samples, as our approach is not differentially private (DP) [16].
By adding Gaussian or Laplacian noise to each sample’s features when computing the S g terms one could, in principle, make the resulting algorithm
DP [1].
However it is unclear to what extent the noise would impact the ﬁnal accuracy of the method.
Finally, we only consider an honest-but-curious setting.
We do not explore the threat of a malicious participant that would purposely deviate from the protocol to either gain more information or to jeopardize the convergence.
We leave this investigation to future work.
Acknowledgement
The authors would like to thank the four anonymous reviewers, as well as the anonymous area chair reviewer for their relevant comments and ideas which signiﬁcantly improved the paper.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We described them in Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The full proof of the main theoretical results, i
e. the convexity of the Yeo-Johnson negative log-likelihood (Proposition 3.1) is provided in Appendix C. 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code of the experiments is not provided, but a detailed pseudo-code of the newly proposed algorithms are provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We speciﬁed all the hyperparameters and the details of the numerical experiment in Appendix E. (c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviations or quantiles of the results with respect to the seed are provided (cf plots)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] The experiment are not heavy and run easily on a personal computer, on a CPU 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets used are open datasets available online and are systematically cited.
(b) Did you mention the license of the assets?
[Yes] The licence of the datasets used are provided in Appendix E. (c)
Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] We are using open datasets available online.
The genomic dataset from TCGA have been previously anonymised by its creator before publication (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional properties of the Yeo-Johnson transformation A.1 Derivation of the Yeo-Johnson log-likelihood
Using the change of variables rule, the probability to draw a set of points { x i } such that { Ψ( λ, x i ) } follows a Gaussian distribution of mean µ and variance σ 2 is given by:
[Equation]
where det J [ x i , Ψ( λ, x i )] is the determinant of the Jacobian matrix J [ { x i } , { Ψ( λ, x i ) } ] deﬁned as:
[Equation]
This matrix is diagonal and each term of its diagonal can be computed using Eq.
For each value of λ and x i , these diagonal terms can be re-written as exp[( λ − 1) sgn
( x i ) log( | x i | + 1)] .
The term P ( { Ψ λ ( λ, x i ) }| λ, µ, σ ) is equal to:
[Equation]
By taking the logarithm of Eq.
(5), we obtain the log-likelihood provided in Section 1 and originally derived on [52].
A.2 Relationship with the Box-Cox transformation
The Box-Cox transformation [5] works similarly to the YJ transformation, but only applies to strictly positive data.
The Box-Cox transformation is based on a function Φ( λ, · ) parametrized by λ and deﬁned for x > 0 as:
[Equation]
It is straightforward to check that for any λ ∈ R the YJ transformation Ψ and the Box-Cox transformation Φ are related by the following equations:
[Equation]
A.3 Analytical formulae for the derivatives of the Yeo-Johnson transformation
The YJ function is inﬁnitely differentiable with respect to both of its variables ( x and λ ).
Here are its successive derivatives with respect to λ :
[Equation]
B Background on exponential search
Exponential search [3] is a method to look for an element in an unbounded sorted array.
The idea is to ﬁrst ﬁnd bounds on the array such that the element is contained within such bounds, and then perform a classic binary search inside these bounds.
Let us consider the task of ﬁnding the smallest element u i {+
+} greater than a threshold C in an unbounded sorted array { u i } i ∈ N ∗ .
The exponential search iteratively looks at u i for i ∈ { 1 , 2 , 2 2 , 2 4 , . . .
} until it ﬁnds a i max such that u i max ≥ C .
This takes log 2 ( i max ) steps.
Then it performs a binary search between i = i max /
and i =
i max , which also takes log 2 ( i max / 2) steps.
If f ( s ) is a strictly increasing function of s taking both positive and negative values, one can adapt the exponential search to ﬁnd the root s 0 of f .
The ﬁrst step is to ﬁnd an upper and a lower bound of s 0 by evaluating f at different points using an exponential grid (e.g. evaluating f in s = 1 , − 1 , 2 , − 2 , 2 2 , − 2 2 , 2 4 , − 2 4 , . . . ).
Once such bounds are found, one can perform a dichotomic search inside these bounds to ﬁnd the root s 0 of f .
This dichotomic search has a linear convergence of order 2 , with each step summarized in Algorithm 2.
It is important to note that this algorithm is correct even if f is not increasing, as long as f ( s ) <
when s < s 0 and f ( s ) > 0 when s > s 0 , as is the case in this work when f is the derivative of the negative YJ log-likelihood.
Figure 6 is an illustration of E XP YJ that is based on exponential search.
[Figure]
A natural extension of the exponential search is to replace the binary search into a k -ary search during the dichotomic search.
In that case, k − 1 values of f are computed at each round.
Such a modiﬁcation reduces the number of steps required for a given accuracy while increasing the number of operations performed at each step.
C Proof of Proposition 3.
We ﬁrst introduce some lemmas that will be required for the main proof.
Lemma C.1.
Let λ (cid:55)→ f i ( λ ) , i = 1 , . .
, I be positive and twice differentiable functions, such that for all i , λ
(cid:55)→ ln[ f i ( λ )] is convex.
Then λ (cid:55)→ ln[ (cid:80)
i f i ( λ )] is also convex.
Proof.
The proof of Lemma C.1 is based on the following lemma:
Lemma C.2.
Let { a i } i =
I , { b i } i =1 ...I , ( c i ) i =1 ...
I be real numbers, such that for all 1 ≤ i ≤ I : a i ≥ 0 , b i ≥ 0 and a i b i ≥ c 2 i .
Then
[Equation]
Indeed, it holds,
[Equation]
[-= (cid:32) I (cid:88) i = a i b i − c 
 i (cid:33) +   I (cid:88) i = (cid:88) i<j ≤ I a i b j + a j b i −-]
[Equation]
The ﬁrst sum contains only non-negative terms as ∀ i, a i b i ≥ c 2 i .
Recalling that a i ≥ 0 and b i ≥ 0 , the second sum also contains non-negative terms
[Table]
Now let us prove Lemma C.1.
The convexity and twice differentiability of ln[ f i ( λ )] implies that ∂ 2 λ ln[ f i ( λ )] ≥ 0 and therefore that
[Equation]
As f i > 0 , we can conclude from Eq. {+
+}
that ∂ 2 λ f i ≥ 0 .
Using Lemma C.2 and the linearity of the derivative, we have:
[Equation]
which means that ∂ 2 λ ln( (cid:80) i f i ) ≥ 0 .
Lemma C.3.
Let { α i } i =1 ,...,n α and { β i } i =1 ,...n β be two non-empty sets of real numbers, and let us denote { γ i } i = { α 1 , . . .
, α n α , β 1 , . . . , β n β } and n γ = n α + n β .
Let ¯ α = 1 n α (cid:80) i
α i , ¯ β = 1 n β (cid:80) i
β i , ¯ γ = 1 n γ (cid:80)
i γ i and σ
α = 1 n α (cid:80) i ( α i − ¯ α ) 2 , σ 2 β = 1 n β (cid:80) i ( β i − ¯ β ) 2 , σ 2 γ = 1 n γ (cid:80) i ( γ i − ¯ γ ) 2 .
Then:
[Equation]
Proof.
This identity is easily obtained using the deﬁnitions of σ 2 α , σ 2 β and σ 2 γ .
C.1 Proof of Proposition 3.
Proof.
We start by proving the only shows the convexity of − log L YJ ( λ ) , and prove strict convexity in Appendix C.4.
Let { x i } i =1 ··· n be our data points and let us split this dataset into non-negative values { x + i } = { x i | x i ≥ 0 } and negative values { x − i } = { x i | x i < 0 } .
Let γ i
= Ψ( λ, x i ) , α i =
Ψ( λ, x + i ) , and β i =
Ψ( λ, x − i ) .
We denote n α , n β , n γ the lengths of the sets { α i } , { β i } and { γ i } .
For clarity, let us consider the case where both { x + i } and { x − i } have at least two distinct items and therefore n α ≥ 1 , n β ≥ 1 and σ 2 α > 0 , σ 2 β > 0 .
We relegate to Appendix C.3 the other edge cases.
According to Lemma C.3, the expression of negative log-likelihood of the YJ transformation provided in Eq.
(2) can be reformulated as:
[Equation]
The ﬁrst term is constant and the second one is linear in λ
so we only have to prove the convexity of the last term to prove that the full negative log-likelihood is convex.
Using Lemma C.1, we only need to show that λ (cid:55)→ ln (cid:16) n α n γ σ 2 α (cid:17) , λ (cid:55)→ ln (cid:16) n β n γ σ 2 β (cid:17) and λ (cid:55)→ ln (cid:16) n α n β n 2 γ (¯ α − ¯ β ) 2 (cid:17) are convex.
We can get rid of the constant factor and show that λ (cid:55)→ ln (cid:0) σ 2 α (cid:1) , λ
(cid:55)→ ln (cid:16) σ 2 β (cid:17) and λ (cid:55)→ ln (cid:0) (¯ α − ¯ β ) 2 (cid:1) are convex.
The key idea of the proof is to use the fact that, according to [26], for any set of positive real numbers { a i } , λ
(cid:55)→ ln σ [Φ( λ, { a i } ] is convex, where Φ( λ, · ) denotes the Box-Cox transformation.
Besides we have (see Appendix A.2):
[Equation]
Therefore ln σ [ α i ]
=
ln
σ [Φ( λ, ( x + i + 1)] which is a convex function of λ .
Similarly, σ [ {− Φ(2 − λ, 1 − x − i ) } ] 2 = σ [ { Φ(2 − λ, 1 − x − i ) } ] 2 .
The function λ (cid:55)→ σ [ { Φ(2 − λ, 1 − x − i ) } ] 2 is convex as the composition of the linear function λ (cid:55)→ 2 − λ with the convex function λ
(cid:55)→ σ [ { Φ( λ, 1 − x − i ) } ] 2 .
Let us ﬁnally prove the convexity of λ (cid:55)→ ln (cid:2) (¯ α − ¯ β ) 2 (cid:3) .
We recall that ¯ α > 0 and ¯ β <and that ln (cid:2) (¯ α − ¯ β ) 2 (cid:3) = 2 ln (cid:2)
¯ α − ¯ β (cid:3) .
Using Lemma C.1, we only need to prove that λ
(cid:55)→ ln (¯ α ) and λ (cid:55)→ ln (cid:0)
− ¯ β (cid:1) are convex.
As ¯ α and ¯ β are deﬁned as sums, still using Lemma C.1, we only need to prove that λ
(cid:55)→ ln (cid:0) Ψ( λ, x + i ) (cid:1) and λ
(cid:55)→ ln (cid:0) − Ψ
( λ, x − i ) (cid:1) are convex for any i .
Using, Eqs.
(8) and (9), it is sufﬁcient to prove that for any real number a ≥ 1 , the function λ (cid:55)→ ln[Φ( λ, ( a )]
= ln
[( a λ − 1) /λ ] is convex, which is proved in Appendix C.2.
C.2 Proof that λ (cid:55)→ ln[Φ( λ, ( a )] is convex
Let a ≥ 1 u ( λ ) =
( a λ − 1) /λ
and g ( λ ) =
ln u ( λ ) .
For λ (cid:54) = 0 , the second derivative of g is positive if and only if D def = λ 4 ( uu (cid:48)(cid:48) − ( u (cid:48) ) 2 ) ≥ 0 .
We have
[Equation]
Let us show that D ≥ 0 when λ (cid:54) = 0 .
D ( a = 1 , λ ) = 0 , so we just need to show that ∂ a D ( a, λ ) >when a > 0 .
As
[Equation]
let us deﬁne T ( a, λ ) as:
[Equation]
We just need to show that T ( a, λ ) >
when λ > 0 and T ( a, λ ) < 0 when λ < 0 .
As T ( a, 0) =
0 , we just need to show that ∂ λ T ( a, λ ) > 0 when λ (cid:54) = 0 .
[Equation]
As a > 1 , log( a ) > 0 , so we just need to show that ( a λ − λ log( a ) − 1) > 0 which can be done by replacing x by λ log( a ) in the following inequality: exp( x ) > x + 1 for x > 0 .
To conclude, when λ (cid:54) = 0 and a ≥ 1 , D ( λ, a ) ≥ 0 , and if a > 1 then D ( λ, a ) > 0 .
Therefore, the second derivative of g is positive for any λ ≥ 0 .
Using continuity, we can conclude that the second derivative of g is positive for any λ and that λ (cid:55)→ ln[Φ( λ, ( a )] is convex.
Note that if a > 1 , then D >
and we can conclude that λ
(cid:55)→ ln[Φ( λ, ( a )] is strictly convex.
C.3 Edge cases not covered by the main proof of Proposition 3.
In the main proof we assume that n α ≥ 2 , n β ≥ 2 and that σ 2 α > 0 , σ 2 β > 0 .
Said otherwise, we assume that both { x + i } and { x − i } have at least two distinct elements.
The proof is almost unchanged if this is not the case, as we can discard any term inside the logarithm of Eq.
For example, let’s assume that n α = 1 .
Therefore σ 2 α = 0 .
We can then rewrite Eq.
7 as:
[Equation]
We only need to show that λ (cid:55)→ ln (cid:16) n β n γ σ 2 β (cid:17) and λ (cid:55)→ ln (cid:16) n α n β n 2 γ
(¯ α − ¯ β ) 2 (cid:17) are convex as in the main proof.
Any other edge case can be treated similarly, and the proof holds as soon as { x i } has at least two distinct elements.
C.4 Strict convexity of the Yeo-Johnson negative log-likelihood.
To prove the strict convexity of the YJ negative log-likelihood, let us notice that under the hypotheses of Lemma C.1, if at least one function λ (cid:55)→ ln
( f i ) is strictly convex, then λ
(cid:55)→ ln[ (cid:80) i f i ( λ )] is strictly convex.
Besides, according to [26], for any set of positive real { a i } with at least two distinct elements, λ (cid:55)→ ln
σ [Φ( λ, ( a i )] is strictly convex.
Therefore, in the case where either { x + i } or { x − i } has two distinct elements, we can conclude that the YJ negative log-likelihood is strictly convex.
The only problematic case is when both σ 2 α = 0 and σ 2 β = 0 .
In that case { x i } has only two distinct element: one positive or null and one strictly negative.
In that case, λ (cid:55)→ σ [ { Φ( λ, 1 − x − i ) } ] 2 is strictly convex as λ (cid:55)→ ln[Φ( λ, ( a )]
= ln
[( a λ − 1) /λ ] is strictly convex for a > 1 .
D Secure Multi-Party Computation D.1 Shamir Secret Sharing
Secure Multiparty Computation (SMC) consists in evaluating functions without disclosing their inputs.
One way to achieve this result is to use secret sharing.
The main idea is that a value h is split into different secret shares h k , k = 1 , · · · , K where K is the number of clients.
Each client k only knows the value of the secret share h k , and one needs at least p shares with 1 < p ≤ K to recover the initial value h .
The set of the secret shares h k of h is denoted (cid:74) h (cid:75) .
Schematically, SMC consists in three main steps: (i) secret sharing , where each client splits its input into secret shares and sends them to the other clients (ii) computation , where the clients perform mathematical computations on the secret shares and obtain secret shares of the output and (iii) reveal steps, where the clients send each other the secret shares of the output in order to reconstruct and reveal the output.
In the Shamir Secret Sharing method [44], the secret shares of h correspond to the values of a given polynomial P h ( x ) of order K at different points x k where P h (0) = h .
The values x k are arbitrarly chosen by the protocol with the constraint that all x k should be distinct.
If all the clients disclose their secret share h k
= P h ( x k ) , then the secret h can be recovered by polynomial interpolation.
In this framework the addition can be done trivially.
If (cid:74) h (cid:75) =
{ h k } k =1 , ··· K and (cid:74) g (cid:75) =
{ g k } k =1 , ··· K are the shares of g , then (cid:74) g
+ h (cid:75) =
{ g k + h k } k =1 , ··· K are shares of g + h .
Said otherwise, (cid:74) g
+ h (cid:75) =
(cid:74) g (cid:75)
+ (cid:74) h
(cid:75) .
Therefore adding two shared secrets requires no communication between the clients.
Similarly, multiplying a shared secret by a public constant c is done without communication as (cid:74) cg (cid:75) =
c (cid:74) g (cid:75) .
However, multiplying two shared secrets, i.e. computing shares of (cid:74) gh (cid:75) is more involved and requires one round of communication.
More precisely, each client has to send one scalar quantity to all the other clients during this process, as explained for example in [37], section 3.
D.2 Fixed-Point Representation
The secret shares in SMC belong to a ﬁnite set Z p where p is a prime number and all the operations are integer operations done modulo p .
In practice we consider integers encoded using l bits, then we choose the smallest prime number p such that 2 l < p
and we perform each operation modulo p .
Therefore any value has to be encoded as an integer using a ﬁnite number of bits.
To encode negative integers, we consider that encoded integers between 0 and 2 l − 1 − 1 are positive and encoded integers between 2 l − 1 and 2 l − 1 are negative.
We have to choose a value of l large enough such that the highest absolute value considered is below 2 l − 1 .
Real-value numbers are encoded using ﬁxed-point precision, as described in [8], where the f least signiﬁcant bits of the encoding correspond to the decimal part, and the l − f most signiﬁcant bits correspond to the integer part.
The addition of two ﬁxed-point numbers in SMC can be done as described in Appendix D.1.
However, multiplying two ﬁxed-point representation numbers in SMC is more complex as the result must be divided by 2 f , i.e. the 2 f least signiﬁcant bits are discarded.
As explained in detail in [8], multiplying two ﬁxed-point numbers requires two rounds of communication (instead of one round of communication for the multiplication of two integers).
D.3 Comparison in SMC
In S ECURE F ED YJ , we need to compute in SMC the sign of an expression, which is equivalent to making a comparison with 0 .
As we are using ﬁxed-point representation encoding, computing the sign amounts to computing the most signiﬁcant bit of the binary decomposition of a given shared secret.
In order to do so, we use the method described in [38], which works for any SMC framework supporting addition and multiplication.
This method requires 10 rounds of communication among the clients ( 6 of which can be done ofﬂine, i.e. they correspond to random values exchanged
Présentation sans titre - Google Slides https://docs.google.com/presentation/d/1c5BcW8Cjeh2hjziLyG-AFFalTEiC4V8JbUg1JWsOJ98/edit#slide=id.g1274125999d_0_ 1/
Présentation sans titre Fichier Édition Affichage Insertion Format Diapositive Réorganiser Outils Modules complémentaires Aide Dernière modication il y a 7 minutes Arrière-plan Mise en page Thème Transition 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24135791113① t t t t t t … Δ t Δ t Δ t Δ t … 𝛌 ( Δ , 𝛌 ) t t t+1 ClientK ① ①
SECURE COMPUTATION ② ③ ③ ③ 𝛌 ( Δ , 𝛌 ) t t t+1 Client 𝛌 ( Δ , 𝛌 ) t t t+1 Client
Cliquezpourajouterlescommentairesduprésentateur
Présentation sans titre
Diaporama
Partager beforehand and can be done regardless of the value of the input). During these 10 rounds of communication, 153 l + 423 log l + 24 multiplications are performed, 135 l + 423 log l + 16 of which can also be done ofﬂine. Notice that other SMC primitives could be used, such as the one described in [17] which provides more efﬁcient way to do SMC comparison.
D.4 MPyC
To implement S ECURE F ED YJ we used the python library MPyC [42].
MPyC is based built upon
VIFF framework [11] and is based on Shamir Secret Sharing [44].
We refer to [30] for a discussion of the performance of this library for various SMC tasks.
D.5 Further details on Algorithm
The pseudo-code provided in Algorithm 3, is a schematic overview of S ECURE F ED YJand relies on the SMC routines described above.
For example, the following line of the pseudo-code:
[Equation]
implies that: (i) each client k computes S k,ϕ , divide it into secrets and send these share secrets to all the other clients; (ii) Using the SMC routines described in Appendices D.1, D.2 and D.4, the clients compute together the share secrets of (cid:74) S ϕ (cid:75) where S ϕ
=
(cid:80) k S k,ϕ .
After this step in Algorithm 3, the value of S ϕ is therefore shared using share secrets across all the clients.
Notice that the server only plays an orchestration roles in this process.
D.6 Complexity of S ECURE F ED YJ
At each step of the exponential search, we share 6 secrets (the values of S g ), perform 10 ﬁxed-point multiplications (including multiplying and dividing by n ), and one comparison (i.e. computing the sign of ∂ λ L YJ .
The 6 secrets can be shared in parallel in one round of communication.
Some of the multiplications can also be done in parallel, and only 3 successive rounds of multiplications have to be performed, which require 6 rounds of communications.
As stated in Appendix D.3, the comparison requiresrounds of communications.
Revealing the secret ∆ also requires one round of communication.
Notice that the additions do not require any round of communication.
This amounts to 18 communications per exponential search step.
Besides, computing (cid:74) S φ (cid:75) at the beginning of the algorithm and computing and revealing µ ∗ and σ 2 ∗ at the end of the algorithm requires 6 more rounds of communication.
Overall, performing 40 steps of exponential search with S ECURE F ED YJ costs 18 × 40 + 6 =rounds of communications.
For each elementary operation, such as sharing a secret, revealing a secret or making a multiplication, the order of magnitude of the size of the message sent by each client to the other clients is (cid:100) log 2 ( p ) (cid:101) bits.
Notice that log 2 ( p ) is of the same order of magnitude of l as p is the smallest prime number above 2 l .
More precisely, each client sends around l bits to each of the other clients for these elementary operations.
The overall size of the messages exchanged during the 726 rounds of communications mentioned above is mainly dominated by the 153 l + 423 log l + 24 multiplications done at each of the 40 comparisons.
Taking l = 100 , we ﬁnd that each client sends overall around 6 . 5 10 7 bits (or ∼ 8 Mega-bytes) to each of the other clients during S ECURE F ED YJ.
E Details of the numerical experiments E.1 Datasets used in this work
Datasets exposed by scikit-learn API used in Figure 2 and Figure 3 For numerical experiments, we use four public datasets available in the UC Irvine Machine Learning repository [15] under a
Creative Commons Attribution 4.0 International (CC BY 4.0) license and exposed by the scikit-learn datasets API.
These datasets are the Iris dataset [18] (150 samples, 4 features), the Wine Data Set (178 samples, 13 features), the Optical Recognition of Handwritten Digits Data Set (1797 samples, 64 features) and the Breast Cancer Wisconsin (Diagnostic) Data Set (569 samples, 30 features).
Only keeping features that have at least two distinct values, these datasets provide a total of 108 different features.
Extra UC Irvine Machine Learning repositories used to test Brent minimization method
Genomic data used in Figure 4 For genetic experiments, we rely on RNA-seq expression data from The Cancer Genome Atlas, expressed in Fragments per Kilobase Million (FPKM).
We focus on 3 cancers: colorectal cancer (COAD), lung cancer (LUAD + LUSC), and pancreatic adenocarcinoma (PAAD).
These datasets are available on https://portal.gdc.cancer.gov/ under Open Access.
E.2 Experiments on TCGA data
Based on FPKM counts, we load all available data for each cancer of interest, removing genes with null expression for all samples.
Pipeline Our pipeline consists of three steps:
Normalization: either whitening , log , or Yeo-Johnson transformation; 2.
Dimensionality reduction: a PCA was applied on normalized data to reduce dimension (dimension 128 for lung and colorectal cancer, 90 for pancreatic cancer); 3.
Cox Proportional Hazards (CoxPH) [10] model ﬁtting.
Normalization All normalization steps are performed on counts, regardless of the genes, as counts are related to the same underlying phenomenon induced by next-generation RNA sequencing.
In other words, for the plain whitening, a single mean and variance is computed.
For log , following application of log(1 + · ) to all entries, a similar count-level whitening is performed.
For the YJ transformation, we perform 10 iterations of the proposed algorithm.
CoxPH model training CoxPH models are ﬁtted with lifelines (0.26.4).
We use an (cid:96) 2 regularization of magnitude 10 for each cancer, without any hyperparameter optimization.
Cross-validation Results are computed following 5 -fold stratiﬁed group cross-validation, repeatedtimes with different seeds.
Stratiﬁcation is performed to ensure a balanced set of censored patients in each fold, while ensuring that samples belonging to the same patients end up in the same group to avoid over-estimating the generalization of the model.
E.3 Experiment on synthetic data
To generate the results of Section 5, we sampled for each of the 10 centers 200 datapoints using
Eq.
We then apply an optional preprocessing steps before ﬁtting a linear regression model using Datasetname #ofsamples #offeatures(withatleasttwodistinctvalues)
#ofinstabilitiesofBrentminimization airfoilselfnoise  0 bloodtransfusion  1 boston [Figure] breastcancerdiagnostic  2 california  0 climatemodelcrashes [Figure]
concretecompression
[Figure]
concreteslump
[Figure]
[Figure]
connectionistbenchvowel
[Figure]
ecoli
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
scikit-learn LinearRegression model on the pooled data.
Another dataset of 200 points was then generated, and we computed the R2 on this unseen dataset.
This experiment was repeated 1000 times using each time a different seed and the box plot in Section 5 presents the min-max, the median the ﬁrst third quartile.
The different preprocessing steps shown are
None: no preprocessing step is applied •
Whitening: for each of the three dimensions of X i , we subtract the empirical mean and we divide by the empirical standard deviation computed across all ten centers to the train dataset and the test dataset • LocalYJ: we use one center randomly chosen to perform E XP YJ with t max = 20 to each of the dimensions of the dataset.
The ﬁtted triplets λ ∗ , µ ∗ , σ 2 ∗ found for each column are then used to normalize the dataset of all 10 centers and the test dataset.
• Federated YJ: We apply S ECURE F ED YJ with t max = 20 on the 10 centers to each of the dimensions of the dataset.
The ﬁtted triplets λ ∗ , µ ∗ , σ 2 ∗ found for each column are then used to normalize the dataset of all 10 centers and the test dataset.
E.4 Testing Brent minimization on more dataset
As explained in the paragraph Numerical stability of E XP YJ of Section 3, applying blindly the Brent minimization method of scikit-learn to minimize the Yeo-Johnson negative log-likelihood might result in numerical instabilities and might collapse all the values of the dataset into a single value.
To check further whether this phenomenon is likely to appear, we apply the scikit-learn Yeo-Johnson transformation to various real-life tabular datasets of the UC Irvine Machine Learning repository [15] (which are under a Creative Commons Attribution 4.0 International, CC BY 4.0).
For each dataset, we only kept the features that have at least two distinct values.
We found that for the 484 fetaures out of 24 datasets, this issue arises 5 times, as summarized by Table[-
-]
F Further details on Proposition 4.
Proposition 4.1 states that all intermediate quantities of S ECURE F ED YJ can be recovered from its ﬁnal result λ ∗ .
We provide in Algorithm 4 a way to construct the function F introduced in
Proposition 4.1 that can perform this recovery.
We apply Algorithm 4 on the 108 features used in Figure 3, with a ﬁxed-point precision of f = 50 .
We numerically check that the output of F from Algorithm 4 matches the intermediate quantities revealed by Algorithm 3 up to machine precision.
Input: Hyperparameters λ t =0 , λ − t =0 , λ + t =0 number of steps t max , λ ∗ for t = 1 to t max if λ t − 1 < λ ∗ then ∆ t =else ∆ t
= −end if λ t , λ − t , λ + t ← E XP U PDATE ( λ t − 1 , λ − t − 1 , λ + t − 1 , ∆ t ) end for
Output: ( λ t , λ − t , λ + t , ∆ t ) t =0 ,...,t max