VF-PS: How to Select Important Participants in Vertical Federated Learning, Efficiently and Securely?
Abstract
Vertical Federated Learning (VFL), that trains federated models over vertically partitioned data, has emerged as an important learning paradigm.
However, existing
VFL methods are facing two challenges: (1) scalability when # participants grows to even modest scale and (2) diminishing return w.r.t.
# participants: not all participants are equally important and many will not introduce quality improvement in a large consortium.
Inspired by these two challenges, in this paper, we ask: How can we select l out of m participants, where l ≪ m , that are most important?
We call this problem Vertically Federated Participant Selection , and model it with a principled mutual information-based view.
Our first technical contribution is VFMINE—a Vertically Federated Mutual INformation Estimator —that uses one of the most celebrated algorithms in database theory—Fagin’s algorithm as a building block.
Our second contribution is to further optimize VF-MINE to enable VF-PS, a group testing-based participant selection framework.
We empirically show that vertically federated participation selection can be orders of magnitude faster than training a full-fledged VFL model, while being able to identify the most important subset of participants that often lead to a VFL model of similar quality.
Introduction
The quality of machine learning models heavily relies on the quality and volume of available data.
As our understanding of machine learning shifts towards such a data-centric view, mechanisms of making data available to machine learning have attracted intensive interests [1, 2].
Towards this goal,
Federated Learning [3–5] is an emerging research area that focuses on training a single ML model using all data available in a federated “ data consortium ” in a secure and privacy-preserving way.
Vertical Federated Learning.
In this paper, we focus on a specific federated learning scenario:
Vertical Federated Learning (VFL) [5].
As illustrated in Figure 1(a), the VFL setting involves a data consortium that consists of m participants P
= { P 1 , ..., P m } .
Each participant holds a disjoint subset of features associated with the same entity (e g., different hospitals holding medical records of the same patient but for different types of diseases).
The goal of vertical federated learning is to train a single ML model over the joint feature space, in a secure and privacy-preserving way, without communicating their data to each other [6–10].
Technical Challenges of VFL.
Despite many recent efforts that significantly improved the scalability and efficiency of model training in the VFL setting, two great challenges linger, as illustrated in
Figure 1(b).
The first challenge is scalability w.r.t # participants.
Because of the requirement on security and privacy, many VFL approaches have to resort to expensive methods such as Homomorphic
Encryption [11, 12] and Secure Multi-party Computation [13, 14].
This greatly limits the scalability of today’s VFL systems.
Another challenge is diminishing return , also when # participants grows.
Since not all participants contribute equally informative knowledge, it is not unusual to observe that adding more participants does not provide significant quality gain on the final ML model.
These two challenges, when put together, form a quite “ironic” picture, as illustrated in Figure 1(b)(iii) — supporting a large data consortium makes training time significantly longer, however, a dominating amount of these computations might not be translated into any significant quality gain .
Participant 𝑃
[Figure]
(Coordinator)
Features
Labels
[Figure]
𝑃
[Figure]
𝑃
[Figure]
Secure/Privacy-Preserving Communication
Features
Features (a) In the
Vertical Federated Learning
VFL) setting
, each participant
holds
a
disjoint
set of
features
for the same
set of entities
; the coordinator also holds the ground
truth
labels.
These participants communicate in
a secure and privacy preserving way in order
to
train
a single
machine learning model
over the joint feature space.
[Figure]
faces
[Figure]
challenges: (i) Scalability
[Figure] [Figure]
Diminishing
[Figure]
w
[Figure]
, since many participants in a large consortium might not provide significant quality gain.
As a result
[Figure]
a dominating amount of
[Figure]
computation cost for a large consortium
might
[Figure]
lead to
[Figure]
Vertically Federated Participant Selection (VF-PS).
Whereas designing more efficient model training methods in the VFL setting continues to be crucially important, in this paper, we focus on an orthogonal direction to accommodate these challenges.
Given a data consortium with m participants, instead of trying to speed-up the training of a single ML model over all m participants, we ask: “ Can we identify l ≪ m participants that are most important, and only train expensive VFL models on them instead? ”
We call this problem Vertically Federated Participant Selection (VF-PS) .
From Feature Selection (FS) to VF-PS: Connections and New Challenges.
At first glance, the VFPS problem is closely connected to traditional feature selection problems [15, 16].
This is definitely true and is what, in our opinion, makes VF-PS promising — the last half-century of study on feature selection provides not only principled theoretical framework but also strong evidence that we should be able to significantly decrease # participants (# features) involved but still obtain similar quality.
However, VF-PS imposes unique challenges compared to feature selection.
VF-PS should (1) provide no weaker security and privacy guarantees compared with other VFL training systems [9] and (2) should be much more efficient than training a single VFL model over all participants.
Summary of Technical Contributions.
In this paper, we focus on a specific, while one of the most natural, way of modeling VF-PS: we should select the subset of participants that jointly preserve the greatest amount of mutual information (MI) between features and labels.
Under this view, designing a VF-PS algorithm involves two technical issues: (1) how to estimate MI for a subset of participants?
and (2) how to search the subset of participants that preserve the greatest amount of MI?
• (C1)
Our first contribution is VF-MINE—a Vertically Federated Mutual INformation Estimator.
Given
a subset of participants, VF-MINE estimates MI between the features of these
participants and the labels
VF-MINE
protects data privacy via homomorphic encryption.
The most surprisingly
technical
observation is
that one
of the most popular MI estimators
], used by sklearn
can be made orders of magnitude faster.
This is by optimizing data movement between different participants, with Fagin’s algorithm [18] as a building block.
• (C2)
Our second contribution is VF-PS, that uses VF-MINE to search for the subset of participants that preserves the greatest amount of MI in a group testing-based framework.
We provide a set of optimization techniques to further optimize data movement between different participants.
• (C3)
We provide rigorous security analysis for the proposed methods.
We then empirically show that, our VF-PS framework can be up to two orders of magnitude faster than training full-fledged VFL models.
Moreover, VF-PS can often select a much smaller subset of participants but maintain a similar model quality.
Preliminaries
Federated Learning.
Federated learning is a category of distributed learning technique that enables different organizations or users to collaboratively learn machine learning (ML) models, without exposing their personal data to other parties [19–21].
Assume X denotes instance features, Y denotes instance labels
and I denotes instance IDs, federated learning can be classified into two categories, i.e., horizontal federated learning and vertical federated learning, according to how ( X , Y , I ) are distributed across different participants.
Privacy Protection Techniques.
As a critical property of federated learning, data privacy has been extensively studied.
Briefly speaking, there are three major categories of privacy protection techniques.
1) Differential Privacy randomly perturbs the transferred data (e.g., gradient) with noises, such as
Gaussian noise [22], Laplacian noise [23], and Binomial noise [24].
Homomorphic Encryption encrypts all data before they are transferred [12, 25, 26] and supports arithmetic computation on encrypted data.
3) Secure Multiparty Computation (SMC) enables multiple parties to collaboratively compute an agreed-upon function without leaking local input to any other party except for what can be inferred from the output [13, 14, 27, 28].
Vertically Federated Participant Selection via Mutual Information
In this section, we describe our algorithm for Vertically Federated Participant Selection (VF-PS).
3.1 Vertically Federated Participant Selection
We assume a standard data model, following previous efforts in VFL [7, 8] — let there be m participants and N data instances, and X ∈ R N × F be the joint feature space where F is the dimension of the joint feature space.
In the VFL setting, this joint feature space X is vertically partitioned over different participants — each participant P i ∈ P holds a subset of features (columns) of X , denoted by X i .
Without loss of generality, we have: X =
[ X 1 ...X i ...
X m ] .
Among all m , there is a leader participant , which holds the instance labels Y .
We assume that all participants agree on the same identifier (ID) of each data instance.
Security Requirements.
Following previous systems, e.g., Pivot [9], VF 2 Boost [29], we have the following standard security requirements.
1) Feature Security : Features X i on each participant cannot be shared with any other party.
2) Aggregation Security : Only the leader participant can obtain an aggregated result over all participants ’ local data.
3) Identity Security : No other party, except the participants, can obtain the identifiers (IDs) of instances.
Mutual Information-based VF-PS.
Similar to the classic feature selection problem, we focus on one of the most natural ways to model VF-PS — we aim at selecting the subset of participants that jointly preserve the greatest amount of mutual information with the label.
Formally, the computation that we hope to conduct, under the previous security requirements and vertically partitioned data model, is
[Equation] [Equation]
i
[Equation]
[ l ] :
[Equation]
i
[Equation] [Equation]
and ∀ i, j ∈ [ l ] :
[Equation]
j .
[Equation]
where MI ( - ) estimates the mutual information from finite samples, and l < m is a pre-defined constant specifying the number of participants to select.
3.2 VF-MINE: A Vertically Federated Mutual Information Estimator
As we see in Equation 1, solving this problem involves two different questions: (1) how to estimate
MI given a fixed subset of participants, i.e., MI
[ X α 1 ...
X α l ]; Y ) ?
and (2) how to efficiently search for the best subset, i.e., max α 1 ....α l , without enumerating exponential many possibilities?
We focus on the first problem in this section, and leave the second question to Section 3.3.
3.2.1 Baseline Method
KNN-based MI Estimator.
We focus on a standard family of MI estimators that is widely used in practice (e.g., in sklearn ’s feature selection module), which is based on k -nearest neighbors (KNN)[17, 30].
Let there be N data pairs D = { ( x j , y j ) } where y j is a label and x j is a feature vector.
A KNN-based MI estimator enumerates multiple query data pairs.
For each query q
=
( x q , y q ) :
We first compute N q , the number of examples in D that share the same labels as y q ,
i
e.,
[Equation]
We then construct NN k,y q , containing all k -nearest neighbors among data examples that share the same label.
Let ¯ d q be the maximal distance between examples in NN K,y q and x q . 3.
We then compute m q , the number of examples in D that has a distance to x q smaller than ¯ d q .
Given Q as a subset of query data pairs, we estimate the mutual information as follows:
[Equation]
)) , where ψ is the digamma function ψ ( x ) =
[Equation] [Equation]
Baseline Implementation of VF-MINE.
We now provide a baseline implementation of this estimator in the vertically federated setting.
We assume that we use Euclidean distance as our distance metric, but our technique can be naturally applied to other distance metrics.
We choose homomorphic encryption (HE) to protect data privacy since it is widely adopted in the literature [8, 11, 25]; however, our proposed algorithm can also be instantialized with other privacy protection techniques such as SMC (see discussion in Appendix C).
We use HE.Enc ( - ) /HE.Dec
( - ) to denote the encryption/decryption function, and [ - ] an encrypted data item.
Below, we present a single VF-MINE routine.
There are three roles in the system — key server , aggregation server , and participant :
• Key server.
The key server generates public and private HE (homomorphic encryption) keys ( pk, sk ) , allocates sk to the leader participant, and pk to all participants and the aggregation server.
• Aggregation server.
An aggregation server provides an addition operator that securely aggregates multiple encrypted data items:
[ d ] =
HE.Sum ( { [ d i ] } ; pk ) .
• Participant.
Each participant P i holds a feature subset of all N data instances.
For a query
data pair q = ( x q , y q ) , each participant P i calculates d i =
[( x q,i − x j,i ) 2 for ( x j , y j ) ∈ D ] , the distances between its local features of x q and data pairs in D .
We call d i the partial distances.
Then, P i encrypts partial distances and sends [ d i ]
=
HE.Enc ( d i ; pk ) to the aggregation server via the addition operator.
There is a leader participant that holds the labels.
It receives the complete distances [ d ] from the aggregation server, decrypts it to d =
HE.Dec ([ d ]; sk ) , and calculates MI, responding to the procedure of KNN-based estimator.
3.2.2 Fagin-Inspired Method
The baseline method has a potential performance issue, that is, it has to aggregate all the data instances.
Since the operation through homomorphic encryption is often time-consuming, this baseline method can be inefficient in practice, especially for many large-scale datasets.
To decrease the cost, a potential way is to decrease the number of aggregated instances.
Motivated as such, our first question is — can we aggregate fewer data instances and meanwhile obtain the correct results?
We observe that the vertical KNN task can be seen as aggregating multiple “sub-ranking”s into a “global ranking”.
Based on this interpretation, we propose to leverage a textbook aggregation algorithm, called Fagin , to efficiently find k -nearest neighbors.
Fagin Algorithm.
The problem of top- k query aims at finding k instances with the highest (or lowest) scores from multiple lists of instances.
Assume there are m parties and N instances, each party i has a score for each instance j , denoted by s ij .
On each party, the instances are sorted by their scores.
Globally, each instance is assigned an overall score by combining the scores on all parties using an aggregate function.
This problem has been extensively studied by a series of algorithms [31].
In this work, we choose Fagin as it is widely used [18].
Fagin requires that the aggregation function aggr is monotone: aggr ( s 1 , s 2 , ..., s m ) ≤ aggr ( s ′ 1 , s ′ 2 , ..., s ′ m ) if s i < s ′ i for every i .
Figure 2(a) shows an example of Fagin algorithm that selects top-2 instances from three parties with the addition aggregation function.
The processing of Fagin algorithm contains three major steps: 1) sequentially scan the sorted lists from all parties until there are k instances that have been seen in all lists, 2) obtain the scores of all the instances seen so far (including those have not yet occurred in all lists), and 3) use the aggregation function to compute global scores for candidate instances found in the previous step and select top- k instances therein.
In the example of Figure 2(a), Fagin scans three rows and finds that s 1 and s 3 are in three lists.
Afterwards, the scores of all instances in three rows (i.e., s 1 , s 2 , s 3 , s 4 ) are aggregated. We denote math. Fagin ( - ) the standard Fagin algorithm in this paper, that takes multiple lists of instance IDs as input and outputs candidate IDs.
PartyA
Party B
Party C
Instance Score ! !
$" 0.
Instance Score !
$' 0.5 ! &' 0.
Instance Score !
$( 0.
$ 0.
(a) Fagin Algorithm ! ! " !
Participant ! !
(Leader)
Aggregation Server !
Enc(*)
Participant ! "
[" $ ]
Dec(*) "
$
MI & %,!
Fagin
Key Server
Shuffle
Sort  # !
Reindex
IDmap # " [" !
$ ] [ " $ ]
Addition… ! '
Participant !
# - # (# [" !$ ] - [" ($ ] # → # #
[Figure]
keys
[Figure]
Fagin-inspired Optimization for VF-MINE.
We now implement the Fagin-inspired method.
The key allocation (step ① ) is the same as the baseline method.
The key is to aggregate “sub-ranking”s with
Fagin algorithm while preserving their privacy.
The implementation of the secure Fagin operator is shown in Figure 2(b) (step ② - ⑤ ).
We propose a shuffle strategy — each participant uses the same seed to randomly shuffle local data instances (step ② ).
Then, we generate a “pseudo ID” for each instance that is equal to the index after shuffling.
Each participant P i sends the sub-ranking of pseudo IDs, denoted by I ′ i , to the aggregation server, instead of sending the real IDs (step ③ - ④ ).
The aggregation server runs the standard Fagin algorithm to find k -nearest candidates from all sub-rankings.
Since the estimation of MI depends on the label, the leader participant is responsible for stopping Fagin.
The aggregation server sends the pseudo IDs of Fagin candidates, denoted by I ′ k , to each participant (step ⑤ ).
After the Fagin operator, the participants re-index these pseudo IDs to their original IDs (step ⑥ ).
The following steps are similar to the baseline method (step ⑦ - ⑩ ).
3.3 VF-PS: Group Testing-based Participant Selection
The second component in our framework is to search for the subset of participants that maximizes
MI.
We adopt a standard group testing-based search procedure.
Group-testing Framework.
The VF-PS problem under the group testing framework runs as follows:
We select a collection of T tests , each of which corresponds to a subset of participants S t ⊆
P ; 2.
For each S t , we estimate its mutual information and use it as the score ; 3.
For each participant, we compute its importance as the average score in all tests it participated in.
We pick the top- k most important participants.
Formally, let U ( S ) be the score of group S ⊆
P defined as the mutual information over N v validation data pairs: V = { ( x vj , y vj ) } , and let A of dimension T × m be the participating matrix, where
A t,i = 1 if participant i belongs to the t -th test group.
Corresponding to each row of A is a “score”
U ( A [ t ]) =
U ( { i | A t,i = 1 } ) .
Consider A as the test design, the total score of participant P i is u i =
A [:][ i ] · [ U ( A [0]) , ..., U ( A [ T − 1])] .
The participant selection is conducted using these scores.
Implementation of Group Testing.
The implementation of group testing for Fagin-inspired method is quite natural.
Specifically, we run Fagin-inspired MI estimation over each test group (subset of participants).
The counterpart using the baseline method can be obtained by skipping the Fagin stage.
Generate testing groups.
The leader participant randomly generates T groups {+(e.g., the first group S 
+}
= { P 1 } and the second group S 2 = { P 1 , P 2 } ), which are sent to other participants.
Run Fagin-inspired method.
The proposed framework handles each test group t independently.
Specifically, for each validation instance, the participants in test group
A [ t ] jointly run Fagininspired KNN and calculate the mutual information.
The score of the group, U ( A [ t ]) , is averaged over all the validation instances.
Participant selection.
Once running all the groups, the leader participant calculates the MI-based score of each participant, with which most valuable participants are selected. Key Server: homomorphic Send
Participants: function randomly
3.3.1 Group Testing with Communication Batching
The naive implementation of the group testing routine has a critical drawback.
The Fagin candidates generated by different groups may contain overlapping instances, causing significant amount of redundant overheads.
As we will show, by leveraging what we call the Inclusion Property of the
Fagin algorithm, we can batch the communication and computation of all test groups.
Inclusion Property of Fagin.
The principle of the Fagin algorithm is to scan the sub-rankings from multiple participants until k distinct instances are found in all the sub-rankings.
Given two subsets of participants, S 1 and S 2 , what we can say about their data access pattern?
Theorem 1.
(Fagin’s Inclusion Property)
Given two groups of participants S 1 ⊆ P and S 2 ⊆ P , their k -nearest candidates generated by Fagin algorithm satisfy F 1 ⊆
F 2 if S 1 ⊆ S 2 .
Please refer to Appendix A for the proof of Theorem 1.
This indicates that the Fagin candidates of all groups can be found in those of the “complete group” that consists of all the participants.
Based on this observation, we propose to batch the Fagin tasks of different groups in a single execution.
Implementation of Batch Optimization.
The batching optimization is illustrated in Algorithm 1.
The leader participant generates groups G = { S 1 , ...S T } and sends G to the aggregation server.
The following shows the steps after getting the Fagin candidates, I k , for the complete group P .
Each participant P i encrypts partial distances with homomorphic encryption.
(line 1) 2.
Each participant sends encrypted partial distances of Fagin candidates, i e., [ d F i ] , to the aggregation server through Server.addition (line 2).
For each group S ∈ G , the aggregation server calculates the sum of all partial distances in S .
A random scheduler, Server.scheduler , on the aggregation server sends the aggregated distances of each group (denoted by [ d S ] ) to a participant randomly (line 3).
For each received [ d S ] , the participant performs decryption and gets the sorted index I ′ S (line 4).
The sorted indices are sent to the aggregation server through Server.scheduler callback (line 5).
The aggregation server lets the random scheduler associate them with the corresponding groups.
Then, the aggregation server sends the sorted IDs of all the groups, { I S } , to the leader participant.
The leader participant calculates the MI-based scores using the method of group testing and performs participant selection (line 6). 3.4 Security Analysis Threat Model.
Without loss of generality, we assume that the key server is honest and does not collude with any involved party.
The aggregation server and all the participants are honest-but-curious, a common knowledge threat model used in the federated learning literature [28, 32].
Datasets
Synthesis
G2G2-
Bank
Unbalance
Letter
Birch
Birch2 # instances     20K 100K 100K # features       2 # classes       100 # partitions
The security properties of our proposed methods are summarized as follows (recall security requirements in Section 3.1).
Feature security is protected against any curious party since the participants do not share their local features.
Aggregation security is assured against curious aggregation server by homomorphic encryption.
Regarding curious or colluding participants, aggregation security is also assured if the leader participant is benign.
Identity security is achieved against a curious server if there is no server-participant collusion, because the proposed methods do not share instance identifiers
I with the server.
For detailed analysis, please refer to Appendix B.
Evaluation
We conducted experiments to validate the efficiency and effectiveness of the proposed methods.
4.1 Experimental Setting
Implementation.
Numpy and PyTorch libraries are used to perform data loading and tensor manipulations.
We implement RPC communication with proto2 and gRPC.
TenSEAL [33] is a homomorphic library built on top of Microsoft SEAL [34].
We run all experiments in a cluster, in which each machine is equipped with 24 GB memory, 8 cores and 10 Gbps bandwidth.
Datasets.
Table 1 illustrates all datasets that we used.
Synthesis is a synthesis dataset by randomly generating a separation hyperplane, based on which data instances are sampled.
The other datasets are collected from online repository [35, 36] and prior work [37–39].
Each dataset is split into a training set (80%), a validation set (10%), and a test set (10%).
We randomly partition the training set into several vertical partitions and put each partition on one machine.
ML Models.
We choose Pivot [9], a vertically federated tree training engine, as the downstream ML model.
Additionally, we also implement logistic regression (LR), and k-nearest neighbor (KNN).
Participant Selection Baselines.
We compare the following baselines on participant selection.
R ANDOM : We randomly choose l participants and run ML model.
The result is averaged over ten [Table] runs to assure the robustness.
2) L ASSO .
Lasso regression is a standard feature selection method in scikit-learn .
We train a Lasso model and choose high-importance participants according to the absolute sum of coefficients on each participant.
3) V F -P S is our proposed method.
Protocols.
We implement Adam [40] and vanilla SGD as the optimization algorithm for LR and
Lasso.
We grid search the optimal learning rate in { 0 . 001 , 0 . 01 , 0 . 1 } and the regularization term in { 10 − 4 , 10 − 3 , 10 − 2 } , and set batch size to 256.
We terminate the task after 50 epochs or the validation loss does not decrease within 5 epochs.
For Pivot, we choose GBDT tree, and tune tree depth in { 2 , 3 , 4 } and tree number in { 1 , 2 , 3 } .
The number of groups T is set to 10.
The other hyper-parameters are set as default.
4.2 Evaluation of Participant Selection
Selection Performance.
We first study the following question: can our proposed method outperform baselines regarding the selection of high-importance participants?
We compare R ANDOM , L ASSO and V F -P S in terms of their performance of participant selection.
When the downstream classification model is Pivot, Figure 3 shows the performance of all methods for selecting 50% participants.
R ANDOM is the fastest by instantly choosing participants; however, its selection accuracy can be much worse on many datasets.
L ASSO is the slowest since it needs to run multiple epochs until convergence and tunes the hyper-parameters.
V F -P S is only slightly slower than R ANDOM and significantly faster than L ASSO .
Meanwhile, V F -P S selects comparable, and often better, subsets of participants, indicated by the accuracy of training Pivot only on selected subsets.
When the classification model becomes LR and KNN, we observe similar phenomena, as shown in Figure 4-5.
We also study cases when the selection becomes more aggressive (25%), and find that V F -P S also achieves the best trade-offs between selection speed and selection quality (refer to Appendix D.1).
Comparison to Full-fledged Training.
We then turn to another question: does participant selection provide benefits compared to training with all the participants?
To answer this question, we also show the result of training using all participants (“All-Train”) in Figure 3-5.
As we can see, the selection time of V F -P S is orders of magnitude faster than the training time using all the participants; meanwhile, if we were to train a model only on the selected subset, we can achieve similar test accuracy on most datasets.
On some datasets, e.g., Synthesis, G2-128, and Letter, we even observe higher accuracy after participant selection compared to the full-fledged training.
This shows that some participants may have negative impacts on the model quality, and it is beneficial to choose high-importance participants.
These experimental results verify the effectiveness and efficiency of participant selection.
Comparison to Brute-force Strategy.
Intuitively, the brute-force solution for participant selection is to enumerate all the possible groups and choose the best one.
Although this method is too timeconsuming (up to two orders of magnitude slower according to our results), it provides the best possible selection strategy that can be used to understand — how big is the gap between our selection strategy and the optimal (brute-force strategy)?
Table 2 shows the results of V F -P S and brute-force on KNN (other models are too slow to run the brute-force strategy).
When the selection ratio is 50%,
V F -P S gets the same selection quality on six datasets out of eight.
Overall, the gap of selection quality is moderate on most datasets.
It is an interesting future direction to come up with better participant selection criteria, going beyond mutual information.
Ablation Study.
We now validate that both system optimizations (Fagin and Batching) significantly improve the system performance.
We compare V F -P S with three variants—w/o Fagin, w/o batching, and w/o both—as illustrated in Figure 6.
We see that both optimizations are crucial to the performance — disabling either of them significantly slows down the runtime on all datasets. More Experiments.
Due to space constraints, we put additional experiments in the appendix, including the selection performance on deep learning models (Appendix D.2) and the impact of group number
T (Appendix D.4).
Related Work
Federated Learning.
Federated Learning (FL) is a category of distributed machine learning approach that lets multiple clients jointly train machine learning models over their personal data without data leakage
Since the feature space and sample space of participating parties may not be identical, federated learning can be classified into horizontally federated learning and vertically federated learning.
Horizontally federated learning is a scenario in which different parties share the same feature space but different sample space.
In contrast, in vertically federated learning, different parties have the same sample space but different feature space.
Vertically Federated Learning.
Hardy et al.
[8] proposed a three-party vertically federated logistic regression solution by entity resolution and additively homomorphic encryption.
Yang et al.
[41] from
WeBank and Yang et al.
[42] from Baidu further extended vertically federated logistic regression to Quasi-Newton optimization and non-coordinator scenario.
Feng and Yu [6] designed a multiparticipant and multi-class federated learning framework for vertical training data.
VAFL [7] was a vertically federated learning framework that allows each client to asynchronously run stochastic gradient algorithms without coordination with other clients.
Pyvertical [43] was proposed to train multi-headed SplitNNs in the context of vertically federated learning using private set intersection.
Liu et al.
[44] proposed an asymmetrical vertically federated learning framework that protects sample
IDs with the private set intersection protocol.
Pivot [9] was proposed to train privacy-preserving vertical decision tree using SMC protocol.
Federated Participant Selection.
In the horizontally federated scenario, FedCS [45] tried to solve the resource heterogeneity in federated setting and conducted client selection to accelerate system performance.
FedMCCS [46] further introduced a multicriteria approach for client selection, considering clients resources including CPU, memory, energy, and time.
To address limited bandwidth in the presence of many clients, Huang et al.
[47] proposed a selection policy to improve training efficiency while assuring client fairness.
Different from these works that studied participant selection for system heterogeneity [48], our work focuses on selection regarding the contribution of each individual participant for the trained model.
A few early, seminal works studied contribution-based participant selection in horizontally federated learning [49].  Limitation Although V F -P S works well on a range of workloads, it still has several limitations under other use cases, e.g., multiple training stages, reward assesement of participants, generalization of MI estimator, and large-scale federated network. We leave the detailed discussions to Appendix E.
Conclusion
In this work, we study participant selection in the context of vertically federated learning.
We model this problem as a mutual information-based view.
We first propose a novel mutual information estimator, called VF-MINE, that uses Fagin’s algorithm.
To perform participant selection, we propose a group testing-base framework VF-PS on top of VF-MINE.
We implement our proposed method using homomorphic encryption and design a batching optimization.
We analyze the security properties of the proposed methods and show that our methods achieve orders of magnitude improvements.
A Proof of Theorem
Proof.
For the group S 2 , assume the Fagin algorithm scans F rows of sub-rankings { R i , ∀ p i ∈ S 2 } before finding instances D = { x 1 , x 2 , ..., x k } occurring in ∀ p ∈ S 2 .
The ranking of each instance in
D should satisfy:
[Equation]
Since S 1 is a subset of S 2 , we can directly infer:
[Equation]
The above implies that we can find D in each participant p ∈ S 1 before scanning F rows The Fagin candidate set of S 2 contains all the unique instances within F rows:
[Equation]
Similarly, the Fagin candidate set of S 1 is a subset of all unique samples within F rows:
[Equation]
Since S 1 is a subset of S 2 , all the samples appear within F rows of S 1 also appear in S 2 :
Set ( ∪ p i ∈S 1 R
i [: F ]) ⊆
Set ( ∪ p i ∈S
R
i
[: F ])
Therefore, we can conclude:
[Equation]
Based on Theorem 1, we can deduce the following Lemma:
Lemma 1.
If the Fagin candidates F of the complete group P are known, we can find top-k instances in F for any S ⊆ P .
B Security Analysis
We conduct a comprehensive analysis to discuss the security properties of our methods.
B.1 Threat Model
Below, we define the threat model for three roles in the framework.
• Key server.
Without loss of generalization, we assume the key server is honest and does not collude with any involved party.
• Aggregation server.
The aggregation server is honest-but-curious , a common threat model used in the federated learning literature [28, 32].
The aggregation server may try to recover the received data.
Besides, there is no collusion between the aggregation server and any participant; otherwise, the aggregation server can decrypt anything it receives from the participants.
• Participant.
All the participants are also honest-but-curious .
Each of them provides the correct data and follows the protocol, but may try to learn other participants’ data.
In terms of collusion, some participants may collude with each other and try to recover data from benign participants.
B.2 Baseline Method for MI Estimation
Here we analyze the baseline method for MI estimation.
Feature Security.
Our proposed method does not allow the participants to share their local features.
As a result, the feature security is protected against any curious party.
Aggregation Security.
The aggregation security is assured after encryption if all the participants are benign.
We separately discuss the cases of curious parties and collusion.
• Curious aggregation server.
Since the aggregation server does not have the private key of HE nor collude with any participant, the partial distances are fully protected against a curious aggregation server.
• Curious participant.
A curious participant does not directly receive [ d i ] from other participants, and hence cannot speculate benign participants.
A special case is that there are only two participants, and a curious leader participant can infer the other participant’s partial distance.
Therefore, the baseline method is secure if the leader participant is benign or there are more than two participants.
• Colluding participant.
If the leader participant does not collude, the colluding participants cannot speculate other participants’ partial distances, since they do not have the complete distances.
On the contrary, if the leader participant does collude, it can directly get the partial distances of the colluding participants.
However, as long as there are more than two benign participants, the leader participant still cannot speculate their partial distances.
In summary, the aggregation security is assured if the leader participant is benign or there are at least two benign participants.
Identity Security.
Since the baseline method does not share instance identities I with the server and there is no server-participant collusion, the data identities are protected against the server.
B.3 Fagin-inspired Method for MI Estimation
Feature Security & Aggregation Security.
Same as the baseline method, the Fagin-inspired method does not share local features and original partial distances to the aggregation server.
Therefore, the feature security and aggregation security remain the same.
Identity Security.
The Fagin-inspired method needs to find k -nearest candidates through the aggregation server.
Although the instance IDs are randomly shuffled and replaced by “puesduo” IDs, it is still unknown whether this method can protect the local ranking of each participant.
Conceptually, the
Fagin algorithm can be seen as an extension of private set intersection, which has been studied in [50].
According to their rigorous analysis, the random shuffling operation can assure the security of local rankings if (1) all participants are benign, (2) the aggregation server is curious but the participants are benign, or (2) the aggregation server is benign with any collusion of participants.
B.4 Fagin-inspired Participant Selection
We next conduct a security analysis for the proposed participant selection method.
Feature Security & Identity Security.
Refer to the analysis for Fagin-inspired MI estimation.
Aggregation Security.
If the aggregation server and the participants are benign, the partial distances are protected by homomorphic encryption.
We analyze the following threat models
• Curious aggregation server.
If there is no server-participant collusion, the aggregation security is not harmed as the aggregation server only has the public key.
• Curious participant.
The proposed method allocates aggregated distances across the participants.
This raises a potential risk — can a curious participant recover other participants’ partial distances?
This is possible if the assignment strategy is fixed and the curious participant can somehow guess it.
However, since our proposed approach adopts a random scheduler, it is impossible for any curious participant to speculate the allocation pattern.
• Colluding participant.
It is possible for colluding participants to speculate benign participants if the allocation strategy is easy to speculate, due to the same reason explained above.
Likewise, the random scheduler can protect aggregation security in the presence of colluding participant.
C Other Privacy Preserving Techniques
In principle, our framework can be instantiated with other security protocol, such as Secure Multiparty Computation (SMC).
Providing a full implementation in SMC is beyond the scope of this paper, and we expect, just like the HE implementation is non-trivial, an SMC-based implementation will have its own challenges but also open up new opportunities for optimization.
In the following, we provide some thoughts about a potential SMC-based implementation and our perceived challenges.
Since two oracle operators in our protocols are secure addition and secure Fagin , we will focus on these two operators.
Secure Addition Operator with SMC.
Addition is a basic operation in SMC.
Since our protocol for participant selection needs to handle groups of different sizes, we need to tailor the secure addition operator under two-party and multi-party scenarios.
Secure Fagin Operator with SMC.
As Fagin is not a standard mathematical or logical operator, the existing SMC works have not studied how to implement Fagin.
This is indeed an obstacle for the instantiation of our proposed protocol.
The most similar standard SMC operator is set intersection [50] that finds overlapping items from multiple lists.
But, this cannot fully replicate Fagin because Fagin outputs all the items have seen after finding k overlapping items.
Nonetheless, if SMC-based Fagin has an efficient solution in the future, it can be integrated into our proposed protocol.
Note that, since
SMC operators are generally more costly than plaintext operators, it is important to assess the overall cost when SMC-based Fagin is introduced.
D More Experimental Results
Due to space constraints, we present additional experimental results here.
[Figure]
[Figure] [Figure]
D.1 Selection Performance
Figure 7 to 9 show the results when selecting 25% participants.
We observe similar phenomena as what we have summarized for selecting 50% participants.
D.2 Deep Learning Models
[Table]
Except for linear models and clustering models, it is interesting how V F -P S performs over more sophisticated ML tasks.
In the main manuscript, we run the GBDT model in Pivot which has a non-convex loss.
To further strengthen the evaluation, we conduct an experiment with neural network models.
We run the vertically federated selection method and then train a neural network model over selected features.
The evaluated model is an MLP (multilayer perceptron) that has two fullyconnected layers.
The hidden size is 32 and the batch size is 32.
Table 3 presents the results of three baselines over Letter dataset.
V F -P S achieves the best selection performance for choosing 25% and 50% of participants.
Specifically, the test accuracy trained with the entire dataset is 0.6, and V F -P S can obtain an accuracy of 0.58 after selecting 50% participants.
Note that, in this experiment, we train the neural network model in a centralized way because we cannot find any practical open-source framework.
We will include neural network models into our framework if practical implementations under the vertically federated setting become available in the future.
[Figure]
[Figure]
u r a c y
[Figure]
t i m e
D.3 Scalability
We evaluate the scalability of different participant selection methods.
We partition the Letter dataset into different numbers of partitions ( m =2/4/8/16), and then run participant valuation for a KNN model.
As shown in Figure 10, the brute-force approach encounters timeout failure (longer than one day) when m become larger than 4, because there are too many groups to evaluate.
Unsurprisingly, the R ANDOM approach costs nearly zero time on the selection.
V F -P S w/o batching and w/o Fagin both outperform the brute-force method since the Fagin strategy reduces a large portion of instances to be processed and the batching optimization reduces redundancy.
When the number of participants increases, V F -P S also encounters performance degradation because the Fagin strategy becomes less effective for a large group.
Nevertheless, V F -P S still significantly outperforms other methods with the batching optimization. D.4 Impact of T
Our proposed method has a hyperparameter T , the number of groups.
Intuitively, the value of T can affect the selection performance.
To assess its impact, we train LR over Letter and vary the value of T from 1 to 15.
As shown in Figure 11, the accuracy of selection results increases with a larger
T .
This is unsurprising because adding more groups can try more group combinations and therefore improve the selection quality.
This was also verified by previous works, e.g., Theorem 2.2 and Figure 5 in [31].
Besides, the selection time also increases with more test groups, but the increase is not significant due to our batching mechanism.
In our previous experiments, we set T to 10 since it achieves a good trade-off between accuracy and selection cost.
E Limitations
In summary, V F -P S has the following limitations:
• Multiple training stages.
In this paper, we treat V F -P S as a task that is similar to feature selection, which is a pre-processing step that is often “once-and-for-all”.
However, there are use cases in which the importance of each participant changes during different training stages, then V F -P S may not find the optimal result.
• Reward assessment.
The goal of this work is to measure the mutual information of each participant.
To come up with a “reward”, it needs to be carefully “normalized” over the “importance” of each participant — directly using the mutual information provided by V F -P S as “reward” could cause problems on fairness.
This problem definitely requires careful further studies.
• Generalization of MI estimator.
KNN-based MI estimator can be biased towards smaller local feature subsets with Euclidean distance.
Our technique can also go beyond Euclidean distance — as long as local scores can be aggregated using a monotonic aggregation function, most, if not all, of our optimizations can still be applied.
But still, whatever limitations that MI-based method has for feature selection, our methods will probably also inherit.
We will study other distance metrics in future work.
• Large-scale federated network.
It is an interesting future research direction to understand realworld VFL scenarios where the federated network consists of an even larger number of participants/organizations.
When this number becomes very large, we do expect that VF-PS could encounter performance degradation due to the limitation of the Fagin strategy.
We will study this challenging problem in future work.