A Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Federated learning is a learning paradigm to enable collaborative learning acrossdifferent parties without revealing raw data.
Notably, vertical federated learning(VFL), where parties share the same set of samples but only hold partial features,has a wide range of real-world applications.
However, most existing studies in VFLdisregard the “record linkage” process.
They design algorithms either assuming thedata from different parties can be exactly linked or simply linking each record withits most similar neighboring record.
These approaches may fail to capture the keyfeatures from other less similar records.
Moreover, such improper linkage cannotbe corrected by training since existing approaches provide no feedback on linkageduring training.
In this paper, we design a novel coupled training paradigm, FedSim,that integrates one-to-many linkage into the training process.
Besides enabling
VFL in many real-world applications with fuzzy identifiers, FedSim also achievesbetter performance in traditional VFL tasks.
Moreover, we theoretically analyze theadditional privacy risk incurred by sharing similarities.
Our experiments on eightdatasets with various similarity metrics show that FedSim consistently outperformsother state-of-the-art baselines.
Introduction
Federated learning is a collaborative learning framework to train a model from distributed datasetswith privacy guarantees.
A commonly existing and widely studied scenario of federated learning isvertical federated learning [31, 58] (VFL), where multiple parties sharing the same set of sampleshave different sets of features.
We focus on the setting where only one party holds the labels likemost of the studies [32, 38, 57].
The party holding labels is named primary party ; the parties withoutlabels are named secondary parties .
In VFL, the features that exist on multiple parties are calledcommon features .
The vector of common features in a data record is called the identifier of the record.
Existing studies [21, 26, 38, 39] formulate VFL as two separated processes: linkage and training.
Inthe linkage process, the datasets on different parties are linked according to the identifiers.
In thetraining process, these distributed but linked data records are trained by VFL algorithms.
Specifically,in the linkage process of existing studies, each data record is linked to a data record with the exactlymatched or the most similar identifier (i.e., one-to-one linkage).
Although the one-to-one linkage isintuitive, it can seriously impair the accuracy of the VFL model for the following two reasons.
First, only linking the records with top similarity (i.e., one-to-one linkage) does not necessarilycapture the key features, which can be demonstrated by two real-world applications.
1) Consideringthe VFL between steam games and IGN games linked by the game titles, as shown in Figure 1b,linking not only the exactly matched game but also other games in the same series intuitively benefitsgame recommendation tasks.
Considering the VFL between a real estate company and a houseleasing company linked by GPS locations of houses, as shown in Figure 1a, the houses that house
A
[Figure]
(purple) can benefit from (i.e., with similar prices) are located a bit far from A .
Even in applicationswhere identifiers can be exactly matched (e.g., identifiers are ID), if some features are missing orbiased, one-to-one linkage prevents the training process from enhancing these features from othersimilar records.
These applications commonly exist in practice according to our investigation in
Appendix I. An opposite extreme case is to link each record with all the records in another party( one-to-all linkage), which keeps all the information but is too expensive for both linkage andtraining.
Therefore, a one-to-many linkage approach is needed as a balance between efficiency andperformance.
Second, separating the linkage from the training also harms the performance of VFL.
In existing
VFL approaches, since the linkage process cannot obtain any feedback from the training process, thelinkage is conducted with the goal of finding true-matched pairs of records instead of finding thepairs that reduce the training loss.
Hence, an integrated VFL framework that conducts one-to-manylinkage under the guide of training is desired.
To address these two drawbacks, we link each data record to the records with top- K similar identifiersin another party and design a coupled framework of linkage and training.
Our main challenge is toeffectively exploit these linked pairs and their similarities to boost the performance of VFL.
To tacklethis challenge, we propose a similarity-based coupled VFL framework FedSim on the top of SplitNN[53] which is a VFL algorithm for neural networks.
In FedSim, similarity is a dominant feature thatdetermines the order and the weight of each pair of linked records.
The weights (i.e., impact) ofsimilarities are also adjusted in each training iteration.
After the training, each similarity is mappedto a weight based on its contribution to reducing the loss as visualized in Appendix D.5.
Furthermore, To address the additional data privacy issue incurred by FedSim, we propose an intuitivegreedy attack to infer identifiers from similarities.
Moreover, we theoretically prove that such anattack can be defended by adding Gaussian noise to the similarities.
More advanced attacks that arediscussed in Appendix F will be left as our future work.
Our main contributions can be summarized below.
We propose a novel asymmetric trainingparadigm, named FedSim, and a training-free metric to estimate the improvement of FedSim onbaseline approaches; 2) We propose a greedy attack on FedSim and the corresponding defense methodthat can theoretically bound the success rate of this attack; 3) We conduct extensive experimentson three synthetic datasets and five real-world datasets, which indicates that FedSim consistentlyoutperforms state-of-the-art baselines.
Preliminaries
Privacy-Preserving Record Linkage.
Privacy-preserving record linkage (PPRL) [52] aims to linkthe data records from two parties that refer to the same sample without revealing real identifiers.
Most
PPRL methods [27, 50, 52] consist of three main steps: blocking, comparison, and classification.
First, in the blocking step, data records that are unlikely to be linked are pruned to reduce the numberof comparisons.
Then, in the comparison step, a similarity between identifiers is computed for eachcandidate pair of data records.
Finally, in classification, each candidate pair is classified as “matches”or “non-matches”, which is usually done by a manually set threshold.
Notably, although blockingensures sample scalability, party scalability remains a challenge in PPRL.
Therefore, we focus on thelinkage and training process of two parties and provide an extension to multiple parties in this paper.
[Figure]
Although the training of FedSim does not rely on any specific PPRL framework, our privacy analysisis based on a state-of-the-art PPRL framework FEDERAL [28].
Coordinated by an honest-butcurious server , FEDERAL calculates similarities by comparing Bloom filters [52] generated fromidentifiers.
It is theoretically guaranteed that all the identifiers generate Bloom filters containingsimilar numbers of ones; thus, attackers are hard to distinguish identifiers based on Bloom filters.
Vertical Federated Learning.
In this paragraph, we present a formal definition of VFL betweentwo parties.
Suppose two parties P and S want to cooperate with each other to train a machinelearning model.
P is the primary party holding m samples and labels { x P , y } ≜ { x Pi , y i } mi =1 , S isthe secondary party holding n samples x S ≜ { x Si } ni =1 which can also benefit the machine learningtask.
In order to perform linkage, we assume there are some common features between { x Pi } mi =1and { x Si } ni =1 , i.e., { x Pi } mi =1 = { d Pi , k Pi } mi =1 , { x Si } ni
=
= { d Si , k Si } ni
=1 , where k Pi , k Si are commonfeatures used for linkage and d Pi , d Si are remaining features used for training with dimension l P , l S .
We denote d P ≜ { d Pi } mi =1 , d S ≜ { d Si } ni =1 for simplicity.
Our goal is to enable party P to exploit x Sto train a model that minimizes the global loss.
Formally, we aim to optimize the following formula:
[Equation]
where L ( · ) is the loss function, f ( · ) is the VFL model, and λ Ω( θ ) is the regularization term.
SplitNN.
Most vertical federated learning algorithms only support simple models like logisticregression [24] which are ineligible to handle many real-world applications.
Therefore, we adopt
SplitNN [53] which is a popular and state-of-the-art VFL algorithm that supports neural networks.
The main idea of SplitNN is to split a model into multiple parties and conduct training by transferringgradients and intermediate outputs across parties.
As shown in Figure 2a, a global model is split intoan aggregation model θ agg and H local models θ P , { θ S u | u ∈ [1 , H − 1] } , where H is the numberof parties.
In each iteration, the outputs of local models are calculated by forward propagation andsent to the primary party P 0 which holds the labels.
After concatenating the outputs of local models,
P continues forward propagation to derive the final prediction ˆ y and the loss ℓ .
Then, P performsback-propagation until the input of the aggregation model.
The gradients w.r.t.
the outputs of eachlocal model are calculated and sent to the corresponding secondary party.
Finally, all H parties finishback-propagation with the gradients.
Baseline Approaches
By extensively reviewing the existing VFL approaches, we find that all the existing approachesonly use exactly matched (Exact) or most similar (Top1Sim) pairs of data records in the training.
We denote these approaches that separate the linkage and training and as separated approaches .
To design a coupled approach that effectively exploits linkage information, we analyze the maindrawback of existing algorithms and some proposed baseline approaches.
Exact [7, 18, 25, 34, 35, 56, 57]/Top1Sim [21, 26, 38, 39].
(Figure 2b(a,b))
These baseline separatedapproaches that only link exactly matched or most similar pairs neglect information of less similarbut useful pairs, resulting in a poor performance of VFL model.
This phenomenon is particularly evident on vertical distributed datasets with fuzzy common features (e.g., two examples in Figure 1).
To minimize this information loss, more pairs of records should be considered.
AvgSim.
(Figure 2b(d))
We propose AvgSim as a baseline separated approach that considers multiplepairs of records.
Specifically, each data record in x Pi party P is linked with its K most similardata records in party S.
The prediction of x Pi is an average of prediction when linking x Pi with its
K nearest x Sj .
Though considering more pairs, AvgSim overemphasizes the pairs with mediumsimilarities, leading to redundant noise added to the model.
Unaware of the similarity of each pair,the model is unable to filter out this redundant noise.
FeatureSim.
(Figure 2b(c))
We propose FeatureSim as a baseline coupled approach that adoptssimilarities to the training.
Each data record in x Pi party P is linked with its K most similar datarecords in party S, and the similarity between each pair is appended to the record.
Nonetheless,similarity, which contains critical linkage information, is treated equally to other features.
Thisdrawback limits the ability of VFL models to extract information from similarities, thus affecting theoverall performance.
Such impact can be significant according to our experiments.
Based on the above analyses, we clearly observe that record linkage should be coupled with thedesign of VFL by taking advantage of record similarity not only during the linkage procedure butalso during the training procedure.
Meanwhile, according to the analysis on AvgSim, more advancedmodels are needed to merge the outputs of linked pairs with K largest similarities.
These analysesmotivate our design of a coupled framework of linkage and training.
Our Approach: FedSim
Our approach has two components: soft linkage and similarity-based VFL.
In soft linkage, afterfinding top- K similar pairs by existing PPRL methods, the server preprocesses the similaritiesby normalization and Gaussian noise perturbation.
The scale of noise can be determined by aconstant bound τ of the attacker’s success rate, which is further analyzed in Section 5.
Then, thealigning information and aligned similarities are sent to party P or S which will align the data recordsaccordingly.
In similarity-based VFL, we design a model (as shown in Figure 3) by adding additionalcomponents around SplitNN to effectively exploit similarities.
Finally, this model, taking the aligneddata records and aligned similarities as input, is trained by back-propagation like SplitNN.
Althoughwe use SplitNN as an example, the idea of soft linkage and similarity-based learning can be extendedto other federated learning algorithms.
4.1 Soft Linkage
Soft linkage, following traditional PPRL in “blocking” and “comparison” steps, directly outputs thesimilarities with normalization and noise addition without performing “classification” step.
Same asmany existing PPRL approaches [27–29, 51], we assume there exists an honest-but-curious servercoordinating the linkage process.
Specifically, soft linkage, taking k P and k S as input, outputs thealignment information (i.e., indices that indicate the order of records) and similarities.
First, eachsample x Pi in party P is linked with samples containing K most similar identifiers in party S. Thesimilarities between identifiers of these linked pairs are calculated by a PPRL protocol.
To fully utilizex S for each sample x Pi , K should be large enough to ensure that all the pairs which may benefit themodel are included.
Notably, setting a large K incurs negligible performance loss according to ourexperiments in Appendix D.2.
Second, the server calculates the raw similarities ρ ij between theselinked pairs; the raw similarity ρ ij is defined as normalized negative distance.
Formally,
[Equation]
where µ 0 and σ 0 are the mean and standard variance of all negative distances − dist( k Pi , k Sj ) .
Toprevent the attacker from guessing the vectors k Pi or k Sj from similarities (further discussed in
Section 5), we add Gaussian noise of scale σ to each ρ ij .
Formally, for ∀ i ∈ [1 , m ] , ∀ j ∈ [1 , K ]
[Equation]
For simplicity, we denote s i ≜ { s ij } Kj =0 and s ≜ { s i } mi =0 .
After the similarities are calculated, the server directly sends the similarities s to party P and sendsthe aligning information (i.e., indices that indicate orders) to both parties.
Finally, parties P and S align their samples or similarities according to the aligning information to ensure that each x Pi , x
Siand s i refer to the same sample.
4.2 Similarity-Based VFL
[Figure]
The training process is summarized in Algorithm 1and the model structure is shown in Figure 3.
Asdiscussed in Section 7, FedSim is designed on topof SplitNN [53], which makes preliminary predictions o
i (with l m dimensions) for each d Pi and its
K neighbors d Si (lines 4-7).
Specifically, we addthree gates (weight gate, merge gate, sort gate)around SplitNN and train the whole model similarly to SplitNN.
The main function of these gatesis to effectively exploit similarities s i to merge thepreliminary outputs
o i into the final prediction ˆ y i .
Weight Gate.
One straightforward idea is thatmore similar pairs of samples contribute more tothe performance.
Thus, the rows in o i with highersimilarities should be granted a larger weight.
Directly multiplying the similarities to o i is inappropriate since the values of similarities do notdirectly represent the importance of records.
Therefore, in weight gate, we use similarity model ,a simple neural network with one-dimensional input and one-dimensional output, to non-linearlymap the similarities to weights w i which indicates the importance of the record (line 8).
Then, theweighted outputs are calculated as a matrix multiplication (line 9).
Merge Gate.
The merge gate contains a merge model to aggregate the information in o ′ i (line 11).
For tasks that only require a linear aggregation, setting merge model as an “average over rows” of o ′ iis sufficient to obtain a promising result.
However, for tasks that require a non-linear aggregation,a neural network is required to be deployed as a merge model.
In this case, although merge modelcan be implemented by a multi-layer perceptron (MLP) which takes flattened o ′ i as input, such anapproach usually leads to over-fitting due to a large number of parameters.
Meanwhile, the flattenoperation causes information loss because merge model is unaware of which K features in theflattened o ′
i correspond to the same output feature in o ′ i .
Considering these two factors, we set merge
// Local model FP
// Similarity
// Merge and similarity model BP
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
// Local model BP model as a 2D convolutional neural network (CNN) with kernel size k conv × 1 .
o ′
i , containing Koutput vectors with l m dimensions, is regarded as a 2D input of size K × l m for the CNN merge gate.
Then, the merge gate effectively merges these output vectors with close similarities.
Sort Gate.
The sort gate is an optional but sometimes crucial module depending on the property ofthe chosen merge model.
For merge models that are insensitive to the order of o ′ i (e.g., averaging overrows), sorting is not needed because the order does not affect the output of merge model.
However,for merge models that are sensitive to the order of o ′ i (e.g., neural networks), inconsistent order offeatures can incur irregular sharp gradients which makes merge model hard to converge.
Thus, o ′ ishould be sorted by similarities (line 10) to stabilize the updates on merge model.
Also, grouping thepairs with close similarities together helps the merge gate effectively aggregate the information.
The whole model with SplitNN and three gates is performs back-propagation by transferring gradientslike SplitNN (lines 12, 14).
After gradients are calculated, all the parameters are updated by gradientdescent (lines 13, 15).
According to the experiment in Appendix D.4, FedSim incurs longer butacceptable training time compared to Exact and Top1Sim.
During the inference, for each data recordin the primary party, K most similar data records in secondary parties are linked.
These linked datarecords are fed into the FedSim model to derive the final prediction.
Similar to many existing studies [7, 21, 38, 57] in VFL, we mainly focus on the two-party settingwhich has many real-world applications (e.g., bank and fintech company [57]).
Meanwhile, we alsosupport an extension to the multiple-party setting as elaborated in Appendix B.
4.3 Improvement Estimation
[Figure]
In this subsection, we propose a data-linkage-based metric to estimate the improvement of FedSim over baselines (i.e., AvgSim,
Top1Sim, Exact) without training.
Calculating all the similaritiesbetween x Pi and every { x Sj } nj =1 , we can plot the sorted similaritiesof each pair as a curve.
For example, Figure 4 displays the curveof AvgSim with top-K neighboring records.
The performance ofthese baselines is impeded by two main factors: 1) lost information- some pairs of samples with small similarities are neglected.
Forexample, in Figure 4, the samples above the threshold are neglected;2) redundant information - some pairs of samples with smaller similarities are treated equally to those pairs with large similarities.
For example, in Figure 4, some samples below the threshold withmedium similarities are overestimated.
To jointly estimate these twofactors, we intuitively assume the information contained in each pairis proportional to its similarity; then, the lost information and the redundant information can beestimated by the similarities.
We formally define the metric as follows.
Definition 1.
Given a data linkage between x P ≜ { x Pi } mi =1 and x S ≜ { x Sj } nj =1 , for a data record x Piin P, a VFL algorithm F divide the indices of n records in S into matches and non-matches, denotedas S matches and S non-matches , respectively.
Denote s ij as the scaled similarity between x P i and x S j , theimprovement of FedSim on F for x Pi is defined as ∆ i ( F ) ≜ (cid:80) j ∈ S matches s ij + (cid:80) j ∈ S non-matches (1 − s ij ) ;the overall improvement of FedSim on F is defined as ∆( F ) = 1 m (cid:80) i = m i =1 ∆ i ( F ) .
As shown in the experiments, this metric can effectively estimate the improvement of FedSimcompared with the other baselines without training.
Privacy
5.1 Overall Analysis
The shared information in FedSim includes: 1) similarities s shared to party P; 2) intermediate resultsin SplitNN shared to party P; 3) intermediate results in PPRL shared to the server.
The intermediateresults in SplitNN and PPRL are respectively studied in [54] and [28] (see Section 2 for details),both of which are orthogonal to this paper.
Therefore, we mainly study the privacy risk caused bysimilarities s .
According to Equation 1 and 2, each similarity s ij is calculated from k Pi and k Sj .
Thus, party P, as a potential attacker, may try to reversely predict k Sj from some identifiers { k Pi | i ∈ Q } andthe corresponding similarities { s ij | i ∈ Q } , where Q is the set of indices of used identifiers.
As discussed in Section 2, our privacy analysis is based on the assumption that k P and k S are Bloomfilters, the distances of which are integers.
Our analysis focuses on a greedy attacker who firstpredicts the most likely distance from each s ij and then predicts the most likely Bloom filter from thepredicted distance.
Assuming the attacker already knows the scaling parameters µ 0 , σ 0 , we formulatethe attack method as follows.
Attack Method.
To obtain ˆ k Sj as a prediction of k Sj , the attacker 1) predicts a set of normalizednegative distances ˆ ρ ij ( i ∈ Q ) from s ij ( i ∈ Q ) , respectively, by maximum a posteriori (MAP)estimation with Gaussian prior N (0 , 1) , i.e., ˆ ρ ij = arg max ρ ij p ( ρ ij | s ij ) ; 2) calculates each distanceˆ u ij by scaling back ˆ ρ ij with parameters µ 0 , σ 0 , i.e., ˆ u ij = − σ 0 ˆ ρ ij − µ 0 ( i ∈ Q ) ; 3) uniformlyguesses ˆ k Sj from all possible values satisfying ∀ i ∈ Q, dist( k Pi , ˆ k Sj )
=
ˆ u ij which have the sameprobability of being the real k Sj .
Besides the greedy attack, advanced attackers may predict through the probability distribution ofdistances rather than predict through the most likely distance.
Some attackers may even know someside information like the prior distribution of k Sj and employ this side information to launch attacks.
These advanced attacks are further discussed in Appendix E.
5.2 Privacy Guarantee
Although disclosing a Bloom filter to the attacker is not catastrophic since reversely inferring the rawfeatures from the Bloom filters is infeasible as proved in [28], we further prove that the probability ofpredicting the correct k S is always bounded by a small constant related to σ 0 and σ regardless of thechoice of Q if an attacker follows the attack method (Theorem 1).
Theorem 1.
Given a finite set of perturbed similarities s ij ( i ∈ Q ) between | Q | bloom-filtersk Pi ( i ∈ Q ) in party P and one Bloom filter k Sj in party S, if an attacker knows the scaling parametersµ 0 , σ 0 and follows the procedure of the attack method, the probability of the attacker’s predicted
Bloom filter ˆ k
Sj equaling the real Bloom filter k Sj is bounded by a constant τ .
Formally,
[Equation]
where constant τ = erf (cid:16) √ σ 2 +1 2 √ 2 σσ 0 (cid:17) ; erf( · ) is the error function, i.e., erf( x ) =
2 √ π (cid:82) x 0 e − t 2 dt ; event
A : attackers follow the given attack method.
From Theorem 1, we find two factors that affect the attacker’s success rate: 1) noise added to thesimilarity ( σ ); 2) standard variance of Bloom filters ( σ 0 ).
Among these factors, σ 0 determines thelower bound of the success rate because √ σ 2 + 1 / (2 √ 2 σσ 0 ) <
1 / (2 √ 2 σ 0 ) , and σ determines howclose success rates FedSim can guarantee compared to the lower bound.
When σ is large enough,increasing σ helps little with reducing the success rate.
Therefore, to ensure the good privacy of
FedSim, we should first guarantee a large enough variance among the Bloom filters and then adda moderate noise to the similarities.
Taking house dataset (see Section 6.1) as an example whereσ 0 = 21178 .
By setting σ = 0 .
4 , we have τ = 0 .
Considering the 19479 samples in thetraining set of party S, only 0.988 (less than one) Bloom filters are expected to be disclosed.
Experiment
6.1 Experimental Setup
Dataset.
We evaluate FedSim on three synthetic datasets ( sklearn [10], frog [13], boone [14])and five real-world datasets ( house [1, 42], taxi [8, 49], hdb [22, 43], game [11, 23], and song[6, 12]).
The details (e.g. dimension of identifiers) of these datasets are summarized in Appendix C. For each real-world dataset, we collect two public datasets from different real-world parties andconduct VFL on both public datasets.
For each synthetic dataset, we first create a global datasetby generating with sklearn API [10] ( sklearn ) or collecting from public ( frog , boone ).
Then,we randomly select some features as common features and randomly divide the remaining features
[Table]
[Figure]
[Table]
[Figure]
[Table]
noise on similarities equally to both parties. Common features are not used in training for all methods except Combine.
To simulate the real-world applications, for synthetic datasets, we also add different scales σ cf of Gaussian noise to the common features.
Specifically, for each identifier v i , the perturbed identifierv ′
i =
v i
+ N ( σ 2 cf I ) will be used for linkage.
For datasets with numeric identifiers ( house , taxi ,hdb , syn boone , frog ), Euclidean distance is adopted to calculate similarities.
For datasets withstring identifiers ( song , game ), Levenshtein distance is adopted to calculate similarities.
Training.
Similarity model is a multi-layer perceptron (MLP) with one hidden layer.
Merge modelcontains a 2D convolutional layer with k conv × 1 kernel followed by a dropout layer and an MLPwith one hidden layer.
Both the local model and aggregate model in SplitNN are MLPs with onehidden layer.
We adopt LAMB [59], the state-of-the-art large-batch optimizer, to train all the models.
Each dataset is split into training, validation, and test set by 7:1:2.
We run each algorithm five timesand report the mean and standard variance (range is reported instead in figures) of performance onthe test set.
We present root mean square error (RMSE) or R-squared value ( R 2 ) for regression tasksand accuracy for classification tasks.
The choices of hyperparameters are introduced in Appendix C.
Baselines.
We compare FedSim with nine baselines in our experiments.
Besides the four baselines(Exact, Top1Sim, AvgSim, FeatureSim) introduced in Section 3, the remaining baselines include:1)
Solo: only dataset d P is trained; 2) Combine: [ d P , d S ] is trained (only applicable for syntheticdatasets); 3) FedSim-MLP: the CNN in FedSim is changed to an MLP with a similar number ofparameters.
FedSim w/o sort: FedSim without sorting gate.
5) FedSim w/o weight:
FedSimwithout weight gate (similarities are directly regarded as weights).
Notably, Exact is only evaluatedon game and song because no exactly matched identifiers can be found on other datasets.
6.2 Performance
We evaluate the performance of FedSim on three synthetic datasets under different σ cf and fivereal-world datasets.
The results of synthetic datasets are presented in Figure 5a, from which twoobservations can be made.
First, FedSim consistently has better or close performance compared to all the baselines.
Second, FedSim is more robust to the noise on the identifiers.
For example, in frog ,the accuracy of Top1Sim drops to 84% as σ cf = 0 .
2 , while the accuracy of FedSim remains 91%.
The results of real-world datasets are summarized in Table 1.
We also calculate the estimated improvement ∆( Top 1 Sim ) (denoted as ∆ for simplicity) according to our proposed metric.
The relationshipbetween ∆( Top 1 Sim ) and the relative improvement on Top1Sim is presented in Figure 5b.
Threeobservations can be made from the results.
First, FedSim consistently produces the best performance on all the datasets, while the baselines (especially the separated approaches) only havegood performances on specific datasets .
For example, Top1Sim has close performance to FedSimon game , but fails on bike ; AvgSim has close performance to FedSim on bike , but fails on game .
Second, ∆( Top 1 Sim ) is positively correlated with real improvement on Top1Sim, indicating thatthe metric can be effectively used to estimate the improvement of FedSim without training.
Thisalso implies that FedSim can effectively reduce the effect of lost information and redundantinformation as expected .
Third, comparing the performance of removing each component from
FedSim, the sort gate makes the most significant contribution to the performance of FedSim bystabilizing the updates of merge model.
The improvement of weight gate indicates that adjusting thedistribution of similarities can slightly benefit the performance.
Besides, CNN merge gate can alsoslightly improve on MLP merge gate by reducing overfitting.
6.3 Privacy
In this subsection, to study how additional noise on similarities affects the performance of FedSim,we conduct experiments on five real-world datasets (the result of song is included in Appendix D.1due to page limit).
Specifically, string or numeric identifiers are converted to Bloom filters accordingto [28].
The Hamming distances between Bloom filters are used to calculate raw similarities.
Then,given an acceptable success rate τ , a noise scale σ is calculated according to Theorem 1.
Finally,
Gaussian noise with scale σ is added to the raw similarities according to Equation 2.
The resultsare presented in Figure 6.
Exact is not evaluated since few Bloom filters have exactly the samebits.
From Figure 6, we observe that FedSim is robust to the noise on similarities; therefore, theattacking success rate can be reduced to [10 − 4 , 10 − 3 ] without evident performance loss .
Notably,the performance when τ = 1 is not necessarily the same as the performance in Section 6.2 sincesimilarities are calculated based on different distances.
Related Work
Most studies [18, 25, 36, 47, 53] in VFL focus on training and simply assumes record linkage hasbeen done (i.e., the implicit exact linkage on record ID), which is impractical since most real-worldfederated datasets are unlinked.
Some approaches exactly link the identifiers by exact PPRL [7]or private set intersection (PSI) [7, 35, 57].
Nonetheless, these approaches incur performance lossof VFL and are also impractical since the common features of many real-world federated datasetscannot be exactly linked (e.g. GPS location).
[21] greedily links the most similar identifiers in PPRL,which negatively impacts performance since some beneficial pairs with relatively low similarity maybe neglected.
[38, 39] explore the impact of record linkage on the performance of VFL, which is alsoadopted by [26].
However, all of them focus only on the most similar identifiers and assume there isa one-to-one mapping between the data records of two parties, which is not always true in practice.
Current VFL frameworks support various machine learning models including linear regression [17],logistic regression [24], support vector machine [33], gradient boosting decision trees [7, 57].
FDML[25] supports neural networks but it requires all the parties to hold labels.
SplitNN [53] focuses onneural networks and provides a new idea of collaborative learning where the model is split and heldby multiple parties.
Since we study the scenario where only one party holds the labels and want tosupport commonly used neural networks, we build FedSim on top of SplitNN.
Conclusion
In this paper, we propose FedSim, a novel VFL framework based on similarities to boost theperformance of VFL by directly utilizing the similarities calculated in PPRL and skipping theclassification process.
We also theoretically analyze the additional privacy risk introduced by sharingsimilarities and provide a bound for the success rate of an intuitive attack.
In our experiment, FedSimconsistently outperforms all the baselines.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See the last paragraph of Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix G.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix F.
[Table] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
[Table]
them?
[Yes]2.
If you are including theoretical results...
[Table]
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 5.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes areincluded in the supplementary materials.
The data are all publicly available; the link orcitations of data are presented in Appendix C.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes] All the details including data splits and hyperparameters areintroduced in Section 6.1 and Appendix C.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The variance is reported in all tables and figures.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the “hardware” paragraph ofAppendix C.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.1 andAppendix C.
[Table]
(b) Did you mention the license of the assets?
[Yes] See the “license” paragraph ofAppendix C.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]Our codes are included in supplementary materials.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[Yes] See the “Consent of dataset” paragraph of Appendix F
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content? [Yes] See the “Personally identifiable information”of Appendix F.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
Our research is not related to human subjects.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
Our research is not related to human
[Table]
subjects.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
Our research is not related to human subjects.
[Table]