A Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learning
Abstract
Federated learning is a learning paradigm to enable collaborative learning across different parties without revealing raw data.
Notably, vertical federated learning (VFL), where parties share the same set of samples but only hold partial features, has a wide range of real-world applications.
However, most existing studies in VFL disregard the “record linkage” process.
They design algorithms either assuming the data from different parties can be exactly linked or simply linking each record with its most similar neighboring record.
These approaches may fail to capture the key features from other less similar records.
Moreover, such improper linkage cannot be corrected by training since existing approaches provide no feedback on linkage during training.
In this paper, we design a novel coupled training paradigm, FedSim, that integrates one-to-many linkage into the training process.
Besides enabling VFL in many real-world applications with fuzzy identifiers,
FedSim also achieves better performance in traditional VFL tasks.
Moreover, we theoretically analyze the additional privacy risk incurred by sharing similarities.
Our experiments on eight datasets with various similarity metrics show that FedSim outperforms other state-of-the-art baselines.
The codes of FedSim are available at https://github.com/Xtra-Computing/FedSim .
Introduction
Federated learning is a collaborative learning framework to train a model from distributed datasets with privacy guarantees.
A commonly existing and widely studied scenario of federated learning is vertical federated learning [32, 62] (VFL), where multiple parties sharing the same set of samples have different sets of features.
We focus on the setting where only one party holds the labels like most of the studies [33, 39, 60].
The party holding labels is named primary party ; the parties without labels are named secondary parties .
In VFL, the features that exist on multiple parties are called common features .
The vector of common features in a data record is called the identifier of the record.
Existing studies [22, 27, 39, 40] formulate VFL as two separated processes: linkage and training.
In the linkage process, the datasets on different parties are linked according to the identifiers.
In the training process, these distributed but linked data records are trained by VFL algorithms.
Specifically, in the linkage process of existing studies, each data record is linked to a data record with the exactly matched or the most similar identifier (i.e., one-to-one linkage).
Nonetheless, according to our study on all the completed projects in German Record Linkage Center [4] (GRLC), only 27.3% of the record-linkage applications are linked on exact identifiers.
The other projects rely on fuzzy identifiers such as addresses, in which one-to-one linkage can seriously impair the accuracy of the VFL model for the following two reasons.
First, only linking the records with top similarity (i.e., one-to-one linkage) does not necessarily capture the key features, which can be demonstrated by two real-world applications.
1) Considering the VFL for price prediction between a real estate company and a house leasing company linked by
[Figure]
GPS locations of houses, as shown in Figure 1(i), the closest house to house A (purple) may not be in the same price level as A . 2)
Considering the VFL between steam games and IGN games linked by the game titles, as shown in Figure 1(ii), linking not only the exactly matched game but also other games in the same series intuitively benefits game recommendation tasks.
Even in applications where identifiers can be exactly matched (e.g., identifiers are ID), if some features are missing or biased, one-to-one linkage prevents the training process from enhancing these features from other similar records.
These applications commonly exist in practice according to our investigation.
An opposite extreme case is to link each record with all the records in another party ( one-to-all linkage), which keeps all the information but is too expensive for both linkage and training.
Therefore, a one-to-many linkage approach is needed as a balance between efficiency and performance.
Second, separating the linkage from the training also harms the performance of VFL.
In existing
VFL approaches, since the linkage process cannot obtain any feedback from the training process, the linkage is conducted with the goal of finding true-matched pairs of records instead of finding the pairs that reduce the training loss.
Hence, an integrated VFL framework that conducts one-to-many linkage under the guide of training is desired.
To address these two drawbacks, we link each data record to the records with top- K similar identifiers in another party and design a coupled framework of linkage and training.
Our main challenge is to effectively exploit these linked pairs and their similarities to boost the performance of VFL.
To tackle this challenge, we propose a similarity-based coupled VFL framework FedSim on the top of SplitNN [56] which is a VFL algorithm for neural networks.
In FedSim, similarity is a dominant feature that determines the order and the weight of each pair of linked records.
The weights (i.e., impact) of similarities are also adjusted in each training iteration.
After the training, each similarity is mapped to a weight based on its contribution to reducing the loss.
Furthermore, to address the additional data privacy issue incurred by FedSim, we first add Gaussian noise to the similarities and analyze the privacy of FedSim under differential privacy [15].
Our analysis suggests that differential privacy that provides rigorous guarantees for every individual record regardless of the type of attack is impractical for FedSim.
As an alternative, we propose an intuitive greedy attack to infer identifiers from similarities, followed by a theoretical bound of the probability of its success.
Our main contributions can be summarized below.
We propose a novel asymmetric training paradigm, named FedSim, and a training-free metric to estimate the improvement of FedSim on baseline approaches; 2) we analyze the privacy of FedSim under differential privacy; 3) we propose a greedy attack on FedSim and the corresponding defense method that can theoretically bound the success rate of this attack; 4) we conduct extensive experiments on three synthetic datasets and five real-world datasets, which indicates that FedSim outperforms state-of-the-art baselines.
Preliminaries
Privacy-Preserving Record Linkage.
Privacy-preserving record linkage (PPRL) [55] aims to link the data records from two parties that refer to the same sample without revealing real identifiers.
Most
Local model 𝜃 𝑃
Local model 𝜃 𝑆
Aggregaete model 𝜃 𝑎𝑔𝑔 ො𝐲
Data records on 𝑃
Data records on 𝑆𝑓 𝜃 𝑢 (𝐱 (𝑢) )
𝜕ℓ 𝜕𝑓 𝜃 𝑢 (𝐱 (𝑢) ) 𝒚 ℓ(𝒚,ෝ𝒚) 𝑓 𝜃 0 (𝐱 (0) )
𝜕ℓ 𝜕𝑓 𝜃 0 (𝐱 (0) )
Loss function
Local model 𝜃 𝑆 𝑢
Data records on 𝑆 𝑢 …
Primary Party
Secondary Parties (i) Forward and back-propagation of SplitNN 𝑑 𝑖𝑃 𝐝 𝑖𝑆
SplitNN ෝ𝑦 𝑖
Each record in 𝑃
Top1 nearest neighbor of 𝑑 𝑖𝑃 𝑑 𝑖𝑃 𝐝 𝑖𝑆
𝐾 nearest neighbors of 𝑑 𝑖𝑃
SplitNN 𝐨 𝑖 𝐾 predictions
Average Rows ෝ𝑦 𝑖
Each record in 𝑃 0 𝐝 𝑖𝑆 𝐬 𝑖 𝐾 similarities 𝐾 nearest neighbors of 𝑑 𝑖𝑃 (d) AvgSim (a)Top1Sim (c) FeatureSim 𝐝 𝑖𝑆 Exactly matched records of 𝑑
[Figure]
[Figure]
PPRL methods [28, 53, 55] consist of three main steps: blocking, comparison, and classification.
First, in the blocking step, data records that are unlikely to be linked are pruned to reduce the number of comparisons.
Then, in the comparison step, a similarity between identifiers is computed for each candidate pair of data records.
Finally, in classification, each candidate pair is classified as “matches” or “non-matches”, which is usually done by a manually set threshold.
Notably, although blocking ensures sample scalability, party scalability remains a challenge in PPRL.
Therefore, we focus on the linkage and training process of two parties and provide an extension to multiple parties in this paper.
Although the training of FedSim does not rely on any specific PPRL framework, our privacy analysis is based on a state-of-the-art PPRL framework FEDERAL [29].
Coordinated by an honest-but-curious server, FEDERAL calculates similarities by comparing Bloom filters [55] generated from identifiers.
It is theoretically guaranteed that all the identifiers generate Bloom filters containing similar numbers of ones; thus, attackers are hard to distinguish identifiers based on Bloom filters.
More detailed background of FEDERAL is introduced in Appendix I.
Vertical Federated Learning.
In this paragraph, we present a formal definition of VFL between two parties.
Suppose two parties P and S want to cooperate with each other to train a machine learning model.
P is the primary party holding m samples and labels { x P , y } ≜ { x Pi , y i } mi =1 , S is the secondary party holding n samples x S ≜ { x Si } ni =1 which can also benefit the machine learning task.
In order to perform linkage, we assume there are some common features between { x Pi } mi = and { x Si } ni =1 , i.e., { x Pi } mi =1 = { d Pi , k Pi } mi =1 , { x Si } ni
=
= { d Si , k Si } ni
=1 , where k Pi , k Si are common features used for linkage and d Pi , d Si are remaining features used for training with dimension l P , l S .
We denote d P ≜ { d Pi } mi =1 , d S ≜ { d Si } ni =1 for simplicity.
Our goal is to enable party P to exploit x S to train a model that minimizes the global loss.
Formally, we aim to optimize the following formula:
[Equation]
where L ( · ) is the loss function, f ( · ) is the VFL model, and λ Ω( θ ) is the regularization term.
SplitNN.
Most vertical federated learning algorithms only support simple models like logistic regression [25] which are ineligible to handle many real-world applications.
Therefore, we adopt
SplitNN [56] which is a popular and state-of-the-art VFL algorithm that supports neural networks.
The main idea of SplitNN is to split a model into multiple parties and conduct training by transferring gradients and intermediate outputs across parties.
As shown in Figure 2(i), a global model is split into an aggregation model θ agg and H local models θ P , { θ S u | u ∈ [1 , H − 1] } , where H is the number of parties.
In each iteration, the outputs of local models are calculated by forward propagation and sent to the primary party P 0 which holds the labels.
After concatenating the outputs of local models,
P continues forward propagation to derive the final prediction ˆ y and the loss ℓ .
Then, P performs back-propagation until the input of the aggregation model.
The gradients w.r.t.
the outputs of each local model are calculated and sent to the corresponding secondary party.
Finally, all H parties finish back-propagation with the gradients.
Baseline Approaches
By extensively reviewing the existing VFL approaches, we find that all the existing approaches only use exactly matched (Exact) or most similar (Top1Sim) pairs of data records in the training.
We denote these approaches that separate the linkage and training and as separated approaches .
According to our detailed study in Appendix D, only 27.3% of the record-linkage applications in
GRLC based on exact identifiers are suitable for these approaches.
To design a coupled approach that effectively exploits linkage information, we analyze the main drawback of existing approaches and some baseline coupled frameworks.
Exact [7, 18, 26, 35, 36, 59, 60]/Top1Sim [22, 27, 39, 40].
(Figure 2(ii)(a,b))
These baseline separated approaches that only link exactly matched or most similar pairs neglect information of less similar but useful pairs, resulting in a poor performance of VFL model in many real applications such as those in GRLC.
AvgSim.
(Figure 2(ii)(d))
We propose AvgSim as a baseline coupled approach that considers multiple pairs of records.
Specifically, each data record in x Pi party P is linked with its K most similar data records in party S.
The prediction of x Pi is an average of prediction when linking x Pi with its
K nearest x Sj .
Though considering more pairs, AvgSim overemphasizes the pairs with medium similarities, leading to redundant noise added to the model.
Unaware of the similarity of each pair, the model is unable to filter out this redundant noise.
FeatureSim.
(Figure 2(ii)(c))
We propose FeatureSim as a coupled baseline approach that adopts similarities to the training.
Each x Pi in party P is linked with its K most similar data records in party
S, and the similarity between each pair is appended to the record.
Nonetheless, similarity, which contains critical linkage information, is treated equally to other features.
This drawback limits the ability of VFL models to extract information from similarities, thus affecting the overall performance.
Such an impact can be significant according to our experiments.
Based on the above analyses, we clearly observe that the record linkage should be coupled with the design of VFL by taking advantage of record similarity not only during the linkage procedure but also during the training procedure.
Meanwhile, according to the analysis on AvgSim, more advanced models are needed to merge the outputs of linked pairs with K largest similarities.
These analyses motivate our design of FedSim as a coupled framework of linkage and training.
Our Approach: FedSim
Our approach has two components: soft linkage and similarity-based VFL.
In soft linkage, after finding top- K similar pairs by existing PPRL methods, the server preprocesses the similarities by normalization and Gaussian noise perturbation.
The scale of noise can be determined by a constant bound τ of the attacker’s success rate, which is further analyzed in Section 5.
Then, the aligning information and aligned similarities are sent to party P or S which will align the data records accordingly.
In similarity-based VFL, we design a model (as shown in Figure 3) by adding additional components around SplitNN to effectively exploit similarities.
Finally, this model, taking the aligned data records and aligned similarities as input, is trained by back-propagation like SplitNN.
Although we use SplitNN as an example, the idea of soft linkage and similarity-based learning can be extended to other federated learning algorithms.
4.1 Soft Linkage
Soft linkage, following traditional PPRL in “blocking” and “comparison” steps, directly outputs the similarities with normalization and noise addition without performing the “classification” step.
Same as many existing PPRL approaches [28–30, 54], we assume there exists an honest-but-curious server coordinating the linkage process.
Specifically, soft linkage, taking k P and k S as input, outputs the alignment information (i.e., indices that indicate the order of records) and similarities.
First, each sample x Pi in party P is linked with samples containing K most similar identifiers in party S.
The similarities between identifiers of these linked pairs are calculated by a PPRL protocol.
To fully utilize x S for each sample x Pi , K should be large enough to ensure that all the pairs which may benefit the model are included.
Notably, setting a large K hardly leads to overfitting according to our experiments in Appendix F.2.
Second, the server calculates the raw similarities ρ ij between these linked pairs; the raw similarity ρ ij is defined as normalized negative distance.
Formally,
[Equation]
where µ 0 and σ 0 are the mean and standard variance of all negative distances − dist( k Pi , k Sj ) .
To prevent the attacker from guessing the vectors k Pi or k Sj from similarities (further discussed in
Section 5), we add Gaussian noise of scale σ to each ρ ij .
Formally, for ∀ i ∈ [1 , m ] , ∀ j ∈ [1 , K ] s ij =
[Equation]
ij
[Equation]
N
[Equation] [Equation]
[Equation]
For simplicity, we denote s i ≜ { s ij } Kj =0 and s ≜ { s i } mi =0 .
After the similarities are calculated, the server directly sends the similarities s to party P and sends the aligning information (i.e., indices that indicate orders) to both parties.
Finally, parties P and S align their samples or similarities according to the aligning information to ensure that each x Pi , x Si
and s i refer to the same sample.
4.2 Similarity-Based VFL
[Figure]
The training process is summarized in Algorithm 1 and the model structure is shown in Figure 3.
As discussed in Section 7, FedSim is designed on top of SplitNN [56], which makes preliminary predictions o
i (with l m dimensions) for each d Pi and its K neighbors
d Si (lines 4-7).
Specifically, we add three gates (weight gate, merge gate, sort gate) around SplitNN and train the whole model similarly to SplitNN.
The main function of these gates is to effectively exploit similarities s i to merge the preliminary outputs
o i into the final prediction ˆ y i .
Weight Gate.
One straightforward idea is that more similar pairs of samples contribute more to the performance.
Thus, the rows in o i with higher similarities should be granted a larger weight.
Directly multiplying the similarities to o i is inappropriate since the values of similarities do not directly represent the importance of records.
Therefore, in weight gate, we use similarity model , a simple neural network with one-dimensional input and one-dimensional output, to non-linearly map the similarities to weights w i which indicates the importance of the record (line 8).
Then, the weighted outputs are calculated as a matrix multiplication (line 9).
Merge Gate.
The merge gate contains a merge model to aggregate the information in o ′ i (line 11).
For tasks that only require a linear aggregation, setting the merge model as an “average over rows” of o ′ i is sufficient to obtain a promising result.
However, for tasks that require a non-linear aggregation, a neural network is required to be deployed as a merge model.
In this case, although the merge model can be implemented by a multi-layer perceptron (MLP) which takes flattened o ′ i as input, such an approach usually leads to over-fitting due to a large number of parameters.
Meanwhile, the flatten operation causes information loss because the merge model is unaware of which K features in the flattened o ′
i correspond to the same output feature in o ′ i .
Considering these two factors, we set the merge model as a 2D convolutional neural network (CNN) with kernel size k conv × 1 .
o ′
i , containing
K output vectors with l m dimensions, is regarded as a 2D input of size K × l m for the CNN merge gate.
Then, the merge gate effectively merges these output vectors with close similarities.
Sort Gate.
The sort gate is an optional but sometimes crucial module depending on the property of the chosen merge model.
For merge models that are insensitive to the order of o ′ i (e.g., averaging over rows), sorting is not needed because the order does not affect the output of the merge model.
However, for merge models that are sensitive to the order of o ′ i (e.g., neural networks), inconsistent order of features can incur irregular sharp gradients which makes the merge model hard to converge.
Thus, o ′ i should be sorted by similarities (line 10) to stabilize the updates on the merge model.
Also, grouping the pairs with close similarities together helps the merge gate effectively aggregate the information.
[Table]
t θ aggt , θ Pt +
=
θ
Pt − η t g Pt , θ simt +
= θ simt − η t g simt , θ
mt +
= θ mt − η t g mt ;
// Local model BP
The whole model with SplitNN and three gates performs back-propagation by transferring gradients like SplitNN (lines 12, 14).
After gradients are calculated, all the parameters are updated by gradient descent (lines 13, 15).
According to the experiment in Appendix F.4, FedSim incurs longer but acceptable training time compared to Exact and Top1Sim.
During the inference, for each data record in the primary party, K most similar data records in secondary parties are linked.
These linked data records are fed into the FedSim model to derive the final prediction.
More insights into the weight gate and merge gate are elaborated by visualization in Appendix F.5.
4.3 Improvement Estimation
[Figure]
Similar to many existing studies [7, 22, 39, 60] in VFL, we mainly focus on the two-party setting which has many real-world applications (e.g., bank and fintech company [60]).
Meanwhile, we also support an extension to the multiple-party setting as elaborated in
Appendix C.
In this subsection, we propose a data-linkage-based metric to estimate the improvement of FedSim over baselines (i.e., AvgSim,
Top1Sim, Exact) without training.
Calculating all the similarities between x Pi and every { x Sj } nj =1 , we can plot the sorted similarities of each pair as a curve.
For example, Figure 4 displays the curve of AvgSim with top-K neighboring records.
The performance of these baselines is impeded by two main factors: 1) lost information - some pairs of samples with small similarities are neglected.
For example, in Figure 4, the samples above the threshold are neglected; 2) redundant information - some pairs of samples with smaller similarities are treated equally to those pairs with large similarities.
For example, in Figure 4, some samples below the threshold with medium similarities are overestimated.
To jointly estimate these two factors, we intuitively assume the information contained in each pair is proportional to its similarity; then, the lost information and the redundant information can be estimated by the similarities.
We formally define the metric as follows.
Definition 1.
Given a data linkage between x P ≜ { x Pi } mi =1 and x S ≜ { x Sj } nj =1 , for a data record x Pi in P, a VFL algorithm F divide the indices of n records in S into matches and non-matches, denoted as S matches and S non-matches , respectively.
Denote s ij as the scaled similarity between x Pi and x Sj , the improvement of FedSim on F for x Pi is defined as ∆ i ( F ) ≜ (cid:80) j ∈ S matches (1 − s ij )
+ (cid:80) j ∈ S
non-matches s ij ; the overall improvement of FedSim on F is defined as ∆( F ) = 1 m (cid:80) mi =
1 ∆ i ( F ) .
As shown in Figure 5(ii), this metric can effectively estimate the improvement of FedSim compared with the other baselines in the experiments.
Privacy
The shared information in FedSim includes:
(1) similarities s shared to party P; (2) intermediate results in SplitNN shared to party P; (3) intermediate results in PPRL shared to the server.
The intermediate results in SplitNN and PPRL are respectively studied in [57] and [29] (see Sectionfor details), both of which are orthogonal to this paper.
Therefore, we mainly study the privacy risk caused by similarities s .
5.1 Differential Privacy
According to Equation 1 and 2, each similarity s ij is calculated from k Pi and k Sj .
We formally define this procedure as G below.
Procedure G : Take k S as the input.
Each k Pi ∈ k P is linked to multiple k Sj ∈ k S ; the similarities between k Pi and k Sj are calculated by s ij = − ∥ k Pi − k Sj ∥ 2 + µ 0 σ+ N (0 , σ 2 ) , where N (0 , σ 2 ) refers to
Gaussian distribution with variance σ 2 .
Output the similarities s .
Theorem 1.
Suppose ε > 0 and 0 < δ < 1 / 2 − e − 3 ε / √ 2 πε .
Suppose the size of k P is n × β .
If procedure G is ( ε, δ ) -DP, then σ ≥ ∆ G / √ 2 ε , where ∆ G = n · max (cid:110)(cid:12)(cid:12)(cid:12) 1+ µ 0 σ 0 (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) − 1+ µ 0 σ 0 (cid:12)(cid:12)(cid:12)(cid:111) .
The proof of Theorem 1 is included in Appendix A.1.
As observed from Theorem 1, the noise scale σ increases linearly by n .
The large scale of noise derived from differential privacy would seriously affect the performance of FedSim.
Hence, we further analyze the privacy risk of FedSim against specific attacks.
5.2 Privacy Against Greedy Attacks
In this subsection, our analysis focuses on a greedy attacker who first predicts the most likely distance from each s ij and then predicts the most likely Bloom filter from the predicted distance.
Assuming the attacker already knows the scaling parameters µ 0 , σ 0 , we formulate the attack method as follows.
Attack Method.
To obtain ˆ k Sj as a prediction of k Sj , the attacker 1) predicts a set of normalized negative distances ˆ ρ ij ( i ∈ Q ) from s ij ( i ∈ Q ) , respectively, by maximum a posteriori (MAP) estimation with Gaussian prior N (0 , 1) , i.e., ˆ ρ ij = arg max ρ ij p ( ρ ij | s ij ) ; 2) calculates each distance ˆ u ij by scaling back ˆ ρ ij with parameters µ 0 , σ 0 , i.e., ˆ u ij = − σ 0 ˆ ρ ij − µ 0 ( i ∈ Q ) ; 3) uniformly guesses ˆ k Sj from all possible values satisfying ∀ i ∈ Q, dist( k Pi , ˆ k Sj )
=
ˆ u ij which have the same probability of being the real k Sj .
Besides the greedy attack, advanced attackers may predict through the probability distribution of distances rather than predict through the most likely distance.
Some attackers may even know some side information like the prior distribution of k Sj and employ this side information to launch attacks.
These advanced attacks are further discussed in Appendix B.
Although not satisfying differential privacy, we prove that the success probability of greedy attacks is always bounded by a small constant related to σ 0 and σ regardless of the choice of Q if an attacker follows the attack method (Theorem 2).
Theorem 2.
Given a finite set of perturbed similarities s ij ( i ∈ Q ) between | Q | bloom-filters k Pi ( i ∈ Q ) in party P and one Bloom filter k Sj in party S, if an attacker knows the scaling parameters µ 0 , σ 0 and follows the procedure of the attack method, the probability of the attacker’s predicted
Bloom filter ˆ k
Sj equaling the real Bloom filter k Sj is bounded by a constant τ .
Formally,
[Equation]
where constant τ = erf (cid:16) √ σ 2 +1 2 √ 2 σσ 0 (cid:17) ; erf( · ) is the error function, i.e., erf( x ) =
2 √ π (cid:82) x 0 e − t 2 dt ; event
A : attackers follow the given attack method.
The proof of Theorem 2 is included in Appendix A. From Theorem 2, we find two factors that affect the attacker’s success rate: 1) noise added to the similarity ( σ ); 2) standard variance of
Bloom filters ( σ 0 ).
Among these factors, σ 0 determines the lower bound of the success rate because √ σ 2 + 1 / (2 √ 2 σσ 0 ) <
1 / (2 √ 2 σ 0 ) , and σ determines how close success rates FedSim can guarantee compared to the lower bound.
When σ is large enough, increasing σ helps little with reducing the success rate.
Therefore, to ensure the good privacy of FedSim, we should first guarantee a large enough variance among the Bloom filters and then add a moderate noise to the similarities.
Taking house dataset (see Section 6.1) as an example where µ 0 = − 46237 .
78 , σ 0 = 21178 .
Letting δ = 10 − 5 , σ = 0 .
4 , the differential privacy parameter ε = 2 .
96 × 10 9 implies that there is almost no differential privacy guarantee at all.
Nonetheless, the attacking success rate τ = 1 .
94 × 10 −suggests that the privacy risk from certain attacks is very low.
Considering the 19479 samples in the training set of party S, only 0.378 (smaller than one) Bloom filters are expected to be disclosed.
Experiment 6.1 Experimental Setup
Dataset.
We evaluate FedSim on three synthetic datasets ( sklearn [10], frog [13], boone , [14]) and five real-world datasets ( house [1, 44], taxi [8, 52], hdb [23, 45], game [11, 24], and company [6, 12]).
The details (e.g. dimension of identifiers) of these datasets are summarized in Appendix E.
For each real-world dataset, we collect two public datasets from different real-world parties and conduct VFL on both public datasets.
For each synthetic dataset, we first create a global dataset by generating with sklearn API [10] ( sklearn ) or collecting from public ( frog , boone ).
Then, we randomly select some features as common features and randomly divide the remaining features equally to both parties.
Common features are not used in training for all methods except Combine.
To simulate the real-world applications, for synthetic datasets, we also add different scales σ cf of
Gaussian noise to the common features.
Specifically, for each identifier v i , the perturbed identifier v ′
i =
v i
+ N ( σ 2 cf I ) will be used for linkage.
For datasets with numeric identifiers ( house , taxi , hdb , syn boone , frog ), Euclidean distance is adopted to calculate similarities.
For game dataset,
Levenshtein distance is adopted to calculate similarities.
For company dataset, we first generate
Bloom filters from strings following [29], then calculate similarities based on Hamming distances.
Training.
Similarity model is a multi-layer perceptron (MLP) with one hidden layer.
The merge model contains a 2D convolutional layer with k conv × 1 kernel followed by a dropout layer and an
MLP with one hidden layer.
Both the local model and aggregate model in SplitNN are MLPs with one hidden layer.
We adopt LAMB [63], the state-of-the-art large-batch optimizer, to train all the models.
Each dataset is split into training, validation, and test sets by 7:1:2.
We run each algorithm five times and report the mean and standard variance (range is reported instead in figures) of performance on the test set.
We present root mean square error (RMSE) or R-squared value ( R 2 ) for regression tasks and accuracy for classification tasks.
The choices of hyperparameters are introduced in Appendix E.
Baselines.
We compare FedSim with nine baselines in our experiments.
Besides the four baselines (Exact, Top1Sim, AvgSim, FeatureSim) introduced in Section 3, the remaining baselines include:
1) Solo: only dataset d P is trained; 2) Combine: [ d P , d S ] is trained (only applicable for synthetic datasets); 3) FedSim-MLP: the CNN in FedSim is changed to an MLP with a similar number of parameters.
FedSim w/o sort: FedSim without sorting gate.
5) FedSim w/o weight: FedSim
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Performance
without weight gate (similarities are directly regarded as weights).
Notably, Exact is only evaluated on game and company because no exactly matched identifiers can be found on other datasets.
6.2 Performance
We evaluate the performance of FedSim on three synthetic datasets under different σ cf and five real-world datasets.
The results of synthetic datasets are presented in Figure 5(i), from which two observations can be made.
First, FedSim consistently has better or close performance compared to all the baselines.
Second, FedSim is more robust to the noise on the identifiers.
For example, in frog , the accuracy of Top1Sim drops to 84% as σ cf = 0 .
2 , while the accuracy of FedSim remains 91%.
The results of real-world datasets are summarized in Table 1.
We also calculate the estimated improvement ∆( Top 1 Sim ) (denoted as ∆ for simplicity) according to our proposed metric.
The relationship between ∆( Top 1 Sim ) and the relative improvement on Top1Sim is presented in Figure 5(ii).
Three observations can be made from the results.
First, FedSim consistently produces the best performance on all five datasets, while the baselines (especially the separated approaches) only have good performances on specific datasets.
For example, Top1Sim has close performance to FedSim on game , but fails on bike ; AvgSim has close performance to FedSim on bike , but fails on game .
Second, ∆( Top 1 Sim ) is positively correlated with real improvement on Top1Sim, indicating that the metric can be effectively used to estimate the improvement of FedSim without training.
This also implies that FedSim can effectively reduce the effect of lost information and redundant information as expected.
Third, comparing the performance of removing each component from FedSim, the sort gate makes the most significant contribution to the performance of FedSim by stabilizing the updates of the merge model.
The improvement of the weight gate indicates that adjusting the distribution of similarities can slightly benefit the performance.
Besides, CNN merge gate can also slightly improve on MLP merge gate by reducing overfitting.
As elaborated in Section D, FedSim only boosts performance on the datasets satisfying a widely held assumption: the similarity between identifiers is related to the similarity between data records.
Our experiments in Appendix F.6 indicate that FedSim has close (small K ) or lower (large K ) performance compared to baselines on a synthetic dataset with independent identifiers.
6.3 Privacy
In this subsection, to study how additional noise on similarities affects the performance of FedSim, we conduct experiments on five real-world datasets (the result of company is included in Appendix F. due to page limit).
Specifically, string or numeric identifiers are converted to Bloom filters according to [29].
The Hamming distances between Bloom filters are used to calculate raw similarities.
Then, given an acceptable success rate τ , a noise scale σ is calculated according to Theorem 2.
Finally,
Gaussian noise with scale σ is added to the raw similarities according to Equation 2.
The results are presented in Figure 6.
Exact is not evaluated since few Bloom filters have exactly the same bits.
From Figure 6, we observe that FedSim is robust to the noise on similarities; therefore, the attacking success rate can be reduced to [10 − 4 , 10 − 3 ] without evident performance loss.
Notably, the performance when τ = 1 is not necessarily the same as the performance in Section 6.2 since similarities are calculated based on different distances.
Related Work
Most studies [18, 26, 37, 50, 56, 61] in VFL focus on training and simply assumes record linkage has been done (i.e., the implicit exact linkage on record ID), which is impractical since most real-world federated datasets are unlinked.
Some approaches exactly link the identifiers by exact PPRL [7] or private set intersection (PSI) [7, 36, 60].
Nonetheless, these approaches incur performance loss of VFL and are also impractical since the common features of many real-world federated datasets cannot be exactly linked (e.g. GPS location).
[22] greedily links the most similar identifiers in PPRL, which negatively impacts performance since some beneficial pairs with relatively low similarity may be neglected.
[39, 40] explore the impact of record linkage on the performance of VFL, which is also adopted by [27].
However, all of them focus only on the most similar identifiers and assume there is a one-to-one mapping between the data records of two parties, which is not always true in practice.
Current VFL frameworks support various machine learning models including linear regression [17], logistic regression [25], support vector machine [34], gradient boosting decision trees [7, 60].
FDML [26] supports neural networks but it requires all the parties to hold labels.
SplitNN [56] focuses on neural networks and provides a new idea of collaborative learning where the model is split and held by multiple parties.
Since we study the scenario where only one party holds the labels and want to support commonly used neural networks, we build FedSim on top of SplitNN.
Conclusion
In this paper, we propose FedSim, a novel VFL framework based on similarities to boost the performance of VFL by directly utilizing the similarities calculated in PPRL and skipping the classification process.
We also theoretically analyze the additional privacy risk introduced by sharing similarities and provide a bound for the success rate of an intuitive attack.
In our experiment, FedSim consistently outperforms all the baselines.
Acknowledgements
This research is supported by the National Research Foundation, Singapore under its AI Singapore
Programme (AISG Award
No: AISG2-RP-2020-018).
Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore.
Qinbin is also in part supported by a Google PhD
Fellowship.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See the last paragraph of Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix G. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix H. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 5.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes are included in the supplementary materials.
The data are all publicly available; the link or citations of data are presented in Appendix E. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] All the details including data splits and hyperparameters are introduced in Section 6.1 and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The variance is reported in all tables and figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the “hardware” paragraph of Appendix E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.1 and Appendix E. (b) Did you mention the license of the assets?
[Yes] See the “license” paragraph of Appendix E. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Our codes are included in supplementary materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See the “Consent of dataset” paragraph of Appendix H (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See the “Personally identifiable information” of Appendix H. 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Our research is not related to human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
Our research is not related to human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Our research is not related to human subjects.