FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning
Abstract
Vertical federated learning (VFL) is a privacy-preserving machine learning paradigm that can learn models from features distributed on different platforms in a privacy-preserving way.
Since in real-world applications the data may contain bias on fairness-sensitive features (e.g., gender), VFL models may inherit bias from training data and become unfair for some user groups.
However, existing fair machine learning methods usually rely on the centralized storage of fairnesssensitive features to achieve model fairness, which are usually inapplicable in federated scenarios.
In this paper, we propose a fair vertical federated learning framework ( FairVFL ), which can improve the fairness of VFL models.
The core idea of FairVFL is to learn unified and fair representations of samples based on the decentralized feature fields in a privacy-preserving way.
Specifically, each platform with fairness-insensitive features first learns local data representations from local features.
Then, these local representations are uploaded to a server and aggregated into a unified representation for the target task.
In order to learn a fair unified representation, we send it to each platform storing fairness-sensitive features and apply adversarial learning to remove bias from the unified representation inherited from the biased data.
Moreover, for protecting user privacy, we further propose a contrastive adversarial learning method to remove private information from the unified representation in server before sending it to the platforms keeping fairness-sensitive features.
Experiments on three real-world datasets validate that our method can effectively improve model fairness with user privacy well-protected.
INTRODUCTION
In recent years, the explosion of data volume has enhanced the performance of machine learning (ML) models on many tasks, e.g., personalized recommendation and search [41, 50, 2].
In many real-world scenarios, feature fields of the same sample may be decentralized on different platforms [48, 29].
For instance, video watching behaviors and search engine behaviors of the same user usually spread across different platforms [42].
To learn models from various input features, existing machine learning methods usually rely on centralized storage of different feature fields [31, 21].
However, the data on different platforms is usually privacy-sensitive, and cannot be centralized due to privacy concerns and risks [47, 36].
Thus, vertical federated learning (VFL), which can utilize decentralized features for unified model learning in a privacy-preserving way, has become more and more important [53, 9, 7].
Existing VFL methods usually communicate intermediate results instead of raw data across platforms to enable models utilizing decentralized feature fields [53, 9].
For example, Yang et al.
[49] proposed a vertical federated framework for logistic regression.
They first calculated local results based on local feature fields, and then shared encrypted local results with a server to make predictions and learn parameters.
Che et al.
[7] proposed a vertical federated multi-view learning framework to integrate decentralized medical data.
They locally encoded medical feature fields kept by multiple hospitals into representation vectors and uploaded them to the server for diagnosis prediction.
However, since real-world data usually encodes bias on fairness-sensitive features (e.g., genders and ages) [27, 51],
VFL models may inherit bias from data and become unfair to some user groups [45, 46].
Due to the importance of fairness in machine learning, fair ML methods have attracted increasing attention in recent years [45, 46, 27, 51, 26].
For example, Zafar et al.
[51] proposed a fairness-aware optimization framework to improve the model fairness, which can constrain the difference between prediction distributions for different user groups.
Li et al.
[26] proposed a fair recommendation framework, which quantifies the recommendation unfairness on different user groups as a penalty for item re-ranking.
In general, most of the existing fair ML methods rely on the centralized storage of fairness-sensitive features to achieve model fairness.
However, in vertical federated learning, feature fields are decentralized on different platforms [43, 14], making it difficult to apply existing fair ML methods to improve the fairness of VFL models.
Thus, how to design a fair VFL framework that can protect user privacy and meanwhile improve model fairness, is a rarely studied problem.
In this paper, we propose a fair vertical federated learning framework (named FairVFL ), which can improve the fairness of VFL models in a privacy-preserving way.
In FairVFL , we partition data into two groups based on their fairness sensitivities, i.e., fairness-insensitive features that can be used for the target task and fairness-sensitive features (e.g., genders) that should be causally irrelevant to model predictions.
Along this line, the core of FairVFL is to learn unified and fair representations of samples based on their features distributed across different platforms with user privacy well-protected.
Specifically, platforms with fairness-insensitive features first learn local data representations, which are then uploaded to servers to build unified representations.
Since unified representations encode rich information of raw data, they can be used for the target task without access to the raw data.
Besides, unified representations may inherit bias towards sensitive features (e.g., gender stereotype) from implicit feature correlations in biased data and hurt model fairness [45, 27, 46].
In order to further learn fair unified representations, we apply adversarial learning to unified representations to remove bias encoded in them.
Due to the decentralized storage of feature fields, unified representations are sent to platforms with fairness-sensitive features to learn adversarial gradients.
Besides, since unified representations may still encode some private information of raw data [40, 34, 33], directly sharing them among platforms may incur privacy problems.
Thus, we further use contrastive adversarial learning to remove private information from the unified representations before sending them to other platforms.
Extensive experiments on three real-world datasets show that our FairVFL approach can effectively improve the fairness of VFL models and meanwhile the privacy is well protected.
RELATED WORK 2.1 Vertical Federated Learning
Vertical federated learning is a privacy-preserving paradigm for training ML models from feature fields that are kept by different platforms that share the same data ID space, and has been widely studied in recent years [43, 7, 14, 8].
For example, Wu et al.
[43] proposed a privacy-preserving ad CTR prediction framework with vertical federated learning.
They proposed to coordinate local platforms to learn behavior representations from local features and share the behavior representations with the server to model user interest in ads.
However, since real-world data usually encode bias on fairness-sensitive features (e.g., genders) [45, 46], VFL models may inherit bias from data and make unfair decisions for some user groups (e.g., minority users).
To improve model fairness in VFL, Liu et al.
[28] formulated the model performance gap for different user groups as a fairness regularization and incorporated it into the federated optimization objective.
However, their method needs to share the data grouping information with the server, which may also leak user privacy on fairness-sensitive features (e.g., gender).
Different from these methods, we propose a unified framework that can improve the fairness of VFL models which applies the adversarial learning to learn fair models and employ a novel contrastive adversarial learning method to protect user privacy.
2.2 Fair Machine Learning
With the increasing impacts of ML techniques on our society, the fairness of machine learning models has attracted substantial attention [45, 22, 26, 46].
Existing fair machine learning methods are usually designed to learn unbiased models with respect to various fairness-sensitive features (e.g., genders).
Some of them applied pre-processing techniques [5, 32, 13, 6] (e.g., resampling) or post-processing techniques [26, 3, 22, 18] (e.g., re-ranking) to improve model fairness.
However, these methods usually independently optimized model accuracy and fairness, which may only achieve a sub-optimal trade-off between accuracy and fairness [45, 46].
To better improve model fairness, many works have been proposed to jointly optimize model accuracy and fairness during model training [51, 15, 27, 46].
However, most of the existing fair ML methods rely on the centralized storage of feature fields, making them difficult to be applied in vertical federated learning.
Different from these methods, we propose a fair vertical federated learning framework, which applies adversarial learning to improve the fairness of VFL models based on decentralized feature fields.
Methodology 3.1 Problem Definition of FairVFL
In VFL, different feature fields of the same samples are decentralized on multiple platforms.
From a fairness perspective, these data can be partitioned into two groups, i.e., fairness-insensitive features and fairness-sensitive features.
The former can be taken by models as the inputs while the latter are expected to be causally irrelevant to the model predictions.
Without loss of generality, we assume that there are m types of fairness-sensitive features, and we denote the platform keeping the i -th one with type d ai as P ai .
Besides, we assume that fairness-insensitive feature fields are decentralized on n platforms, and we denote the i -th platform as P bi .
Moreover, we assume that there is a task platform P t keeping labels y of samples on the target task.
Following existing works [43, 50], we assume that there is a trust-worthy server P w for information aggregation.
Next, we present a formal definition of privacy protection and fairness.
The privacy constraint requires that the private information in the local data of a platform should not be disclosed to another platform.
Thus, the raw data of a local platform cannot be disclosed and the intermediate results shared with other platforms should be carefully protected.
Besides, in this paper, we define model fairness from a casual view [27].
The counterfactual fairness requires that the model prediction for a sample should always hold if we only change the fairness-sensitive features.
According to the theoretical analysis in
Li et al.
[27], eliminating fairness-sensitive feature information in the data representation that is used for the model prediction can achieve casual fairness.
We remark that even though fairness-sensitive features are not taken as the inputs, the model may also inherit bias related to sensitive features by mining inherent shortcuts from data.
The goal of fair vertical federated learning is to achieve a good trade-off among performance, privacy, and fairness, i.e., improving the fairness of
VFL models with minimal performance drop in a privacy-preserving way.
3.2 Federated Model Serving
The main framework of FairVFL to utilize decentralized feature fields is to encode them into a unified representation (Fig. 1).
The ML model in FairVFL is partitioned into three parts, i.e., multiple local models {M li | i = 1 , ..., n } , an aggregation model M w and a task-specific model M t , where M li is the i -th local model maintained by the platform P bi .
In FairVFL , local models are used to encode remark that a platform can store multiple sensitive feature fields in practice. [Figure] local fairness-insensitive features into local representations.
We remark that FairVFL is a general framework and the architecture of a specific local model can be adjusted based on the input data.
For example, the local models for encoding textual search logs can be implemented by transformers [39] or PLMs [10].
When the task platform P t needs to serve a target sample s , P t will first distribute the ID of the sample to fairness-insensitive platforms { P bi | i = 1 , ..., n } .
In the i -th platform P bi , the local model M li encodes fairness-insensitive features of the target sample s stored in P bi , and builds a local representation s li .
Thus, we can obtain multiple local representations { s li | i = 1 , ..., n } , which encode different fairness-insensitive feature fields of the target sample.
Following Wu et al.
[43], these local representations are then uploaded to the server P w for task prediction.
The aggregation model M w is maintained by the server P w .
It is used to aggregate information in local representations into a unified representation s .
Specifically, different feature fields usually have inherent relatedness, mining which can enhance the model for the target task [2, 44].
Thus, we first apply a multi-head self-attention network [39] to capture relatedness among local representations, where the contextual representation of s li is denoted as ˆ s i .
Then we apply an attention network to contextual representations to model their relative importance for representing raw data and build the unified representation s .
3 Since s can encode various information in decentralized fairness-insensitive feature fields, we further upload the unified representation s instead of raw data to the task platform
P t to provide information for the target task.
(We apply LDP to protect s before sending it to P t .)
The task-specific model M t is maintained by the task-specific platform P t .
It is used to utilize information in the unified representation s to make prediction for the target task:
ˆ y =
M t ( s ) , where ˆ y is the model prediction.
The architecture of the task-specific model M t is also adjusted based on the target task (e.g., risk prediction).
For instance, for the classification task, we can employ a dense network to implement M t .
In this way, decentralized fairness-insensitive feature fields of a target sample can be utilized by FairVFL for the target task without the disclosure of raw data.
3.3 Fair Representation Learning in FairVFL
As introduced in Sec. 3.2, the task prediction for each sample is computed based on their unified representation s .
Thus, the core of FairVFL to achieve counterfactual fairness is to learn a fair and unified representation s that does not encode fairness-sensitive bias.
Although the fairnesssensitive features are not model inputs, the model may still encode bias related to them by mining inherent shortcuts to these features from data.
Thus, we apply adversarial learning to prevent unified representations from encoding fairness-sensitive bias.
Vanilla adversarial learning techniques usually rely on centralized storage of input data and labels of fairness-sensitive attributes to learn adversarial gradients for model training.
However, in vertical federated scenarios, feature fields are decentralized on different platforms, which brings challenges to the application of existing methods.
An intuitive solution is sharing the unified representation with platforms that store observed fairnesssensitive features { P ai | i = 1 , 2 , ..., m } to learn corresponding adversarial gradients.
However, unified representations usually encode some private information, and directly sharing them across different platforms may lead to user privacy leakage.
To tackle this challenge, we propose a contrastive adversarial learning technique to protect user privacy by modifying unified representations. [Figure]
As shown in Fig.
2, the core idea of the contrastive adversarial learning is to learn a protected representation that only encodes information on a certain fairnesssensitive feature.
The protected representation can be further shared with the platform keeping the corresponding sensitive feature to learn adversarial gradients without leaking user privacy.
Specifically, for each fairness-sensitive feature d ai , we first map unified representation s into a protected representation a i : a i
=
A i ( s ) , where A i is a mapper based on MLP and s is the preimage of a i .
a i is designed to only retain information of s on the fairness-sensitive feature d ai and eliminate other user privacy.
To achieve this goal, we propose the contrastive adversarial learning method to make the preimage of a i indistinguishable among unified representations with the same feature d ai .
Given representations of samples in the same training batch { ( a ji , s j ) | j = 1 , 2 , .., E } , we first rank these samples based on the relevance between their protected representations a ji and a i : r j = a i · a ji , where a ji and s j are the protected and unified representation of the j -th sample, r j is the relevance score, and E is the batch size.
4 Intuitively, highly relevant data samples are likely to share the same fairness-sensitive feature d ai .
Thus, from the top E i samples ranked by relevance scores, we can randomly select a unified representation s − i that is highly likely to be the same with s in d ai , where E i is a hyper-parameter.
s −
i is used as the negative sample in contrastive learning to learn a contrastive discriminator D ci to classify preimage of a i :
[Equation]
where D ci ( · ) is implemented by an MLP network, Ω is the training set and x is a data sample in Ω .
Optimal discriminator ˆ D ci is learned by minimizing L pi with fixed representations s , s − i , a i :
[Equation]
Finally, the optimal discriminator ˆ D ci is used to learn contrastive adversarial loss L ci and the contrastive adversarial gradients ∂
L ci ∂ a i on a i with the fixed optimal discriminator ˆ D ci , s and s − i :
[Equation]
The contrastive adversarial gradient ∂ L ci ∂ a i on a i is further back-propagated to the mapper A i :
[Equation]
where negative contrastive adversarial gradient − ∂ L ci ∂A
i on A i is used for model updating, and representations s and s − i are fixed and not tuned by the contrastive adversarial gradient.
In this way, contrastive adversarial gradients can enforce the mapper A i to protect user private information in a i .
Since user privacy can be reduced from a i , we further share it with the platform P ai keeping label y ai of the fairness-sensitive feature d ai to learn corresponding adversarial gradients.
The platform P ai first employs a bias discriminator D ai to predict the label ˆ y ai of fairness-sensitive feature d ai from the protected representation a i and obtain a bias discrimination loss L di as follows:
[Equation]
where D ai is implemented by an MLP network.
Moreover, besides private information, contrastive adversarial learning may remove bias information from a i and hurt fair representation learning.
Thus, we minimize bias discrimination loss L di to optimize both bias discriminator D ai and mapper A i :
[Equation]
where ˆ D ai is the optimal bias discriminator, ˆ A i is the i -th optimal mapper, and the unified representation s is fixed.
Finally, for the i -th fairness-sensitive feature d ai , we can obtain the adversarial loss L ai and corresponding adversarial gradients ∂
L ai ∂ s on unified representation s :
[Equation]
where the optimal bias discriminator and mapper are fixed when optimizing the unified representation s .
The negative adversarial gradient − ∂ L ai ∂ s is further used to reduce fairness-sensitive bias encoded in the unified representation s to achieve counterfactual fairness.
Besides, we can obtain the task loss
L t based on task label y and predicted label ˆ y .
The task loss function is also adjusted based on the target task.
Finally, we can formulate the overall loss L as:
[Equation]
where L t , L a and L c represents the training objective for model performance, fairness and privacy respectively, λ i is the weight of loss L ai and γ i is the weight of loss L ci .
3.4 Federated Model Training
Next, we will introduce how we train model on decentralized fairness-sensitive and insensitive feature fields.
First, for the task loss L t , the task platform P t can directly calculate gradient ∂ L t ∂ M t on task model and gradient ∂ L t ∂ s on unified representation s .
∂ L t ∂ M t is used to update M t and ∂ L t ∂ s is distributed to the server P w for the further model updating.
Second, it is usually difficult to obtain the optimal discriminator in adversarial learning.
Following Goodfellow et al.
[16], we optimize the discriminator for a single step based on discrimination gradients before executing adversarial learning.
Specifically, for the contrastive adversarial learning, the sever first learns contrastive discrimination gradient ∂ L pi ∂D ci on contrastive discriminator D ci to update it.
Then the server utilizes updated D ci to learn contrastive adversarial gradient ∂ L ci ∂A
i on mapper A i to update it.
Third, for adversarial learning on each fairness-sensitive feature d ai , platform P ai first calculates bias discrimination gradient ∂ L di ∂D ai on the bias discriminator D ai and ∂ L di ∂ a i on protected representation a i .
The former is used to optimize D ai .
The latter is distributed to the server P w to calculate gradient on A i :
∂ L ai ∂A i =
∂
L ai ∂ a i ∂ a i ∂A
i , which is used to optimize A i .
Next, the platform P ai calculates the adversarial gradient ∂
L ai ∂ a i on a i . ∂
L ai ∂ a i is further distributed to P w to calculate adversarial gradients ∂
L ai ∂ s on the unified representation s
based on gradient back propagation:
∂ L ai ∂ s = ∂
L ai ∂ a i ∂ a i ∂ s .
Until now, the server P w can further calculate overall gradients ∂ L ∂ s = ∂
L t ∂ s − (cid:80) mi {+=
 λ+} i ∂
L ai ∂ s on the unified representation s .
The server further calculates gradient ∂ L ∂ M w on the aggregation model:
∂ L ∂ M w =
∂ L ∂ s ∂ s ∂
M a and the gradient ∂ L ∂ s li on each local representation s li :
∂ L ∂ s li =
∂
L ∂ s ∂ s ∂ s li .
Finally, gradients on local representations are distributed to corresponding platforms { P bi | i = 1 , 2 , ..., n } to optimize local models {M li | i = 1 , 2 , ..., n } .
The detailed process of a training round in FairVFL is in Algorithm 1 (Supplementary). 3.5 Discussion on Privacy and Efficiency
Privacy : In FairVFL , each data platform locally keeps its private data and never shares it with the outside, which can protect user privacy to some extent.
Besides, for the vertical federated model learning and inference, the intermediate model results, i.e., data representations and model gradients, need to be exchanged across platforms, which may lead to potential privacy leakage.
To tackle this challenge, we propose a contrastive adversarial learning method to protect private information in data representations, which can compare data representations with the same attribute and remove the attribute-irrelated information in them.
To further protect private information in gradients, we can apply differential privacy techniques to perturb exchanged gradients as standard VFL methods [47].
Efficiency: FairVFL needs multiple times of communications in a single training round, which may arouse concerns on communication efficiency. Fortunately, the extra communication costs are usually small. This is because in the extra communication rounds FairVFL only needs to exchange protected representations and their gradients, and the extra communication cost is O (4 mEH ) , where m is the number of fairness-sensitive platforms, H denotes the dimensionality of protected representations, and E denotes the batch size. Since M , E , and B is usually small in practice, the corresponding cost is usually minor.
For the computation efficiency, compared with the standard VFL framework,
FairVFL only increases the computation cost of the server and platforms with fairness-sensitive attributes.
They need to extra compute the gradients on the attribute discriminators and contrastive discriminators.
Thus, the computation complexities of FairVFL and standard VFL are O ( d 3 + D 3 ) and O ( D 3 ) respectively, where d and D denote the parameter sizes of the discriminators and the main task model.
In widely used settings, the discriminator is usually implemented by a small FFNN network, whose parameters are usually much less than the main task model.
Thus, the computation efficiency of FairVFL is still comparable with standard VFL methods.
Experiment 4.1 Datasets and Experimental Settings
We evaluate model performance and fairness on three real-world datasets.
The first one is ADULT [24], which is a widely used public dataset for fair ML [52, 20, 35, 25].
It contains rich feature fields (e.g., education and social relation) for income prediction.
20,000 randomly selected data samples are used to construct training and validation dataset, and 10,000 randomly selected data samples are used to construct test dataset.
Following existing works [45, 46], we treat gender and age as fairness-sensitive features and utilize other feature fields for income prediction.
Besides, we randomly partitiontypes of feature fields in ADULT into three platforms to simulate the VFL scenarios 5 .
The second one is NEWS , which is constructed by user logs (i.e., news clicks, search, and web-page browsing logs) on Microsoft News and Bing Search platforms during 6 weeks (June 23th - July 20th, 2019) for news recommendation.
These three types of data are decentralized on three platforms under VFL settings.
We randomly select 100,000 news impressions in the first three weeks to construct the training and validation set and select 100,000 news impressions in the last week to construct the test set.
Besides, we also treat gender and age as fairness-sensitive attributes in NEWS .
The third one is CelebA [30], which is a public face attributes dataset.
To simulate the VFL setting, we use the raw images and a part of the raw attributes (i.e., the “Straight Hair” attribute and “Wavy Hair”) as the input features.
The raw images are stored in a data platform and other input attributes are stored in another platform.
[Table] The target task is to classify whether the input sample has the attribute “Smiling”. We also treat gender (the “Male” attribute) as the target fairness-sensitive attribute. In FairVFL , the local representations and unified representations are 400 -dimensional.
The protected representations for gender and age are 32 - and 64 -dimensional, respectively.
The weights of contrastive adversarial loss for different sensitive features are set to 0 .
The weights of adversarial loss for gender and age are set to 23 and 23 on NEWS , respectively.
Besides, the weights of adversarial loss for gender and age are set to 1 e 2 and 1 e 1 on ADULT , respectively.
We exploit Adam algorithm [23] for model optimization with 1 e -4 learning rate.
The size of the mini-batch for model training is set to 32 .
Besides, for both age and gender, we set E i to 5 for simplification.
We also use the dropout technique [38] with a 0 .
2 drop probability to alleviate model overfitting.
Hyper-parameters are selected based on the validation dataset.
Codes are available in https://github.com/taoqi98/FairVFL .
4.2 Performance and Fairness Evaluation
Remark that we can achieve counterfactual fairness by reducing fairness-sensitive features encoded in data representations that are used for the model predictions [27].
Thus, following existing works [12, 4, 46, 27], we utilize the ensemble of 5 attackers to predict fairness-sensitive features from unified representations s to evaluate the independence of fairness-sensitive features and data representations to verify model fairness.
If attackers can distinguish fairness-sensitive bias from unified representations then we can say the model is unfair.
Thus, a lower F1 score on the fairnesssensitive feature classification task means better model fairness.
Besides, we use accuracy and F1 score to evaluate the classification task on ADULT and CelebA .
Moreover, following these works [1, 41], we use AUC and nDCG@10 to evaluate the news recommendation task on NEWS .
Each experiment is repeated five times and we report the average performance.
[Table] We compare our FairVFL method with several recent representative baseline methods for fair representation learning: (1) FairGo
[46]: a user-centric fair representation learning framework that utilizes adversarial learning to reduce fairness-sensitive bias information in representations; (2) FairSM [27]: a unified framework to achieve counterfactual fairness on representations; (3) FairRec [45]: an adversarial representation decomposition framework that can decompose bias and bias-insensitive information.
These methods rely on centralized storage of features to learn fair ML models.
Besides, both FairVFL and these three methods are general frameworks and can be applied to basic models to improve their fairness.
In addition, baseline methods also include a standard vertical federated representation learning method ( VFL ) without fairness criterion [7].
Thus, to compare their effectiveness, we combine them with several basic ML models on the three datasets.
Basic models for ADULT are implemented by three SOTA
[Figure]
[Figure] [Figure]
[Figure]
Education prediction
Social relation prediction Gender
Age 20.
G ende r F
FairVFL 0.
A ge F - age adverisial loss - gender adverisial loss
E du c a t i on F
FairVFL w/o CAL FairVFL
Random
S
o c i a l
R e l a t i on
F
FairVFL w/o CAL FairVFL
Random models for modeling structural features, including MLP [17], TabNet [11] and AutoInt [37].
Basic models for NEWS are implemented by SOTA news recommendation models, including NAML [41],
LSTUR [1] and NRMS [44]. Moreover, the VFL model for CelebA is based on ResNet-10 [19].
Due to space limitations, detailed settings of these basic models are summarized in the Supplementary.
Results on ADULT and NEWS are summarized in Table 1, and results on CelebA are summarized in Table 2, from which we have several observations.
First, basic models based on vertical federated training (e.g., MLP+VFL) can achieve similar performance with the same basic models based on centralized training (e.g., MLP).
These results show that the vertical federated learning technique can effectively enable ML models to utilize decentralized feature fields for target tasks without the disclose of raw data.
Second, models without the protection of fair ML methods (e.g., MLP and MLP+VFL) are biased by fairness-sensitive features.
This is because data may encode bias on sensitive features like genders.
Models may inherit bias from data and become unfair to some user groups.
Third, after applying our FairVFL method to basic models, the fairness of these models effectively improves.
This is because our FairVFL method applies the adversarial learning on bias discrimination to unified representations, which can effectively reduce bias encoded in them and improve model fairness.
Fourth, compared with models protected by centralized fair machine learning methods (i.e., FairGo ,
FairSM and FairRec ), models protected by FairVFL can achieve similar performance and fairness.
These results verify that our FairVFL method can effectively improve the fairness of VFL models.
4.3 Ablation Study
In this section, we first evaluate the effectiveness of adversarial learning in improving model fairness.
Due to space limitations, we only show results on ADULT in the following sections.
Results are shown in Fig.
3, from which we have several observations.
First, after removing adversarial learning on gender discrimination, model fairness on user gender seriously drops.
This is because users of the same gender usually have similar behavior patterns and there may exist gender bias in data.
Models may inherit gender bias from implicit feature correlations in data and become unfair to some user groups.
To tackle this challenge, FairVFL applies adversarial learning on bias discrimination to unified representations to reduce gender bias encoded in them.
Second, removing adversarial learning for age discrimination also seriously hurts model fairness on user age, which further verifies the effectiveness of adversarial learning for improving model fairness.
Next, we verify the effectiveness of contrastive adversarial learning on privacy protection.
Since unified representations include private information of fairness-insensitive features on local platforms, communicating them across platforms may arouse privacy concerns.
Motivated by membership inference attack methods, we evaluate the privacy protection by inferring the input features of a unified representation from corresponding protected representations { a i | i
= 1 , ..., m } that are shared across platforms { P ai | i
= 1 , ..., m } .
Specifically, based on the ADULT dataset, we infer the value of input features from each a i via 5 attackers and average the results.
A lower inference accuracy (the
F1 value) means better privacy protection.
We remark that for fairness evaluation, we distinguish bias on fairness-sensitive features from unified representations instead of the shared representations, and these sensitive features are also not the input of representation learning.
We only show results on two representative feature fields (i.e., education, and social relation) in Fig.
3 due to space limitations.
First, without the protection of contrastive adversarial learning, shared representations are very informative
(a) Influence of CAL weight 𝛾 ! on privacy protection.
E du c a t i on F
FairVFL Random 𝛾 ! {+0 0.
+}
A ge F
FairVFL Random 0 0.1 0.
G ende r F
FairVFL Random (b) Influence of CAL weight 𝛾 ! on model fairness. {+𝛾 ! 𝛾 ! 0 0.1 0.5 1 2 5 
+}
S
o c i a l
R e l a t i on
F
FairVFL Random 𝛾 ! fairness, where contrastive adversarial loss weights for different sensitive features are set to γ c for inferring private user information.
These results verify that directly sharing unified representations across platforms may leak user privacy.
Second, contrastive adversarial learning can effectively protect user privacy in protected representations.
This is because contrastive adversarial learning enforces protected representations to be indistinguishable for predicting its preimage from unified representations of the same fairness-sensitive features, which can effectively reduce bias-irrelevant personal information in shared representations.
4.4 Hyper-Parameter Analysis
Next, we evaluate the influence of the weights of contrastive adversarial learning { γ i | i = 1 , 2 , ..., m } on both privacy protection and model fairness.
To simplify the analysis, we set them to the same value (denoted as γ c ).
The influence of γ c on privacy protection and model fairness is shown in Fig.
4, from which we have two observations.
First, with the increase of γ c , FairVFL can achieve better privacy protection ability.
This is intuitive since larger γ c makes contrastive adversarial gradients more important to update the mappers, making them can reduce user privacy in unified representations more effectively.
Second, larger γ c also hurts model fairness more seriously.
This is because larger γ c makes mappers pay more attention to reducing user private information, which may hurt their abilities on retaining information of fairness-sensitive bias information.
Thus, the protected representations may contain less bias information and hurt the effectiveness of adversarial learning in improving model fairness.
Fortunately, when γ c is small (e.g., 0.25), FairVFL is effective in privacy protection since the performance of user privacy inference is close to random.
Besides, FairVFL can achieve satisfied model fairness (about 0.5 and 0.15 F1 value for gender and age prediction) when γ c is small (e.g., 0.25).
This indicates that FairVFL can achieve effective privacy protection ability without hurting model fairness and we set γ c to 0 .
25 in FairVFL .
CONCLUSION
In this paper, we propose a fair vertical federated learning framework (named FairVFL ), which can improve the fairness of VFL models in a privacy-preserving way.
The core of FairVFL is to learn fair and unified representations to encode samples based on their decentralized feature fields.
Each platform storing fairness-insensitive features first learns local representations from their local features.
Then these local representations are uploaded to a server to form a unified representation used for the target task.
In order to further learn fair and unified representations, we send them to platforms storing fairness-sensitive features and apply adversarial learning to reduce bias encoded in them.
Besides, to protect shared representations from leaking user privacy, we propose a contrastive adversarial learning method to reduce user private information in them before sharing them with other platforms.
Experimental results on three datasets show that FairVFL is comparable with centralized baseline fair ML methods in terms of both performance and fairness, meanwhile effectively protects user privacy. Acknowledgments
This work was supported by the National Key R&D Program of China 2021ZD0113902, the National
Natural Science Foundation of China under Grant numbers 82090053, U1936126, and TsinghuaToyota Joint Research Fund 20213930033.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Supplementary Information.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work focuses on privacy protection and model fairness.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Section 4.
and Supplementary Information.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and Supplementary Information.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Supplementary Information.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[Yes] See Supplementary Information.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Section 4.1.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Supplementary Information.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Supplementary Information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Federated Fair Model Training
The detailed workflow of our FairVFL method is summarized in Algorithm 1. 1: for all i in 1 , 2 , ..., n do 2: Query s li from the platform P bi 3: end for 4: Apply M w to learn s from { s li | i = 1 , 2 , ..., n } on P w 5: Upload s to P t to calculate ∂ L t ∂ s and ∂ L t ∂
M t 6: Use ∂ L t ∂ M t to update M t and distribute ∂ L t ∂ s to P w 7: for all i in 1 , 2 , ..., m do 8: Map s to a i via A i on P w 9: Select a contrastive negative sample s − i on P w 10:
Calculate ∂ L p i ∂D ci based on Eq.
1 to update D ci on P w 11:
Calculate ∂ L ci ∂A
i based on Eq.
3 to update A i on P w 12:
Recalculate a i via updated A i 13:
Upload a i from P w to P ai 14:
Learn ∂ L di ∂D ai based on Eq.
5 to update D ai on P ai 15:
Learn ∂ L di ∂ a i based on Eq.
5 on P ai and distribute it to P w 16:
Learn ∂ L di ∂A i via ∂ L d ∂ a i on P w to update A i 17:
Recalculate a i via updated A i and upload it to P ai 18:
Learn ∂ L ai ∂ a i based on Eq.
7 on P ai and distribute it to P w 19:
Calculate ∂ L ai ∂ s via ∂ L ai ∂ a i on P w 20: end for 21: Calculate ∂ L ∂ s for s based on Eq.
8 on P w 22:
Calculate ∂ L ∂ M w via ∂ L ∂ s to update M w on P w 23: for all i in 1 , 2 , ..., n do 24:
Calculate ∂ L ∂ s li via ∂ L ∂ s on P w and distribute it to P bi 25:
Calculate ∂ L ∂ M li via ∂
Experimental Settings
Next, we will introduce the basic models used for model training in details.
Since ADULT is a tabular feature dataset, we implement three basic models for modeling structural features [17] to predict income on ADULT : (1) MLP [17]: converting user features into embedding vectors and using an MLP network for income prediction.
(2) TabNet [11]: using attentive feature transformer networks to model relatedness of different features and build local representations.
(3) AutoInt [37]: applying multi-head self-attention network to feature embeddings to model their interactions and learn their representations.
In these three methods, dimensions of feature embeddings are set to 32, and dimensions of local representations are set to 400.
In addition, we choose three mainstream news recommendation models as basic models for the news recommendation task on NEWS :
(1) NAML [41]: applying an attentive CNN network to learn behavior representations and an attention network to learn user representations; (2) LSTUR [1]: proposing to learn short-term user representations from recent user behaviors and long-term user representations via user ID embeddings; (3) NRMS [44]: employing multi-head self-attention networks to learn behavior and user representations.
In these three methods, three types of behavior representations (e.g., clicked news) are set to 400-dimensional.
A ge
F0.
G ende r
F
Influence of Adversarial Learning
Next, we will analyze the influence of the weights of adversarial learning on model fairness.
We summarize results in Fig.
5 and have several observations.
First, with the increase of λ g , model fairness on gender can increase.
This is because larger λ g makes models pay more attention to reducing gender bias encoded in unified representations during model training.
Second, when λ g is large enough, model fairness on gender becomes stable.
This is because when λ g is large enough, gender information in unified representations can be effectively reduced.
Third, with the increasing of λ a , model fairness on age also first increases and then converges.
Similarly, this is because users with similar ages usually have similar behaviors and may encode age bias in real-world data.
Larger λ a can more effectively prevent unified representations from encoding age bias from data until age bias is effectively removed.
Limitations and Future Work
Although FairVFL is effective in both model fairness and privacy protection, FairVFL may have more communication latency than other baseline methods during model training.
This is because the adversarial learning and the contrastive adversarial learning in FairVFL make it need to communicate with a fairness-sensitive feature platform multiple times.
In our future work, we plan to study an asynchronous communication mechanism to reduce the communication latency of FairVFL .