Paper: VFLAIR: A Research Library and Benchmark for Vertical Federated Learning (ICLR-2024)

Reviewer comment (Novelty):
Although this paper summarizes most of the VFL settings, some advanced model partition and communication protocol algorithms are missed. In this way, I suggest authors should add these missed algorithms, such as quantization, federated graph neural networks, etc.

Author response:
Thank you for your constructive suggestions. We have included 3 new communication protocols, including Quantizate [1], Top-k [1] and CELU-VFL [2]. We also added FedGNN to benchmark its performance on Cora dataset. Details on these experimental settings including model partition, hyperparameters are shown in Tab. 10 to 12 in Appendix H. Results are all included in Tab. 6 for communication protocols and in Tab. 7 for Cora dataset with FedGNN, both in Section 6.1. For quicker access, please refer to our reply in *"Response to Weakness 4 and Question 3 for Reviewer z5tx"* (communication protocols) and *“Response to Weakness 3 and Question 3 for Reviewer z5tx”* (Cora dataset with FedGNN, Tab(1) in the response). We do not show them here again to avoid repeatitiveness and redundancy.

Reviewer comment (Experiments):
I think some evaluations of this platform should be developed to show how it is lightweight.

Author response:
To demonstrate that our platform is lightweight, we compare it with FATE, one of most widely used FL platform supporting a broad range of VFL functionalities, on their system requirements for deployment.  According to FATE ([link](https://github.com/FederatedAI/FATE/blob/master/deploy/standalone-deploy/README.md)), deploying a stand-alone version of the FATE framework requires at least a 8 core CPU with 16G memory and 500G hard disk and the downloaded docker package for deployment is of size 4.92G for version 1.7.1.1. However, for our VFLAIR, a 1 core CPU with less than 4G memory and less than 4.0G hard disk is required for installation.

(4) Comparision between FATE and VFLAIR
|                                     |       CPU      |              memory              | Installation required hard disk |
|:-----------------------------------:|:--------------:|:--------------------------------:|:-------------------------------:|
| FATE (stand-alone, version 1.7.1.1) |     8 core     |                16G               |              4.92G              |
|                VFLAIR               |     1 core     |                4G                |              $<$4G              |

Reviewer comment (Experiments):
VFL is usually adopted in recommendation tasks instead of image classification. The authors apply too many image classification datasets in the experiments.

Author response:
Thank you for your suggestions. We have added Criteo and Avazu into our experiments, and results are included in Tab. 7 in Section 6.1. For quicker access, please refer to our reply in *“Response to Weakness 3 and Question 3 for Reviewer z5tx”* (Tab(1) in the response). We do not show them again here to avoid repeatitiveness and redundancy.

Reviewer comment (Experiments):
As discussed in limitations, the cryptographic techniques that are significant, are not included in this library.

Author response:
Thanks for the suggestion. We prioritized benchmarking non-cryptographic defense methods because they are emerging more rapidly in recent years due to their capability of defending various types of atttacks, whereas cryptographic techniques typically protect data in-transit during training, can not defend backdoor attacks or inference-time attacks effecitively while incurring significant computation and communiction cost to run. The benchmark of cryptographic techniques focus more on efficiency. Nevertheless, we think that the combination of cryptographic and noncryptographic methods would be an interesting research direction and potential solution for practical deployment. 

For VFLAIR, we have already included one of most commonly used cryptographic techniques, i.e. Paillier Encryption, in tree-based VFL and evaluated its impact on model performance and computation efficiency in Table. 5 in Section 6. We are currently implementing Encryption techniques to NN-based VFL and also plan to add more advanced privacy-preserving cryptographic methods to our VFLAIR in the near future.

Reviewer comment (Experiments):
Please add some advanced model partition and communication protocol algorithms, such as quantization, federated graph neural networks, etc.

Author response:
Thank you for your constructive suggestions. We have included 3 new communication protocols, including Quantizate [1], Top-k [1] and CELU-VFL [2]. We also added FedGNN to benchmark its performance on Cora dataset. Details on these experimental settings including model partition, hyperparameters are shown in Tab. 10 to 12 in Appendix H. Results are all included in Tab. 6 for communication protocols and in Tab. 7 for Cora dataset with FedGNN, both in Section 6.1. For quicker access, please refer to our reply in *"Response to Weakness 4 and Question 3 for Reviewer z5tx"* (communication protocols) and *“Response to Weakness 3 and Question 3 for Reviewer z5tx”* (Cora dataset with FedGNN, Tab(1) in the response). We do not show them here again to avoid repeatitiveness and redundancy.

Reviewer comment (Evaluation):
Add some compassion and discussion on how this platform is lightweight.

Author response:
To demonstrate that our platform is lightweight, we compare it with FATE, one of most widely used FL platform supporting a broad range of VFL functionalities, on their system requirements for deployment.  According to FATE ([link](https://github.com/FederatedAI/FATE/blob/master/deploy/standalone-deploy/README.md)), deploying a stand-alone version of the FATE framework requires at least a 8 core CPU with 16G memory and 500G hard disk and the downloaded docker package for deployment is of size 4.92G for version 1.7.1.1. However, for our VFLAIR, a 1 core CPU with less than 4G memory and less than 4.0G hard disk is required for installation.

(4) Comparision between FATE and VFLAIR
|                                     |       CPU      |              memory              | Installation required hard disk |
|:-----------------------------------:|:--------------:|:--------------------------------:|:-------------------------------:|
| FATE (stand-alone, version 1.7.1.1) |     8 core     |                16G               |              4.92G              |
|                VFLAIR               |     1 core     |                4G                |              $<$4G              |

Reviewer comment (Experiments):
The authors apply too many image classification datasets in the experiments. Some recommendation tasks should be considered, such as Criteo, Avazu, etc.

Author response:
Thank you for your suggestions. We have added Criteo and Avazu into our experiments, and results are included in Tab. 7 in Section 6.1. For quicker access, please refer to our reply in *“Response to Weakness 3 and Question 3 for Reviewer z5tx”* (Tab(1) in the response). We do not show them again here to avoid repeatitiveness and redundancy.

Reviewer comment (Experiments):
As discussed in limitations, the cryptographic techniques that are significant, are not included in this library.

Author response:
Thanks for the suggestion. We prioritized benchmarking non-cryptographic defense methods because they are emerging more rapidly in recent years due to their capability of defending various types of atttacks, whereas cryptographic techniques typically protect data in-transit during training, can not defend backdoor attacks or inference-time attacks effecitively while incurring significant computation and communiction cost to run. The benchmark of cryptographic techniques focus more on efficiency. Nevertheless, we think that the combination of cryptographic and noncryptographic methods would be an interesting research direction and potential solution for practical deployment. 

For VFLAIR, we have already included one of most commonly used cryptographic techniques, i.e. Paillier Encryption, in tree-based VFL and evaluated its impact on model performance and computation efficiency in Table. 5 in Section 6. We are currently implementing Encryption techniques to NN-based VFL and also plan to add more advanced privacy-preserving cryptographic methods to our VFLAIR in the near future.

Reviewer comment (Presentation):
The paper predominantly centers on evaluating attacks and defence strategies. But the paper title implies a broader scope – VFL in its entirety. The paper title could be more specific to align with the focus of the paper.

Author response:
Thank you for the suggestion. As a **research library** designed to facilitate research development in the VFL, our VFLAIR incorporated all the essential components of a VFL system, including 2 model partition strategies, 5 communication protocols (3 added during discussion), 1 encryption protocol, 13 datasets (4 added during discussion) of different modality and data partition methods, 29 model architectures, in addition to 11 attacks and 8 defenses. In addition, our VFLAIR can be easily used and extended to perform evaluations on various aspects of a VFL system, including model performance, computation and communication efficiency under a broad range of settings. 

Therefore, we consider our VFLAIR a comprehensive VFL library and will continue to add more state-of-art privacy-preserving and communication-efficient methods in the future. As for our benchmark evaluations, we have further refined and completed new evaluations on model performance of new datasets (Tab. 7 in Section 6.1), communication and computation efficiency of various communication strategies (Tab. 6 in Section 6.1). Therefore, although comprehensive evaluation of attack and defense is an outstanding feature of our benchmark, we think other aspects should not be overlooked. So we would prefer to **keep our title unchanged**.

Reviewer comment (Evaluation):
A notable contribution is the introduction of new evaluation metrics such as defence capability score (DCS). However, the experiments did not validate the effectiveness of the proposed metrics. What is the evidence that shows that the proposed metrics indeed work, representing the real ability of the evaluated algorithms?

Author response:
We validate the effectiveness of our proposed DCS metrics from the following 3 aspects.

- **Robustness.**  The results of the C-DCS rankings are generally consistent across all datasets (MNIST, CIAFR10 and NUSWIDE) in Tabs. 14 to 16 under the same setting. Also, the C-DCS ranking of the defense methods are still generally consistent under different model partition setting and communication protocols in Tabs. 14, 17 and 18. Below we plot the mean and standard derivation (std) of the ranking of each defense in the above 5 ranking tables with different defense hyper-parameter shown in [this figure (link)](https://anonymous.4open.science/r/VFLAIR_discussion/dcs_roubustness.png). As shown in this figure,  the ranking is relatively stable and the std is small across multiple settings for all the defenses. Finally, as shown in Figs. 4, 14 and 15, the C-DCS ranking is also generally stable with the change of $\beta$, especially when $\beta \leq 0.7$. All these demonstrate the good stability and credibility of our proposed C-DCS metrics. 
- **Sensitivity.** We demonstrate that the changes in DCS value is a good indicator of relative defense capability with the following evidence: (1) From Tabs. 14 to 16, we show that T-DCS$_{FR}$ (T-DCS for feature reconstruction attacks) values are much lower than the T-DCS values of other attack types, indicating inferior defense capabilities under feature reconstruction attacks. This phenomenon is consistent with the fact that feature reconstruction attacks, especially TBM which uses auxiliary data for reconstruction, result in much stronger attack capabilities (shown in Figs. 3, 11 and 12). Here we also show the visualization of the reconstructed images of TBM under different defense methods in [this figure (link)](https://anonymous.4open.science/r/VFLAIR_discussion/5ressfl.png). It is apparent that although the reconstructed images are noisy, the contour can still be seen easily. This further validates the effectiveness of our proposed T-DCS in measuring the relative defense strength of different types of attacks. (2) The differences in DCS scores between splitVFL and aggVFL settings (in Section 6) are also consistent with the human domain knowledge that splitVFL is less vunerable to attacks compared to aggVFL, which is consistent with findings in [1], demonstrating the effectiveness of the metrics.
- **Alignment with human intuition.** By definition (Eq. (1)), our DCS score measures its closeness to a perfect defense (Fig. 2).  Intuitively, a higher DCS score represents a defense that achieves a lower AP at a higher level of MP compared to others. This definition aligns with the defense target, i.e. limiting the AP to a lower level while maintaining a higher level of MP. 

**Reference**

[1] Yan Kang, et al. "A framework for evaluating privacy-utility trade-off in vertical federated learning". arXiv preprint arXiv:2209.03885, 2022

Reviewer comment (Reproducibility):
The paper claims to “implement basic VFL training and evaluation flow under multiple model partition, communication protocols and attacks and defences algorithms using datasets of different modality”. But it lacks a clear exposition of the workflow. What is the training and evaluation flow in VFLAIR and how does the workflow facilitate the benchmarking?

Author response:
Thanks for the suggestion. We further demonstrate the workflow of VFLAIR in Appendix D. We also post Fig. 10 in Appendix D here ([this figure (link)](https://anonymous.4open.science/r/VFLAIR_discussion/workflow_new.png)) for direct access  .
In summary, VFLAIR contains 4 key modules, namely Config Module, Load Module, Train \& Evaluate Module and Metrics Module.

In the workflow, the Config Module first processes the user-specified configurations, then passes it to the Load Module for party preparation. Each party separately loads its dataset and local model in the Load Module. Afterwards, the training pipeline starts in the Train \& Evaluate Module. Communication protocols and all the supported defense (all are training time defense) are implemented and integrated into the training pipeline. Training-time attacks (e.g. TB and NTB attacks) are integrated and evaluated within the training pipeline while inference-time attacks that do not affect the training flow (e.g. LI and FR attacks) are launched and evaluated after the training pipeline. Finally, the Metrics Module processes the recorded results and produces evaluation metrics including MP, AP, communication rounds (\#Rounds), amount of information transmitted between parties each round (Amount), execution time (EXec.Time), DCS, T-DCS and C-DCS.

Reviewer comment (Presentation):
Following on the above point, the evaluation section is not organised systematically according to model partition, communication protocols, and attack and defence algorithms. The current evaluation section only amounts to a compilation of experimental results, which requires a more structured, systematic and coherent organization.

Author response:
Thank you for your constructive suggestions. We have reordered the results and discussions systematically in Section 6 in our paper, which now exhibits the results according to model partition, communication protocols, encryption, number of participants, model architectures, attacks and defenses etc. We have also summarized and highlighted insights from our findings at the beginning of each evaluation section. Please refer to our modified manuscript for details.

Reviewer comment (Presentation):
There is only one paragraph in related work. There is no need to employ a bullet point at the beginning.

Author response:
Thanks for pointing this out. We have removed the bullet point in our Related Work section according to your advice.

Reviewer comment (Presentation):
The insights in Section 6 should be highlighted. Currently, the paragraph is too long (especially Section 6.2) and readers are hard to find interesting results from the experiments. I suggest the authors put the insights at the beginning of each subsection.

Author response:
Thanks for the suggestions. We have added and highlighted the summary of insights at the beginning of each paragraph throughout section 6. Please see our modified manuscript.

Reviewer comment (Experiments):
The communication protocols in the library are not rich. Only two methods are considered.

Author response:
We have added 3 new communication protocols, namely Quantizate [1], Top-k [1] and CELU-VFL [2] into our benchmark esperiments with the results shown in Tab. 6 in Section 6.1. We also plan to include more communication protocols  in the future. We also place the results here for easier access.

(3) Tab. 6 (in paper). MP, communication rounds (\#Rounds), amount of information exchanged per round (Amount) and total amount of exchanged information (Total) with different communication protocols of NN-based VFL under **aggVFL** setting. Q=5 when **FedBCD** and **CELU-VFL** are used, otherwise Q=1. For **Quantize**, b=16 while for **Top-k**, top 90% of elements are kept in forward local model prediction. 'Total' column is the total amount that equals to \#Rounds x Amount.
|          | MNIST           | MNIST   | MNIST      |  MNIST    | NUSWIDE         | NUSWIDE |  NUSWIDE   | NUSWIDE   |
| -------- | --------------- | ------- | ---------- | --------- | --------------- | ------- | ---------- | --------- |
|          | MP              | #Rounds | Amount(MB) | Total(MB) | MP              | #Rounds | Amount(MB) | Total(MB) |
| FedSGD   | 0.972$\pm$0.001 | 150     | 0.156      | 23.438    | 0.887$\pm$0.001 | 60      | 0.039      | 2.344     |
| FedBCD   | 0.971$\pm$0.001 | 113     | 0.156      | 17.656    | 0.882$\pm$0.001 | 26      | 0.039      | 1.016     |
| Quantize | 0.959$\pm$0.006 | 161     | 0.117      | 18.867    | 0.881$\pm$0.002 | 94      | 0.029      | 2.754     |
| Top-k    | 0.968$\pm$0.001 | 150     | 0.148      | 22.266    | 0.887$\pm$0.001 | 60      | 0.037      | 2.227     |
| CELU-VFL | 0.971$\pm$0.002 | 105     | 0.156      | 16.406    | 0.880$\pm$0.001 | 25      | 0.039      | 0.977     |

**Reference**

[1] Castiglia, Timothy J., et al. "Compressed-vfl: Communication-efficient learning with vertically partitioned data." International Conference on Machine Learning. PMLR, 2022.

[2] Fu, Fangcheng, et al. "Towards communication-efficient vertical federated learning training via cache-enabled local updates." arXiv preprint arXiv:2207.14628 (2022).

Reviewer comment (Reproducibility):
Can you demonstrate how to use VFLIR and how to extend it?

Author response:
In order to better illustrate the systematic design of VFLAIR, we add Fig. 10 in Appendix D (see also [this figure (link)](https://anonymous.4open.science/r/VFLAIR_discussion/workflow_new.png)) to demonstrate the key modules and workflow of VFLAIR. We also add a step by step guidance for using and extending VFLAIR in Appendix C. In summary, building on a modularized design of components covering all essential aspects of a VFL system, using VFLAIR only requires 5 easy steps as shown in Fig. 8 in Appendix C (see also [this figure (link)](https://anonymous.4open.science/r/VFLAIR_discussion/demo/demo_how_to_use.png)), with a single line to run: "python main_pipeline.py --configs my_config".  

For extending VFLAIR, we summarized the necessary steps and also included 2 demo cases ("how to add a new attack" and "how to add a new local model") in Appendix C. In summary, extending VFLAIR requires only modifying the configuration file, implementing code by inheriting corresponding classes (e.g. attacker) or by adding the model architecture and run. 

Please see Appendix C and D for more details.