Paper: VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks (ICLR-2024)

Reviewer comment (Experiments):
From another point of view, as synthetic datasets can be split in different ways, focusing on either importance or correlations, real-world datasets are still in more demanded. Thus, offering a more realistic VFL dataset in a related benchmark would be better.

Author response:
2. _"Offering a more realistic VFL dataset in a related benchmark would be better"_

In response to the need for a broader range of real-world VFL datasets, we have developed a new image-to-image VFL dataset, named "Satellite." This dataset fills a crucial gap, as there currently are no image-image VFL datasets available in the existing benchmarks like NUS-WIDE, Vehicle, and FedAds. The construction process and specifics of the Satellite dataset are comprehensively detailed in Appendix E. Additionally, we have conducted evaluations using SplitNN on this dataset, presented in Appendix G.4, to demonstrate its potential for improvement in VFL scenarios. This initiative not only enhances the diversity of our VFL benchmarks but also contributes a novel dataset to the field, broadening the scope for future research and application.

>In this section, we outline the construction process of the Satellite dataset, which was adapted from the WorldStrat dataset~\cite{satellite}, originally intended for high-resolution imagery analysis. The Satellite dataset encompasses Point of Interest (POI) data, each associated with one or more Areas of Interest (AOI). Every AOI incorporates a unique location identifier, a land type, and 16 low-resolution, 13-channel images, each taken during a satellite visit to the location.

In our revised manuscript, we acknowledge the limitations of expanding the real scope due to dataset size constraints. To mitigate this, we provide additional insights in Section 4.4, demonstrating that VertiBench-synthetic datasets effectively represent the relative performance of VFL algorithms on real-world datasets. This approach not only broadens the applicability of our benchmarks but also offers a practical solution for evaluating VFL algorithms in scenarios where large-scale real datasets are not readily available.

Reviewer comment (Experiments):
The proposed correlation-based split method may face scalability challenges when applied to a large number of parties, which could limit its practicality in certain scenarios. Could you provide more discussion about this?

Author response:
3. _"correlation-based split method may face scalability challenges when applied to a large number of parties, which could limit its practicality in certain scenarios."_

In Appendix D.3, we provide detailed timings for feature splits across datasets, encompassing a range of up to 0.58 million instances and 20,000 features. Additionally, we conducted scalability tests for the correlation-based split method on synthetic data of varying sizes.

The importance-based method proves to be highly efficient, completing within 30 seconds across all datasets tested. This method, which randomly splits without necessitating the computation of Shapley values, still retains some Shapley-related properties, ensuring scalability for large datasets.

Conversely, the correlation-based method, while not as rapid, is sufficiently efficient for practical applications. Our scalability tests demonstrate that this method can handle up to 10,000 features, with the time cost increasing approximately linearly with the number of features. This level of efficiency is adequate for most datasets typically used in VFL, such as realsim and CIFAR10.

We emphasize that the permutation-based optimization in VertiBench is an orthogonal problem. Should more advanced optimization algorithms become available, they can be seamlessly integrated to enhance VertiBench's efficiency.

We acknowledge the misunderstanding of the concern (3) regarding the scalability of the correlation split method in relation to party size. To address this, we conducted additional experiments illustrated in Figure 10c (Appendix D.3), focusing on the impact of increasing party numbers on efficiency. The results show that within a fixed global dataset, the number of parties has minimal effect on the efficiency of the correlation split method. We trust this experiment more accurately addresses your concerns.

Reviewer comment (Evaluation):
Is it possible to use one metric for reflecting both importance and correlation? Or say how should a user trade-off between these two factors in comparing different VFL algorithms?

Author response:
4. _"Is it possible to use one metric for reflecting both importance and correlation? Or say how should a user trade-off between these two factors in comparing different VFL algorithms?"_

In our revised Section 2.1 and Appendix H.1, we clarify the rationale behind choosing specific split methods depending on the scenario. 

>For datasets with nearly independent features, the low inter-party correlation makes correlation-based splits less meaningful, suggesting the superiority of importance-based feature splits. Conversely, in datasets with highly correlated features, assessing individual feature importance becomes impractical, making correlation-based splits more suitable due to varying inter-party correlations.
>
>Importance and correlation are treated as orthogonal evaluation factors applicable in distinct scenarios. While there may be an intrinsic link between them, our experiments indicate that focusing on one factor at a time yields explainable results reflective of real-world performance. As discussed in Appendix H.1, the interplay between importance and correlation can be complex. A joint optimization for both factors might be computationally intensive and less explainable, while providing limited additional insights. 

Essentially, splitting based on individual factors offers two key advantages:

- **Efficiency**: Joint optimization of multiple factors is substantially more complex and resource-intensive compared to optimizing a single factor.

- **Explainability**: Separate factor optimization allows for clearer insights into VFL algorithm performance. For instance, as the balance parameter $\alpha$ increases, GAL's performance tends to drop while FedOnce improves. This suggests GAL is better suited for imbalanced datasets and FedOnce for balanced ones. This inference aligns with real-world VFL dataset performances. In the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. Conversely, in the NUS-WIDE dataset, GAL's performance is comparable to FedOnce, differing by less than 0.3%.

Furthermore, although VertiBench doesn't support simultaneous factor-based splitting, it is capable of evaluating both importance and correlation effectively. This is exemplified in Figure 2, showcasing its capability in assessing both factors concurrently.

Reviewer comment (Theory):
**Unclear Independent Assumption**: From the beginning (the end of Section 2.1), the reason for omitting the correlation between Party Importance and Party Correlation is not well-described. A rough or intuitive relationship would be better to explain such a choice (e.g., the independent assumption is more concise and sufficient for evaluating most cases, or explain why it is so hard to capture their correlation). Since there are two metrics representing feature diversity, it can be confusing to determine which one to use, which one is better suited in specific cases, and whether it is possible and reasonable to evaluate a specific method in a 2D grid of $\alpha$ and $\beta$.

Author response:
In our revised Section 2.1 and Appendix H.1, we clarify the rationale behind choosing specific split methods depending on the scenario. 

>For datasets with nearly independent features, the low inter-party correlation makes correlation-based splits less meaningful, suggesting the superiority of importance-based feature splits. Conversely, in datasets with highly correlated features, assessing individual feature importance becomes impractical, making correlation-based splits more suitable due to varying inter-party correlations.
>
>Importance and correlation are treated as orthogonal evaluation factors applicable in distinct scenarios. While there may be an intrinsic link between them, our experiments indicate that focusing on one factor at a time yields explainable results reflective of real-world performance. As discussed in Appendix H.1, the interplay between importance and correlation can be complex. A joint optimization for both factors might be computationally intensive and less explainable, while providing limited additional insights. 

Essentially, splitting based on individual factors offers two key advantages:

- **Efficiency**: Joint optimization of multiple factors is substantially more complex and resource-intensive compared to optimizing a single factor.

- **Explainability**: Separate factor optimization allows for clearer insights into VFL algorithm performance. For instance, as the balance parameter $\alpha$ increases, GAL's performance tends to drop while FedOnce improves. This suggests GAL is better suited for imbalanced datasets and FedOnce for balanced ones. This inference aligns with real-world VFL dataset performances. In the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. Conversely, in the NUS-WIDE dataset, GAL's performance is comparable to FedOnce, differing by less than 0.3%.

Furthermore, although VertiBench doesn't support simultaneous factor-based splitting, it is capable of evaluating both importance and correlation effectively. This is exemplified in Figure 2, showcasing its capability in assessing both factors concurrently.

Reviewer comment (Evaluation):
**Seemingly Case-Restricted Significance**: As shown in Figure 5, only C-VFL reflects a significant response to the variation of feature diversity, and all other methods perform consistently robust on all datasets. It seems like there are some specific reasons tailored to C-VFL that cause this observation, and further, **it makes me doubt whether it is meaningful to conduct such feature diversity evaluation for non-compression-based VFL methods**. This may largely reduce the potential impact of this work. Besides, as shown in the right part of Figure 7, the performance rank of GAL observed on synthetic datasets is different from real ones and shows significantly low performance. The consistency level is not as strong as claimed by words.

Author response:
To address concerns about the effect of $\alpha,\beta$, we introduced a new baseline, FedOnce, significantly influenced by split factors. For the detailed values of the performance shifts of other VFL algorithms, which may be overshadowed by C-VFL's variability, refer to Table 9. Here's a summary of how each algorithm is affected by $\alpha$ and $\beta$:

- C-VFL: Performance varies up to 40% with $\alpha$ and up to 20% with $\beta$.
- FedOnce: Shows a 10% variation with $\alpha$ on the letter dataset, and 5-7% on covtype, gisette, and epsilon; affected by 1-3% with $\beta$ on covtype and epsilon.
- GAL: Impacted by 8% with $\alpha$ on letter and radar, and 2-5% on *all* other datasets; minimal impact from $\beta$.
- SplitNN: Shows minimal variation due to robustness (though at a higher communication cost).
- FedTree/SecureBoost: Not affected by feature split, theoretically matching centralized training performance.

Notably, on the letter dataset, the performance ranking between FedOnce and GAL switches as $\alpha$ increases from 0.1 to 100. This suggests that GAL is better suited for imbalanced VFL scenarios, while FedOnce excels in balanced splits. 

This trend aligns with real VFL datasets: For the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. In contrast, on the NUS-WIDE dataset, both GAL and FedOnce show closely matched performance (within 0.3% difference), reinforcing that these synthetic dataset conclusions can effectively mirror real-world scenarios.

Reviewer comment (Presentation):
It is confusing when reading "VertiBench, when parameterized with the same $\beta$, exactly reconstruct the real feature split of Vehicle dataset." in Section 3.2 and comparing Figure 4 (a) vs (b), since you do not introduce how we can get $\alpha$ and $\beta$ for real VFL datasets. There should be a kind hint to remind readers to refer to Section 3.3 for details.

Author response:
We have moved this part to Appendix D.2 according to Reviewer myAZ's advice.

Reviewer comment (Experiments):
There are 2 real VFL datasets and 7 centralized datasets used in the experiment. However, the introduction of the two real VFL datasets is missing, it should be clearly described in section 4.2. This confused me a lot at the beginning.

Author response:
We acknowledge the earlier confusion and have now added clarifying introductions in both Section 4.2 and Appendix F for enhanced understanding.

Reviewer comment (Writing):
The citation for Shapley-CMI is likely incorrect in line 5 of Section 2.2; the given citation is about the "Shapley Taylor Interaction Index". However, the correct one is given in Line 3 after Theorem 1.

Author response:
We have corrected this wrong citation.

Reviewer comment (Novelty):
**Citation Absence or Overclaim**: As is commonly expected of benchmark papers, they are usually anticipated to provide a comprehensive literature review of related methods in this domain and clarify their connection with existing benchmark papers. However, the discussion of a recent benchmark paper, [1]FedAds, is omitted, and some splitNN-based methods (e.g., [2]FedHSSL, [3]JPL, and others listed in Table 3 of work[1]) as well as other types of tree-based methods (e.g., [4]Federated Forest) are omitted in discussion. Although it is not necessary to evaluate all these methods, a comprehensive literature review is usually beneficial for a benchmark paper to be solid. From this perspective, it would be more appropriate to restrict it as a benchmark **only for major VFL method types**, since the paper excludes some sub-type methods.

Author response:
In Appendix B, we have thoroughly discussed and cited all mentioned studies, including a new addition, the FedOnce algorithm, to our benchmark for deeper insights. We acknowledge the challenge of evaluating all existing VFL codes, primarily due to the unavailability of open-source versions of many algorithms. Our primary objective is to demonstrate the efficacy of VertiBench across various VFL types, rather than evaluate as many algorithms as possible. Recognizing the diverse subtypes within VFL, as indicated, we have stated this limitation in our introduction and Section 4.1.

> In this paper, we concentrate on the primary types of VFL, acknowledging that there are various subtypes as identified in (Liu et al., 2022). Exploring these subtypes in depth will be an objective of our future research efforts.

Reviewer comment (Experiments):
Section 2.3 discusses various party correlation metrics, with the paper employing Spearman rank correlation in Algorithm 1. It would be beneficial to explore whether Spearman rank correlation can be replaced by the other mentioned correlations and, if possible, provide a comparison of their performance. This comparison would enhance the understanding of the selected correlation metric's effectiveness.

Author response:
We have include the experiments of performance split by Pearson coefficient in Appendix G.10.

> In comparing Spearman rank and Pearson correlation coefficients, we find Spearman rank effective for non-linear correlations, while Pearson is suitable for linear correlations. Figure 26 illustrates the performance differences on datasets split using these metrics.
>
>For linearly correlated datasets like gisette, Pearson's coefficient is preferable. It accurately captures the enhanced performance of C-VFL on highly correlated splits, a nuance Spearman rank-based splitting misses. Conversely, for non-linearly correlated datasets such as radar, Spearman rank-based correlation is more effective, highlighting variations in C-VFL performance that Pearson-based splits overlook. Therefore, the choice of correlation coefficient should align with the dataset's correlation characteristics.

Reviewer comment (Experiments):
The paper discusses the communication efficiency of VFL algorithms, which seems to be minimally impacted by data distribution. Given this, it might be more appropriate to allocate more space to extending the experiments in Section 4.5. Moving additional explanations and results, which may be included in the appendix, to the main paper, would provide a more comprehensive understanding of the proposed method and its real-world applicability.

Author response:
We have relocated the original Section 4.4 to the Appendix, noting that communication efficiency in VFL is largely unaffected by variations in feature split. Specifically, if each party possesses an equal number of features, the communication cost remains consistent. Moreover, even in scenarios where the number of features differs across parties, the impact on communication cost is negligible compared to the communication cost gap arising from distinct algorithmic designs.

Reviewer comment (Theory):
**The specific form of the correlation of two party correlation $\textrm{Pcor}$**. It’s not clear to me why the metric Pcor is defined as roughly speaking (upto an additional $\frac{1}{\sqrt{d}}$ constant) the standard deviation of the correlation matrix’s singular values. Consider a case where $m_i = m_j$ and the two parties have exactly the same set of features $X_i = X_j$ where the column features of $X_i$ are pairwise independent. In this case, the correlation matrix is the identity matrix $\textrm{cor}(X_i, X_j) = I_{m_i \times m_i}$ with singular values all being $1$. Then under the metric Pcor, these two parties' features would have 0 “correlation” because the singular values have zero variance while in reality there is indeed a somewhat strong notion of correlation (their features are identical). I would appreciate if the authors can provide an explanation of the reasoning in choosing this new metric and whether my suggested scenario would indeed be measured as having 0 “correlation” using pcor.

Author response:
We have refined the definition of Icor in our revision, shifting from the original average of $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_j)$ to averaging the relative difference $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_j) - \text{Pcor}(\mathbf{X}_i,\mathbf{X}_i)$. This new approach leads to several key advancements:

- Theoretical validation that **optimizing Icor results in ideal feature splits in optimal scenarios** (Theorem 1), consistent with our empirical findings on complex real datasets (see Figure 5 in Appendix D.2).
- Provision of intuitive explanations for the design of Pcor metrics.
- Visualization of the relationship between singular values, singular vectors, and inter-party correlation.
- Theoretical demonstration that the range of Pcor is within $[0,1]$.
- Theoretical proof of that Pcor and mcor (Taylor, 2020) are equivalent in measuring inner-party correlation.
- Comprehensive re-evaluation of all experiments concerning correlation-based splits.

Icor effectively measures the overall relative inter-party correlation within a global dataset. Previously, overlooking inner-party correlations of $\mathbf{X}_i$ and $\mathbf{X}_j$ led to discrepancies. High inner-party correlation typically results in increased Pcor values with other parties. The amendment to relative Pcor addresses this issue. For instance, consider two independent features $a_1, a_2$ forming two parties such that $\mathbf{X}_i=\mathbf{X}_j=[a_1,a_2]$. In this setup, $\text{Icor}(\mathbf{X}_i, \mathbf{X}_j)=0$, reflecting significant relative inter-party correlation (Icor is usually negative). Conversely, if we swap a feature to have $\mathbf{X}_i=[a_1,a_1], \mathbf{X}_j=[a_2,a_2]$, we get $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_i)=1$, $\text{Pcor}(\mathbf{X}_j,\mathbf{X}_j)=0$, leading to $\text{Icor}(\mathbf{X}_i, \mathbf{X}_j)=-1$, indicative of a low inter-party correlation, aligning intuitively with expectations.

Reviewer comment (Experiments):
**Exploring the performance variations of feature partitioning under the same $\alpha$ and $\beta$**. For party balance using $(\alpha_1, \ldots, \alpha_K)$, the feature partitioning procedure (algorithm 5) has internal randomness, thus making two runs of the same algorithm produce feature partitions that are likely different. In this case, I would expect to see some results on the performance distribution of the same split learning method over these different feature splits and understand how much variation there is. Similarly, I’m not sure if the argmin step (line 6) of Algorithm 1 could also yield multiple different solutions. If so, I think the authors should also explore the performances variation over these different partitions. Taking one step further, I wonder if there are algorithm ranking differences for the same $\alpha$ and $\beta$ but different feature partition realizations. If there are such ranking differences, then the authors should provide more detailed suggestions on how to use their benchmark (how many such splits should be used/averaged for such comparison). If there aren’t, I suggest the authors provide official splits after performing the sampling/optimization themselves so that future papers using these evaluation benchmarks would have a consistent comparable reporting.

Author response:
In our analysis presented in Figure 24 and 25, we investigated the convergence of both mean and variance in algorithm performance as the number of test repetitions increased. We observe both mean and variance converges after 5-10 runs. After the convergence, there are notable differences among the algorithms: C-VFL exhibited the largest variance, whereas SplitNN and FedTree maintained low variance levels. This study highlights the significant dependency of performance variance on the specific algorithm employed.

It's important to note that variance in dataset splits is a characteristic to be leveraged, not a problem to be solved. Given that feature splits at similar importance or correlation levels are inherently non-unique, observing performance variances under identical split configurations can actually offer insights into the robustness of VFL algorithms. For instance, while SplitNN and FedTree demonstrate consistent robustness across various splits, C-VFL exhibits considerable variance in most scenarios.

To facilitate a consistent comparison among algorithms, we recommend conducting at least 5-10 repeated experiments and reporting both the mean and variance of performances. Our observations indicate that five repetitions generally yield stable mean and variance for most evaluated methods. If an algorithm displays large variance after five runs, it probably suggests potential challenges in handling real-world feature splits with similar levels of importance or correlation.

We also acknowledge and appreciate the suggestion for providing official splits to enable more consistent evaluations and rankings, recognizing its potential utility. We will provide them when releasing benchmark.

Reviewer comment (Presentation):
**Figure 2, 3, 4.** I appreciate the authors including Figure 2, 3, 4 in the paper. However, I believe they are basic sanity checks of the correctness of Theorem 1 and the correctness of the implementation of Algorithm 1, and do not provide additional new insights. I believe they should instead be put into the Appendix.

Author response:
We have moved Figure 2,3,4 and their related contents to Appendix D.2.

Reviewer comment (Experiments):
__Communication efficiency__ Section 4.4 seems detached from the rest of the paper, as they do not explore in the axes of party balance ($\alpha$) and party correlation ($\beta$) explored in the rest of the paper.

Author response:
We have relocated the original Section 4.4 to the Appendix, noting that communication efficiency in VFL is largely unaffected by variations in feature split. Specifically, if each party possesses an equal number of features, the communication cost remains consistent. Moreover, even in scenarios where the number of features differs across parties, the impact on communication cost is negligible compared to the communication cost gap arising from distinct algorithmic designs.

Reviewer comment (Experiments):
Can the authors provide further discussion of the scalability of their two partitioning methods? For example, how scalable is the computation of Shapley values used in the experiments (run-time as a function of the number of features)? How many partition dimensions can the optimization method BRKGA efficiently solve?

Author response:
In Appendix D.3, we provide detailed timings for feature splits across datasets, encompassing a range of up to 0.58 million instances and 20,000 features. Additionally, we conducted scalability tests for the correlation-based split method on synthetic data of varying sizes.

The importance-based method proves to be highly efficient, completing within 30 seconds across all datasets tested. This method, which randomly splits without necessitating the computation of Shapley values, still retains some Shapley-related properties, ensuring scalability for large datasets.

Conversely, the correlation-based method, while not as rapid, is sufficiently efficient for practical applications. Our scalability tests demonstrate that this method can handle up to 10,000 features, with the time cost increasing approximately linearly with the number of features. This level of efficiency is adequate for most datasets typically used in VFL, such as realsim and CIFAR10.

We emphasize that the permutation-based optimization in VertiBench is an orthogonal problem. Should more advanced optimization algorithms become available, they can be seamlessly integrated to enhance VertiBench's efficiency.