Paper: Effective and Efficient Federated Tree Learning on Hybrid Data (ICLR-2024)

Reviewer comment (Theory):
I did not understand theorem 3 at first. How does a meta rule that "ended by $F_g$ differ from a meta rule where $F_g$ is the last layer? Aren't they the same thing? Does the intersection operator here suggest ordering as well? After reading the proof of theorem 3, I understood what the author(s) meant. In any path from the root node to a leaf node, there is a possibility that a guest split condition stays above a host split condition, you just want transform the tree so that such possibility does not exist, is that right? If so, I suggest to restate theorem 3 in a way that is more understandable.

Author response:
Yes, we want to transform the tree such that the guest split condition stays in the bottom layers. To clarify this transformation process, we have revised Theorem 3 as follows:

**Theorem 3.** Suppose $S_{m}:=F_h \cap ... \cap F_g$ is a meta-rule in tree $\theta_A$ where $F_g$ is a split condition using the feature from the guests. For any tree path in tree $\theta_A$ involving the split nodes in $S_m$, we can always reorder the split nodes in the tree path such that $F_g$ is in the last layer. Moreover, naming the tree after the reordering as $\theta_B$, we have $E[f(\textbf{x}; \theta_A)] = E[f(\textbf{x}; \theta_B)]$ for any input instance $\textbf{x} \in \mathcal{D}$.

Reviewer comment (Theory):
What has caught me a surprise is that theorem 3 is not used in a conventional way. Instead of using theorem 3 to improve upon a given tree, the authors redesign a new tree where guest split conditions are always below host split conditions. This is a good point that appears to have been presented somewhat lightly in the paper.

Author response:
We appreciate your observation regarding our Theorem 3. Indeed, we employ Theorem 3 as a foundational principle to design a new tree structure where guest split conditions are strategically positioned below those of the host. We have highlighted this point in the revision, including **Section 1, Section 3.2, and the beginning of Section 4**.

Reviewer comment (Presentation):
In my view, the flow of the paper could better if the authors presented the HybridTree inference before presenting the HybridTree training. Once the audience knows how inference works, the training components make much more sense.

Author response:
We think it may not be easy for some audiences to appreciate the inference process before introducing how the tree is divided among the parties. To address your concern, we have added a paragraph to briefly introduce the training and inference processes at the beginning of **Section 4** in the revision. In this paragraph, we mentioned that each tree is divided into multiple parties, and the host and guest parties collaboratively build the split path of an input instance and make the prediction. Given this background, the audience can understand the training component more easily.

Reviewer comment (Presentation):
The paper is not easy to follow.

Author response:
We apologize for any difficulties encountered in following the initial version of our paper and appreciate your feedback on this aspect. We have improved the writing in the revision, including 1) We have updated the problem statement and added **Figure 1** to clarify the hybrid data setting more clearly. 2) We have improved **Definition 1, Theorem 3, and Figure 3(b)** to make our tree transformation theorem easier to understand. 3) We have added a paragraph to introduce the training and inference process of our algorithm in general at the beginning of **Section 4** to improve readability. 4) We have added a paragraph and added more annotations in Algorithm 1 to further demonstrate the training process of our algorithm in **Section 4.1**. 5) We have added a table to summarize the notations used in the paper in **Table 4 of Appendix B.1**.

Reviewer comment (Writing):
It is better to provide more analysis or explanation on the training process in section 4.1 to let readers well understand how HybridTree handles hybrid data and makes their contribution to the improvement.

Author response:
We appreciate your suggestion for a more detailed explanation of the training process, especially in the context of handling hybrid data. In response, we have enriched **Section 4.1** with an additional paragraph that delves into the nuances of the training process of HybridTree. Furthermore, we have enhanced Algorithm 1 with more detailed annotations for greater clarity. 

In general, there are three steps in the whole training process: 1) The host party updates a subtree individually (Lines 1-9); 2) The host party sends the encrypted intermediate results to the guest parties (Lines 10-13); 3) The guest parties update the bottom layers individually and send back the encrypted prediction values (Lines 14-21). Since HybridTree does not require accessing all features and instances when updating each node, it can handle the hybrid data case where each party only has partial instances and features. Moreover, based on our analysis in Section 3, by updating the bottom layers using the guests' features, the meta-rule knowledge of the guest parties can be effectively incorporated.

Reviewer comment (Novelty):
Although there is a relatively thorough literature review in the related work part, I prefer to see a discussion on the relation of this work, especially the specific methods.

Author response:
Thank you for your suggestion to delve deeper into the comparative aspects of HybridTree in relation to existing federated GBDT studies. We have added the comparison between HybridTree and related federated GBDT studies in **Table 13 of Appendix D**. 

1) Novel Setting for Federated Learning: HybridTree introduces an innovative approach tailored for the hybrid data setting. This setting, which has not been extensively explored in previous federated GBDT studies, is increasingly relevant in diverse real-world applications where data characteristics vary significantly across different parties.

2) Layer-Level Knowledge Aggregation: Contrasting the common node-level or tree-level knowledge aggregation methods found in existing studies, HybridTree employs a layer-level aggregation approach. This method allows for a more efficient and effective incorporation of knowledge from guest parties. By appending layers specific to each party’s data, HybridTree ensures that the model benefits from the diverse insights and characteristics inherent in the hybrid data.

Reviewer comment (Writing):
"Eq 2" —> "Eq (2)".

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
$D_{G_i}= \{ x \} (x \in R^{ d_{g_i}})$ —> $D_{G_i}= \{ x | x \in R^{ d_{g_i}}\} $,  $D_H= { x, y } (x \in {R^{d_h}})$ —> $D_H= \{ x, y | x \in R^{d_h}\} $.

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
"Last, guests update the following lower layers of the tree using their local features and received encrypted gradients and send back the encrypted prediction values." —> "Last, guests update the following lower layers of the tree using their local features and **receive** encrypted gradients**,** and send back the encrypted prediction values."

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
"Then, the gain of the split value is defined by the loss reduction after split, which is" —>"Then, the gain of the split value is defined by the loss reduction after **the split**, which is".

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
"The best-split point is selected among these split candidates. When reaching the maximum depth or the gain is always negative, the current node will become a leaf node." —> "The **best split** point is selected from these candidates. If the tree reaches the maximum depth or **if** the gain remains negative, the current node becomes a leaf node."

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
"FL on hybrid data is rarely exploited in the current literature. Zhang et al. (2020) proposes to…. Liu et al. (2020) applies transfer" —> "Zhang et al. (2020) **propose** to…. Liu et al. (2020) **apply** transfer".

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Writing):
“For simplicity, we start from a single-host with multi-guest setting” —> "For simplicity, we start from a scenario involving a single host with multiple guests."

Author response:
We greatly appreciate your attention to detail in identifying these minor errors. We have reviewed the entire manuscript and corrected all the typos and inaccuracies you pointed out. To ensure thoroughness, we conducted an additional round of proofreading to identify and rectify any remaining errors.

Reviewer comment (Presentation):
The paper is not very well-presented and is hard to follow. In the hybrid setting considered, it is unclear what are the relative relations of the guest parties. In the introduction, it appears that they share the same feature space but have different sample IDs; however, in section 3.1 they appear to have different dimensions and unclear alignment. It is suggested that the paper properly define the problem setting. A figure on how data is partitioned by different parties would also help.

Author response:
We apologize for any confusion caused by our presentation. Our primary focus is on scenarios where guest parties share the same feature space but possess different sample IDs. Examples include a payment system across multiple banks or a hospital network utilizing patient data from various sources such as iWatches. We have revised **Section 3.1**, concentrating on the homogeneous guest feature space setting to prevent ambiguity. Additionally, we have added **Figure 1** in the revised version, illustrating the data partitioning among different parties. This visualization should provide a clearer understanding of our setting.

Though our focus is on setting where guest parties share the same feature space but different sample IDs, our algorithm does not pose a limitation on the feature and sample space of the guest parties. In **Appendix C.3**, we also provide experimental results where the guest parties have different feature spaces and overlapped samples, where our approach still outperforms the other baselines.

Reviewer comment (Writing):
The algorithm is very messy with many undefined variables and notations, making it hard to process. It is also not clear how features from different guests are used in a collaborative manner in the approach due to the previously mentioned issues on presentation.

Author response:
Thank you for pointing out it. To enhance clarity, we have added **Table 4 in Appendix B**, which summarizes all the notations used in our algorithm. We have also expanded the annotations in Algorithm 1 for better understanding. 

Regarding the collaborative use of features from different guests, this is articulated in the GuestTrain function of Algorithm 1 (line 14, lines 16-21). Here, following the host party's update of a tree with its features and transmission of encrypted gradients and instance ID sets to the guest parties, these guests further expand the tree with deeper layers using their local features.

We have added an additional paragraph to summarize the training process. In general, there are three steps in the whole training process: 1) The host party updates a subtree individually (Lines 1-9); 2) The host party sends the encrypted intermediate results to the guest parties (Lines 10-13); 3) The guest parties update the bottom layers individually and send back the encrypted prediction values (Lines 14-21). Since HybridTree does not require accessing all features and instances when updating each node, it can handle the hybrid data case where each party only has partial instances and features. Moreover, based on our analysis in Section 3, by updating the bottom layers using the guests' features, the meta-rule knowledge of the guest parties can be effectively incorporated.

Reviewer comment (Experiments):
The experimental results are biased. The paper compares its model performance using data from multiple guests against FedTree and Pivot using data from one guest; therefore, the model performance gap is largely due to the utilization of additional data parties, not to the benefits of the algorithm. To be fair, the paper should compare them under the same settings, that is, multiple guests and host.

Author response:
As claimed in the paper, to the best of our knowledge, we are **the first ones to develop the federated GBDT algorithm on hybrid data**. While FedTree and Pivot support the vertical FL setting, they cannot handle the hybrid data setting with multiple guests. Another baseline TFL utilizes the data from all parties and adopts a tree-level aggregation. HybridTree significantly outperforms TFL as shown in our experiments.

Reviewer comment (Experiments):
Important baselines such as Secureboost [1] are missing.

Author response:
Initially, we excluded SecureBoost based on its similarity in model performance with FedTree, as both employ a comparable node-level aggregation design, which often results in overlapping model performance [2]. We have now included SecureBoost in our revised version.

SecureBoost has been added to the experimental comparisons in **Tables 1-4, Table 6, and Tables 8-12**. To maintain clarity and readability in our graphical representations, we opted not to include SecureBoost in figures such as Figure 6, where its performance curve closely aligns with that of FedTree. Our updated results confirm that FedTree's performance is indeed very similar to SecureBoost. Moreover, the efficiency of HybridTree also significantly outperforms SecureBoost.

[2] FedTree: A Federated Learning System For Trees

Reviewer comment (Evaluation):
The model performance of the proposed methods still appears to be a little inferior to the centralized setting, not exactly "comparable" as claimed. It is important to understand whether the proposed method is "lossless" or "lossy" and why. More detailed examinations and explanations are needed here.

Author response:
We appreciate your observation regarding the performance comparison with the centralized setting. Our method, indeed, does not achieve a completely lossless transfer compared to centralized solutions, and there are two primary reasons for this: 1) While our tree transformation in Section 3 ensures that the expected prediction value for any input remains unchanged, this mathematical expectation is an average measure. In practice, the prediction value for each individual sample may not align perfectly with this expected value, leading to some discrepancies. 2) Our algorithm is designed to integrate meta-rule knowledge through the training of additional layers by guest parties. As illustrated in Figure 3(a), this meta-rule knowledge is significant. However, certain aspects of knowledge contributed by guest parties are not encapsulated by meta-rules (e.g., about 10% of trees do not have meta-rules in AD). While our algorithm efficiently aggregates meta-rule knowledge, it does so at the cost of some accuracy to enhance overall practicality and efficiency.

Reviewer comment (Experiments):
Maybe the authors can consider the scenarios where clients have multi-modal data, where the data modalities are hybrid across clients.

Author response:
Thank you for highlighting this significant aspect. Indeed, our current method focuses on tabular data, primarily due to the inherent structural limitations of tree-based methods when dealing with multi-modal data. The scenario you've mentioned, involving clients with hybrid multi-modal data, presents a very interesting and challenging direction. We consider it as a future work using models besides trees.