Paper: Unsupervised Federated Graph Matching with Graphlet Feature Extraction and Separate Trust Region (ICLR-2024)

Reviewer comment (Experiments):
Despite the promising result, I have some questions regarding the fundamental assumption of this paper. The assumption is that nodes across networks with similar graphlet degree vectors are likely to be the same nodes. However, as graph structures are different in different networks, it may well happen that a node's structure changes a lot, e.g. an author who changes his main research focus. Thus, I would like to know how accurate the assumption is. Maybe the authors can provide some numbers to show.

Author response:
Thanks for the constructive comments. 
A graphlet is a small graph of size up to $k$ nodes of a larger graph, such as triangle, wedge, or $k$-clique, which describes the local topology of a larger graph.
A node's local topology can be measured by a graphlet feature vector, where each component denotes the frequency of one type of graphlets. Thus, a graphlet feature vector is one of node structure representation (Shervashidze et al., 2009; Kondor et al., 2009; Soufiani \& Airoldi, 2012; Jin et al., 2018; Tu et al., 2019). 
Based on structure, attribute, or embedding features, existing graph matching efforts often choose the node pairs with the smallest distances or largest similarities across the graphs are selected as the matching results (Man et al., 2016; Zhou et al., 2018a; Yasar \& Catalyurek, 2018; Li et al., 2019b;a; Chu et al., 2019; Fey et al., 2020).
This submission just follows this mainstream strategy for structure-based graph matching to conduct graph matching based on the graphlet features.
This work explores a general solution of federated graph matching for addressing the most common graph matching question: structure-based graph matching without considering other types of features.

For the case of the graphs with quite different structures, a popular method is to combine other types of features to alleviate the structure inconsistency issue, say attribute-based graph matching. When other types of features are similar with only dissimilar structure, it is still possible that two copies of the same nodes in different graphs are still most similar to each other and can be identified as the matching results, as long as more other features are integrated or by reducing the weight of structure features.

Reviewer comment (Theory):
I wonder whether a split-learning technique can achieve the goal of the separate trust region algorithm. From my understanding, the separate trust region algorithm is designed so that the pseudo matchings are kept at servers and not exposed to clients. This may well be achieved with some kind of split learning, where clients maintain their graph feature encoders, and a matching unit is trained at the server with the pseudo labels. In this case, there would be even no requirement of the Hessian matrices. Can the authors justify the necessity of using the separate trust region algorithm and the Hessian?

Author response:
Thanks for the kind suggestion. 
The time complexity of multiple graph matching typically increases exponentially with the number of graphs to be matched. If we choose to conduct both evaluation and optimization of the graph matching model at the server, then the computational capability of each client are idle. This will dramatically degrade the algorithm efficiency, especially large-scale graph matching is often time-consuming.

Thus, we separate model optimization from model evaluation in the trust region algorithm: (1) the server aggregates the local model parameter on each client into a global model parameter, runs and evaluates the global parameter, and computes the individual loss and the first-order gradient for each client; (2) each client computes the second-order Hessian and optimizes the local model.
The reason of using the second-order Hessian in the trust region algorithm is that the second-order Hessian can provide fast convergence in terms of number of training steps.
The second-order Hessian computation and the local model optimization are most compute-intensive over large graphs. Thus, we move these two operations to the clients for making full use of the computational capability of each client to improve the efficiency, while maintaining the privacy constraints.

Reviewer comment (Experiments):
I wonder what the effect is of the number of pseudo labels to the overall algorithm. Intuitively, this seems like a tradeoff, as when you include more pseudo labels, they tend to be less accurate and bring noise to the overall learning. I would like to see a result of such tradeoff.

Author response:
Thanks for the comments. 
Like other machine learning tasks, supervised graph matching methods usually achieve better performance than unsupervised graph matching approaches, since the former employs the strength of training data (i.e., pre-matched node pairs across graphs).
In this work, we propose to capture nodes' graphlet features to generate pseudo matched node pairs on different graphs across clients as the pseudo training data for leveraging the strength of supervised graph matching and improving the quality of unsupervised graph matching.
Thus, the pseudo training data has a positive effect on the graph matching results.

In the revision, Table 8 tests the influence of the pseudo training data for the performance of graph matching by varying the ratio of the pseudo training data from 20\% to 100\%. The ratio 100\% corresponds to the number of the pseudo matched node pairs used in our current experiments. The numbers are 3,041 on SNS, 1,264 over PPI, and 2,817 on DBLP respectively. As we can see, the performance scores continuously increase with increasing pseudo training data. This is consistent with the fact that more training data makes the graph matching models achieve better performance.

Reviewer comment (Experiments):
I also wonder what are the effects of the monte-carlo markov chain sampling methods and the quasi-Newton methods on the overall method runtime/efficiency. The authors made no experiments to analyze the effectiveness either design.

Author response:
Thanks for your helpful comments. 
In the revision, Figure 9 (b) reports the corresponding running time of our UFGM model by varying sampled graphlet number $O$ from 10 to 1,000. We make the observation on the quality and efficiency over three datasets: both the performance scores and the running time keep increasing when the sampled graphlet number is increasing. A rational guess is that a larger sampled graphlet number exchanges better performance with more sampling and processing time.

In the revision, we evaluate two versions of UFGM to show the strength of the quasi-Newton approximation for improving the efficiency while maintaining the quality federated graph matching. UFGM is the approximate version with the quasi-Newton approximation. UFGM-E is the exact version with the exact Hessian computation. The experiment results in Table 7 exhibit that the approximate version UFGM achieves slightly lower performance than the exact version UFGM-E but has much smaller running time. This demonstrates that the quasiNewton approximation method is able to dramatically improve the efficiency while maintaining the utility constraints.

Reviewer comment (Theory):
It seems that the loss function is operated on the 'encrypted' embeddings $\hat{v}$ instead of $v$. At this time, a normal non-singular matrix $K$ may just twist some dimensions of the vectors, while shrink some other dimensions. Or in other words, a normal non-singular matrix $K$ does not preserve distance (which is exactly the loss function). Thus, how does the loss work on encrypted vector embeddings when they are applied a transformation that does not maintain distance is a little beyond me. Do you need orthogonal ones (those that maintain distance)?

Author response:
Thanks for your great comments. Sorry for any confusion caused. In our current implementation, we first produce a random nonsingular matrix $K$ and then orthogonalize it to preserve the distances between the embedding vectors. We have added this clarification to the revision.

Reviewer comment (Evaluation):
How accurate are the pseudo labels from graphlet degrees?

Author response:
Thanks for the comments. 
Like other machine learning tasks, supervised graph matching methods usually achieve better performance than unsupervised graph matching approaches, since the former employs the strength of training data (i.e., pre-matched node pairs across graphs).
In this work, we propose to capture nodes' graphlet features to generate pseudo matched node pairs on different graphs across clients as the pseudo training data for leveraging the strength of supervised graph matching and improving the quality of unsupervised graph matching.
Thus, the pseudo training data has a positive effect on the graph matching results.

In the revision, Table 8 tests the influence of the pseudo training data for the performance of graph matching by varying the ratio of the pseudo training data from 20\% to 100\%. The ratio 100\% corresponds to the number of the pseudo matched node pairs used in our current experiments. The numbers are 3,041 on SNS, 1,264 over PPI, and 2,817 on DBLP respectively. As we can see, the performance scores continuously increase with increasing pseudo training data. This is consistent with the fact that more training data makes the graph matching models achieve better performance.

Reviewer comment (Experiments):
How necessary is the Hessian matrix? What if we use some sort of split learning?

Author response:
Thanks for the kind suggestion. 
The time complexity of multiple graph matching typically increases exponentially with the number of graphs to be matched. If we choose to conduct both evaluation and optimization of the graph matching model at the server, then the computational capability of each client are idle. This will dramatically degrade the algorithm efficiency, especially large-scale graph matching is often time-consuming.

Thus, we separate model optimization from model evaluation in the trust region algorithm: (1) the server aggregates the local model parameter on each client into a global model parameter, runs and evaluates the global parameter, and computes the individual loss and the first-order gradient for each client; (2) each client computes the second-order Hessian and optimizes the local model.
The reason of using the second-order Hessian in the trust region algorithm is that the second-order Hessian can provide fast convergence in terms of number of training steps.
The second-order Hessian computation and the local model optimization are most compute-intensive over large graphs. Thus, we move these two operations to the clients for making full use of the computational capability of each client to improve the efficiency, while maintaining the privacy constraints.

Reviewer comment (Experiments):
What is the effect of the number of pseudo labels?

Author response:
Thanks for the comments. 
Like other machine learning tasks, supervised graph matching methods usually achieve better performance than unsupervised graph matching approaches, since the former employs the strength of training data (i.e., pre-matched node pairs across graphs).
In this work, we propose to capture nodes' graphlet features to generate pseudo matched node pairs on different graphs across clients as the pseudo training data for leveraging the strength of supervised graph matching and improving the quality of unsupervised graph matching.
Thus, the pseudo training data has a positive effect on the graph matching results.

In the revision, Table 8 tests the influence of the pseudo training data for the performance of graph matching by varying the ratio of the pseudo training data from 20\% to 100\%. The ratio 100\% corresponds to the number of the pseudo matched node pairs used in our current experiments. The numbers are 3,041 on SNS, 1,264 over PPI, and 2,817 on DBLP respectively. As we can see, the performance scores continuously increase with increasing pseudo training data. This is consistent with the fact that more training data makes the graph matching models achieve better performance.

Reviewer comment (Evaluation):
What are the effects of the MCMC and the quasi-Newton method on the overall efficiency?

Author response:
Thanks for your helpful comments. 
In the revision, Figure 9 (b) reports the corresponding running time of our UFGM model by varying sampled graphlet number $O$ from 10 to 1,000. We make the observation on the quality and efficiency over three datasets: both the performance scores and the running time keep increasing when the sampled graphlet number is increasing. A rational guess is that a larger sampled graphlet number exchanges better performance with more sampling and processing time.

In the revision, we evaluate two versions of UFGM to show the strength of the quasi-Newton approximation for improving the efficiency while maintaining the quality federated graph matching. UFGM is the approximate version with the quasi-Newton approximation. UFGM-E is the exact version with the exact Hessian computation. The experiment results in Table 7 exhibit that the approximate version UFGM achieves slightly lower performance than the exact version UFGM-E but has much smaller running time. This demonstrates that the quasiNewton approximation method is able to dramatically improve the efficiency while maintaining the utility constraints.

Reviewer comment (Theory):
Does a random non-singular method suffice in privacy-preservation and loss computation? Or do we need an orthogonal one?

Author response:
Thanks for your great comments. Sorry for any confusion caused. In our current implementation, we first produce a random nonsingular matrix $K$ and then orthogonalize it to preserve the distances between the embedding vectors. We have added this clarification to the revision.

Reviewer comment (Experiments):
The scale of the experiment is a bit small. What will the performance look like on large-scale dataset?

Author response:
Thanks for your great comments. In the appendix, in order to validate the scalability of our UFGM method for the problem of unsupervised federated graph matching on large-scale datasets, we selected and split the original DBLP dataset into 20 graphs by publication year, ranging from 2002-2022, such that each graph has around 100,000 and 200,000 authors as nodes and coauthor relationships as edges respectively. We have reported the $Hits@K$ scores in Table 5. The experiment results exhibit that our UFGM method scales well on different datasets . This demonstrates that our UFGM method is scalable for addressing the problem of unsupervised federated graph matching, while maintaining the privacy requirement of FL.

Reviewer comment (Novelty):
Another concern is the benchmark comparison. Authors claim it is the first algorithm for federate graph matching, so it is better to emphasize how innovative compared with peer works.

Author response:
Graph matching (i.e., network alignment) is one of the most important research topics in the graph domain, which aims to match the same entities (i.e., nodes) across two or more graphs. There is still a paucity of techniques of effective federated graph matching (FGM), which is much more difficult to study and has many real applications, such as user account linking in different social networks and financial crime detection. However, directly sharing and inferring matched node pairs on different graphs across clients and local graphs over multiple clients gives rise to a serious privacy leakage concern, which is the motivation of this work, compared with centralized graph matching methods.

To our best knowledge, this work is the first to offer an unsupervised federated graph matching solution for inferring matched node pairs on different graphs across clients while maintaining the privacy requirement of FL, by leveraging the graphlet theory and trust region optimization. Our UFGM method exhibits three compelling advantages: (1) The combination of the unsupervised FGM and the encryption of local raw graph data is able to provide strong privacy protection for sensitive local data; (2) The graphlet feature extraction can leverage the strength of supervised graph matching with the pseudo training data for improving the matching quality; and (3) The separate trust region for pseudo supervised FGM is helpful to enhance the efficiency while maintaining the privacy constraints.

Reviewer comment (Experiments):
Scalability test and more discussions.

Author response:
Thanks for your great comments. In the appendix, in order to validate the scalability of our UFGM method for the problem of unsupervised federated graph matching on large-scale datasets, we selected and split the original DBLP dataset into 20 graphs by publication year, ranging from 2002-2022, such that each graph has around 100,000 and 200,000 authors as nodes and coauthor relationships as edges respectively. We have reported the $Hits@K$ scores in Table 5. The experiment results exhibit that our UFGM method scales well on different datasets . This demonstrates that our UFGM method is scalable for addressing the problem of unsupervised federated graph matching, while maintaining the privacy requirement of FL.

Reviewer comment (Experiments):
It seems the scope of the proposed method is specific as it seems to only be designed for federated graph matching. I wonder how hard it will be to generalize the proposed method to general federated learning.

Author response:
Thanks for your helpful comments. Different from the FL for computer vision and other graph learning tasks, such as image classification and graph classification, where the data on a client are often independent of the data on another client and stochastic gradient descent (SGD) optimization can be used to evaluate and optimize the local FL models without the need of accessing the data on other clients. However, graph matching aims to match the same entities (i.e., nodes) across two or more graphs. Thus, the FGM needs to infer matched node pairs on different graphs across clients. Directly accessing and sharing local graphs over multiple clients gives rise to a serious privacy leakage concern.
Therefore, the SGD optimization widely used in deep learning and FL fails to work on the clients in the FGM, since each client can access only its own local graph data and thus cannot update local loss based on the pseudo matched node pairs. We propose a separate trust region algorithm to separate local model optimization from model evaluation in the trust region algorithm for pseudo supervised FGM while maintaining the privacy constraints.

Reviewer comment (Experiments):
I can understand the limitations on the experimental side, however, it would be great to hear from authors regarding how the performance of the centralized variant of the proposed federated graph matching approach (i.e., no federated learning)? Does this approach have much better performance?

Author response:
Thanks for the kind suggestion. In the appendix, we evaluate two versions of UFGM to show the strength of our UFGM method for federated graph matching. UFGM is the federated version with graph data encryption, graphlet feature extraction, model evaluation on the server, model optimization with the trust region on the clients, and Hessian approximation. UFGM-C is the centralized version with raw graph data uploaded to the server, graphlet feature extraction, model evaluation and model optimization with the standard stochastic gradient descent on the server. The experiment results in Table 4 exhibit that the performance of the centralized version, UFGM-C, is close to our federated version, UFGM, over all three datasets. This further validates that our UFGM algorithm can achieve superior performance for the federated graph matching.
In addition, the performance of our UFGM-C method is close to (NextAlign, NetTrans, and SeedGNN) or better than (CPUGA, ASAR-GM, and SIGMA) among six centralized graph matching methods. Notice that NextAlign, NetTrans, and SeedGNN are semi-supervised or supervised methods with 20\% training data.

Reviewer comment (Writing):
The challenges and applications of applying federated training on unsupervised graph matching are not clear.

Author response:
**Weakness 1:** The challenges and applications of applying federated training on unsupervised graph matching are not clear.

**Answer:** Thanks for the thoughtful comment. Since FedAvg was proposed by Google in 2017 [1], federated learning (FL) has been a hot research topic in the field of machine learning.
With the increasing regulation restrictions, user concerns, and commercial competition, the government agencies limit data collection and the users are not willing to share the data.
The original intention of FL proposed by Google is to reduce privacy and security risks by limiting the attack surface to only the device, rather than the device and the cloud [1]. 
Federated graph learning (FGL) has led to state-of-the-art innovations in various applications, such as node classification, graph classification, network embedding, and link prediction. There is still a paucity of techniques of effective federated graph matching (FGM), which is much more difficult to study.

A good real-world application of FGM is that financial crime detection on transaction networks with millions to billions of bank customers and transactions [2-4]. Data exchange among clients and server about sensitive bank customer and transfer data should be limited for privacy risks.
Especially, US and UK governments launched a privacy-enhancing technology (PET) challenge about federated learning for financial crime detection in July 2022 [5,6]. The dataset contains SWIFT transfer data between bank accounts and individual bank account transaction data. Frequent interbank transactions between the same or affiliated entities may be potential money laundering activities.
Another real application is user account linking in different networks, since the user information contain many sensitive information. Directly collecting and uploading the raw user data to the server for centralized graph matching (CGM) gives rise to a serious privacy concern.

[1] Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017.

[2] Towards federated graph learning for collaborative financial crimes detection. CoRR, abs/1909.12946, 2019.

[3] A semi-supervised graph attentive network for financial fraud detection. ICDM 2019.

[4] Federated graph learning - A position paper. CoRR, abs/2105.11099, 2021.

[5] https://research.ibm.com/blog/privacy-preserving-federated-learning-finance

[6] https://www.drivendata.org/competitions/group/nist-federated-learning/

The unsupervised graph matching fails to employ the strength of training data and thus often leads to sub-optimal solutions, compared with supervised graph matching. However, the latter using the pre-matched node pairs as the training data is improper for the FGM scenarios due to privacy risks of direct cross-graph information exchange. 

Thus, we propose to capture nodes' graphlet features to generate pseudo matched node pairs on different graphs across clients as the pseudo training data for leveraging the strength of supervised graph matching and improving the quality of unsupervised graph matching.
However, graphlet enumeration one by one on large graphs is impossible due to expensive cost. We propose to leverage Monte Carlo Markov Chain (MCMC) technique for sampling a small number of graphlets. 

Stochastic gradient descent (SGD) optimization widely used in deep learning fails to work on the clients in the FGM, since each client can access only its own local graph data and thus cannot update local loss based on the pseudo matched node pairs.
We propose a separate trust region algorithm for pseudo supervised FGM while maintaining the privacy constraints.
We separate model optimization from model evaluation in the trust region algorithm: (1) the server aggregates the local model parameter on each client into a global model parameter, runs and evaluates the global parameter, and computes the individual loss and the first-order gradient for each client; (2) each client computes the second-order Hessian and optimizes the local model.

Reviewer comment (Novelty):
The technical advancement of the method is unclear. Traditional graph-matching algorithms with encrypted aggregation on the server side can solve such a problem. Such encryption can be a huge computation and communication cost during training.

Author response:
**Weakness 2:** The technical advancement of the method is unclear. Traditional graph-matching algorithms with encrypted aggregation on the server side can solve such a problem. Such encryption can be a huge computation and communication cost during training.

**Answer:** Thanks for the valuable suggestion. As discussed above, two main contributions of this paper are (1) The graphlet feature extraction can leverage the strength of supervised graph matching with the pseudo training data for improving the matching quality; and (2) The separate trust region for pseudo supervised FGM is helpful to enhance the efficiency while maintaining the privacy constraints.

In the separate trust region optimization, the second-order Hessian computation and the local model optimization are most compute-intensive over large graphs. Thus, we let the server compute the individual loss and the first-order gradient and let the clients compute the second-order Hessians and optimize the local models. This design is helpful to make full use of the computational capability of each client to improve the efficiency, as large-scale graph matching is often time-consuming.

The data encryption and communication is a one-time operation that happens before the training. Thus, its computation and communication cost is relatively trivial, compared with the cost of iterative model training and optimization.

Reviewer comment (Experiments):
The algorithm comparison is unreasonable. There is an unreasonable number of comparison methods. All these federated methods are used for supervised training and should not be compared methods with unsupervised training.

Author response:
**Weakness 3:** The algorithm comparison is unreasonable. There is an unreasonable number of comparison methods. All these federated methods are used for supervised training and should not be compared methods with unsupervised training.

**Answer:** Thanks for the kind suggestion. In our experiments, three federated domain adaption baselines, including DualAdapt, EFDA, and FedKA, are unsupervised learning methods. As discussed above, supervised learning methods usually achieve much better performance than unsupervised learning approaches, since the former employs the strength of training data. In the experiments in the submission, our UFGM method achieves better performance than all supervised or unsupervised federated learning methods in most tests, and even obtains the close or better performance than the supervised centralized graph matching method.

In the revision, Table 6 and Figures 7 and 8 exhibit the quality of our UFGM method with two unsupervised centralized graph matching approaches of GANN-GM (Wang et al., 2023) and REGAL (Heimann et al., 2018), two unsupervised federated domain adaption algorithms of LADD (Shenaj et al., 2023) and FMTDA (Yao et al., 2022), and two unsupervised federated graph learning methods of FedWalk (Pan \& Zhu, 2022) and Lumos (Pan et al., 2023). Similar trends are observed for the comparison among these unsupervised federated earning methods: our UFGM method outperforms these baselines in all experiments, in terms of both final performance and convergence. Notice that REGAL is a matrix factorization-based graph alignment method and thus there are no loss functions in it.

Reviewer comment (Theory):
What is the key takeaway of the theoretical analysis? Can it guide the experiments?

Author response:
**Question 1:** What is the key takeaway of the theoretical analysis? Can it guide the experiments?

**Answer:** Thanks for the comment. Theorems 1 and 2 give an explicit solution of the probability of getting a $k$-subgraph via subgraph expansion and the variance of the estimation of graphlet counts. In addition, the theoretical analysis derives that the estimation variance can be reduced with a smaller normalizing factor. In this case, the estimated graphlet count is close to the actual count, which implies that the graphlet samples and all graphlets share similar distributions.
Theorems 3 and 4 derive the error introduced by the separate trust region due to the Hessian approximation and conduct the convergence analysis of the approximation method.