Personalized Subgraph Federated Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
In real-world scenarios, subgraphs of a larger global graph may be distributedacross multiple devices or institutions, and only locally accessible due to privacy restrictions, although there may be links between them.
Recently proposed subgraph
Federated Learning (FL) methods deal with those missing links across private localsubgraphs while distributively training Graph Neural Networks (GNNs) on them.
However, they have overlooked the inevitable heterogeneity among subgraphs, caused by subgraphs comprising different parts of a global graph.
For example,a subgraph may belong to one of the communities within the larger global graph.
A naive subgraph FL in such a case will collapse incompatible knowledge fromlocal GNN models trained on heterogeneous graph distributions.
To overcomesuch a limitation, we introduce a new subgraph FL problem, personalized subgraph
FL, which focuses on the joint improvement of the interrelated local GNN modelsrather than learning a single global GNN model, and propose a novel framework,
FEDerated Personalized sUBgraph learning (FED-PUB), to tackle it.
A crucialchallenge in personalized subgraph FL is that the server does not know whichsubgraph each client has.
FED-PUB thus utilizes functional embeddings of thelocal GNNs using random graphs as inputs to compute similarities between them,and use them to perform weighted averaging for server-side aggregation.
Further,it learns a personalized sparse mask at each client to select and update only thesubgraph-relevant subset of the aggregated parameters.
We validate FED-PUB forits subgraph FL performance on six datasets, considering both non-overlappingand overlapping subgraphs, on which ours largely outperforms relevant baselines.
Introduction
A graph, which defines the relationships among instances, can model a wide range of structured dataincluding social [7], co-purchasing [23] and collaboration networks [36].
Most of the previous workson graph representation learning focus on a single graph, whose nodes and edges collected frommultiple sources are stored in a central server.
For instance, in a social network platform, every user,with his/her social networks, contributes to creating a giant network consisting of all users and theirconnections.
However, in some practical scenarios, each user/institution collects its own private graph,which is only locally accessible due to privacy restrictions.
For instance, as described in Zhang et al.[45], each hospital may have its own patient interaction network to track their physical contacts orco-diagnosis of a disease, however, such a graph may not be shared with others.
An obvious challengefor such a scenario is how to deal with potentially missing edges between subgraphs [42, 45] that arenot captured by individual data owners, that may carry important information (See Figure 1 (A))
How can we then collaboratively train, without sharing actual data, a neural network with its subgraphsdistributed across multiple participants (i.e., clients) over different devices or institutions?
The moststraightforward way is to perform Federated Learning (FL) with Graph Neural Networks (GNNs).
Inparticular, in such an FL framework, each client will individually train a local GNN on the private
[Figure]
L o c a l A cc u r a c y (
[Figure]
[Figure]
FedAvgFedGNNFedSageFedPerFED
[Figure]
[Figure] [Figure]
C
[Figure]
mm
[Figure] [Figure]
local data, while a central server aggregates the locally updated GNN weights from multiple clientsinto one, and then transmits it back to the clients.
Recent subgraph FL methods work in such amanner [42, 45] while additionally tackling the problem of missing edges between subgraphs.
This isdone as illustrated in Figure 1 (B), where the local subgraph is expanded either by exactly augmentingthe relevant nodes from the other subgraphs at the other clients [42], or by estimating the nodes usingthe node information in the other subgraphs [45].
However, such sharing of node information maycompromise data privacy and can incur high communication costs.
Also, there exists a more important challenge that has been overlooked by the existing subgraph FLmethods.
We observe that they suffer from large performance degeneration (See Figure 1 right), dueto a lack of consideration of the heterogeneity among the subgraphs, which is natural since subgraphscomprise different parts of a global graph.
Notably, there could be multiple communities withina global graph, each of which is formed by a group of densely connected subgraphs with similarcharacteristics (Figure 1 (A)).
For example, some of patient networks from hospitals can be groupedby their specialized sectors according to the disease categories, namely psychiatric or ophthalmology.
Motivated by this challenge, we introduce a novel problem of personalized subgraph FL, whose goalis the joint improvement of interrelated local models trained on the interconnected local subgraphs,for instance, subgraphs belonging to the same community (See Figure 1 (C)), by sharing weightsamong them.
However, tackling personalized subgraph FL is challenging, since we do not knowwhich subgraph each client has, due to their local accessibility.
To resolve this issue, we use functionalembeddings of GNNs on random graphs to obtain similarity scores between two local GNNs, inspiredby a work for neural network search that effectively represents entire neural networks in the vectorspace [17], and then use them to perform weighted averaging of the model weights at the server.
However, the similarity scores only tell how relevant each local model from the other clients is, butnot which of the parameters are relevant.
Thus we further learn and apply personalized sparse maskson the local GNN at each client to obtain only the subnetwork, relevant for the local subgraph.
Werefer to this subgraph FL framework as FEDerated Personalized sUBgraph learning (FED-PUB).
We extensively validate our FED-PUB on six different datasets with varying numbers of clients,under both overlapping and disjoint subgraph FL scenarios.
The experimental results show that ourssignificantly outperforms relevant baselines.
Further analysis shows that our method can discovercommunity structures among subgraphs, and the subgraph-specific masking localizes the knowledgewith respect to subgraphs belonging to each community.
Our main contributions are as follows:
• We introduce a novel problem of personalized subgraph FL, which aims at collaborative improve-
ments of the related local models (e
g. subgraphs belonging to the same community), which has
been relatively overlooked by previous works on graph and subgraph FL.•
We propose a novel framework for personalized subgraph FL, which performs weighted averaging
of the local model parameters based on their functional similarities obtained without accessing
the
data, and learns sparse masks to select only the relevant subnetworks for the given subgraphs.•
We validate our personalized subgraph FL framework on six real-world datasets under two different
settings, demonstrating its effectiveness over existing subgraph FL baselines.
Related Work
Graph Neural Networks Graph representation learning with Graph Neural Networks (GNNs) [10,48, 43, 18, 3], which aims to learn the representations of the nodes, the edges, and the entire graph, isan extensively studied topic.
Most existing GNNs under the message passing scheme [8] iterativelyrepresent a node by aggregating features from its neighboring nodes as well as itself.
For example, Graph Convolutional Network (GCN) [22] approximates the spectral graph convolutions [12],yielding a mean aggregation over neighboring nodes.
Similarly, for each node, GraphSAGE [11]aggregates the features from its neighbors to update the node representation.
Such advances in GNNshave led to successes on node and link prediction tasks [22, 47].
However, they are not directlyapplicable to real-world systems with locally distributed graphs, where graphs from different sourcesare not shared across participants, which gives rise to federated learning approaches to train GNNs.
Federated Learning Federated Learning (FL) [32, 41, 19, 24], aiming to learn a model by aggregating model weights trained on local data, is an essential approach for our distributed subgraph learningproblem.
To mention a few, FedAvg [32] locally trains a model for each client and then transmits thetrained model to a server, while the server aggregates the model weights from local clients and thensends the aggregated model back to them.
However, since the locally collected data from differentclients may largely vary, heterogeneity is a crucial issue.
To tackle this, FedProx [25] proposes theregularization term that minimizes the weight differences between local and global models, whichprevents the model from diverging by overfitting to the local training data.
However, when the localdata is extremely heterogeneous, it is more appropriate to collaboratively train a personalized modelfor each client rather than learning a single global model [2, 30, 26, 46, 6].
FedPer [2] is such apersonalized FL method, which shares only the base layers while having local personalized layersfor each client, to keep the local knowledge.
Unlike the commonly studied image and text data,graph-structured data is defined by connections between instances, and consequently introducesadditional challenges: missing edges and shared nodes between private subgraphs.
Note that, regarding architectures, there is literature [29, 27, 38, 49] that leverages outputs of neural networks forpredicting/minimizing outputs across different client models; however, we use functional outputs ofneural networks to identify interconnected subgraphs, thus ours differs from them methodologically.
Graph Federated Learning Few recent studies propose to use the FL framework to collaborativelytrain GNNs without sharing graph data [13], which can be broadly classified into subgraph- andgraph-level methods.
Graph-level FL methods assume that different clients have completely disjointgraphs (e.g., molecular graphs), and recent works [44, 14] focus on the heterogeneity among non-IIDgraphs (i.e., difference in graph labels across various clients).
In contrast to graph-level FL methodsthat have similar challenges to general FL scenarios, the subgraph-level FL problem we target hasa unique graph-structural challenge, that there exist missing yet probable links between subgraphs,since a subgraph is a part of a larger global graph.
To deal with such a missing link problem amongsubgraphs, existing methods [42, 45] augment the nodes by requesting the node information in theother subgraphs, and then connecting the existing nodes with the augmented ones.
However, thisscheme could compromise data privacy constraints, and also increases communication overheadacross clients.
Unlike existing subgraph FL that focuses on the problem of missing links, our subgraph
FL method tackles the problem with a completely different perspective, focusing on discoveringsubgraph communities [35, 9, 34], which are groups of densely connected subgraphs.
Personalized Subgraph Federated Learning
We provide the general descriptions of Graph Neural Networks (GNNs) and Federated Learning (FL),and then define our novel problem of personalized subgraph FL lying at the intersection of them.
Graph Neural Networks A graph G =
( V , E ) consists of a set of nodes V with n elements and aset of edges E with m elements along with its node feature matrix X ∈ R n × d , where each columnrepresents a d -dimensional feature for each node.
Further, ( u, v ) ∈ E represents an edge from a nodeu to a node v .
Then, given the graph, Graph Neural Networks (GNNs) [8, 10] generally representeach node based on features from its neighbors as well as itself, formally defined as follows:
[Equation]
where H ( l ) v is the feature matrix for node v at l -th layer, N ( v ) denotes a set of adjacent nodes ofnode v : N ( v ) = { u ∈ V | ( u, v ) ∈ E} , AGGREGATE aggregates the features of v ’s neighbors, and
UPDATE updates the node v ’s representation given its previous representation and the aggregatedrepresentations from the neighbors.
H (1) is initialized as input node features X .
Federated Learning The objective of Federated Learning (FL) is to collaboratively train a modelwith local private data.
Let assume that we have K participants with locally collected data that is notaccessible from others: D k = { X i , y i } N k
i
=1 , where X i is a data instance, y i is its corresponding classlabel, and N k is the number of data instances at k -th client.
Then, for decentralized training withlocal data, a popular FL algorithm, FedAvg [32], works as the following three steps:
(Initialization)
At the initial communication round r =
the
central
server
first selects
K clients
that are available for training
and initializes their local model parameters as the global parameter
¯ θ , represented
as
follows: θ (0)
k ← ¯ θ (0) ∀ k , where θ (0) k is the parameters for k -th client.
(Local Updates)
Each active local model performs training on private local data D k to minimize
the task loss L ( D k ; θ (0) k ) , consequently updating the parameters θ (1)
k ← θ (0) k − η ∇L .
(Global Aggregation)
After local training, the server aggregates the locally learned knowledge
with respect to the number of training instances, i.e., ¯ θ (1) ← N k N (cid:80)
Kk =1 θ (1) k with N = (cid:80) k N k ,
and distributes the updated global parameters ¯ θ (1) to the local clients selected at the next round.
This FL algorithm iterates between Step 2 and 3 until reaching the final round R .
Challenges in Subgraph FL
While the above FL works well on image and text data, due to theunique structure of graphs, there exist nontrivial challenges for applying this FL scheme to graphstructured data.
In particular, unlike with an image domain where each instance X i is independentfrom the other images, each node v in a graph is always influenced by its relationships to adjacentnodes N ( v ) .
Moreover, a local graph G i could be a subgraph of a larger global graph G : G i ⊆
G .
Insuch a case, there could be missing edges between local subgraphs in two different clients: ( u, v )with u ∈ V i and v ∈ V j for clients i and j , respectively.
To tackle this missing edge problem, fewexisting subgraph FL methods [42, 45] estimate the nodes from a local subgraph G k
based on thenode information from the subgraphs at other clients G i ∀ i ̸
= k , and then extend the existing nodeswith the estimated ones.
However, this augmentation scheme incurs high communication costs as itrequires sharing node information across clients, which may also violate data privacy constraints [1].
Yet, there exists another issue that makes subgraph FL even more challenging.
Assume that we havea global graph consisting of all the subgraphs.
Then, there exists communities of such subgraphs [35,9, 34], where subgraphs within the same community are more densely connected to each otherthan subgraphs outside the community.
Formally, a global graph G can be decomposed into Tdifferent communities: C i ⊆
G ∀ i = 1 , ..., T , where i -th community C
i
=
( V i , E i ) consists ofdensely connected nodes.
Then, in a subgraph FL problem, each client has a local subgraph G j thatbelongs to at least a single community 1 : C i = (cid:83)
Jj =1 G j .
Note that, based on the theory of networkhomophily [33], such connected subgraphs within the same community have similar properties, whilesubgraphs in two opposite communities are not.
Such distributional heterogeneity across communitiesmay lead a naive FL algorithm to collapse incompatible knowledge across different communities.
Personalized Subgraph FL To prevent the above knowledge collapse issue, we aim to personalizethe subgraph FL algorithm by performing weighted averaging of the local model parameters atthe server, rather than learning a single set of global parameters; thereby capturing the subgraphcommunity structures among interrelated subgraphs.
Formally, the objective of existing subgraph
FL [42, 45, 28] is as follows: min θ (cid:80) G i ⊆G L ( G i ; θ ) .
However, a major drawback of such a schemeis that, since the subgraphs in two different communities with sparse connections are extremelyheterogeneous due to network homophily [33], finding a universal set of parameters (i.e., θ ) that workon all tasks will result in finding a suboptimal parameter set.
To address such limitations of existing subgraph FL, we formulate a novel problem of personalized subgraph FL, formalized as follows:
[Equation]
where θ i is the weight for subgraph G i belonging to community C , and α ij is the coefficient for weight aggregation which we will specify in Section 4.1.
This formulation promotes the collaborative learning across multiple local models that work on the interrelated subgraphs that belong to the same community, by assigning larger weights on them.
Community A
Community B
User 3 Subgraph
User 1 Subgraph User 2 Subgraph
Model
Model
Model1 Random
Graphs, ෩𝑮 𝒇( ෩ 𝑮;𝜽 𝟏 ) 𝒇(෩𝑮;𝜽 𝟐 ) 𝒇(෩𝑮;𝜽 𝟑 ) ෩𝒉෩𝒉෩𝒉 𝟑 High Similarity 𝜽𝜽𝜽⊙ 𝜽𝝁(A) Community Structure (B) Subgraph Similarity Matching in Server 𝜽⊙ 𝜽𝝁𝜽⊙ 𝜽𝝁𝜽
Personalized Mask for Client𝝁 𝟏(𝟎)
𝝁 𝟏(𝟏𝟎) 𝝁 𝟏(𝟐𝟎)
Personalized Mask for Client𝝁 𝟐(𝟎)
𝝁 𝟐(𝟏𝟎) 𝝁 𝟐(𝟐𝟎)
Personalized Mask for Client𝝁 𝟑(𝟎)
𝝁 𝟑(𝟏𝟎)
𝝁 𝟑(𝟐𝟎)
(C) Weight Masking in Client
Federated Personalized Subgraph Learning (FED-PUB) Framework
Our goal of personalized subgraph FL is to jointly improve the local models trained on the interconnected local subgraphs forming the community structures.
To this end, we propose to computesubgraph similarity scores for detecting communities, and to mask subgraph-irrelevant weights.
4.1 Subgraph Similarity Estimation for Detecting Subgraph Community
We aim to reflect the community structure consisting of a group of densely connected subgraphs, bysharing more weights among subgraphs in the same community, as formalized in equation 2.
Due tonetwork homophily where similar instances in the graph are more associated with each other [33], thesubgraphs within the same community should have similar properties.
Therefore, if one can measurethe subgraph similarities, we can group the similar ones into the community.
However, measuring thesimilarity between local subgraphs is challenging since we do not know which subgraph each clienthas due to local accessibility.
How can we then compute subgraph similarities, without accessingthem?
To this end, we aim to approximate the subgraph similarity at local clients using auxiliaryinformation obtained from the local GNN models that work on the subgraphs.123456789101112131415 Clients 123456789101112131415 C li e n t s
Parameter 123456789101112131415 Clients 123456789101112131415 C li e n t s
[Figure]
[Figure]
C li e n t s
[Figure]
Subgraph Similarity Estimation with Model Parameters Formeasuring the similarity between subgraphs at each client, withoutaccessing them, we may use the model parameters as proxies, asfollows: S ( i, j )
= ( θ i · θ j ) / ( ∥ θ i ∥∥ θ j ∥ ), where θ is a flattened parameter into the vector, and S is a similarity measure.
This maysound reasonable since the GNN model trained on the subgraph willembed its knowledge into its parameters.
However, this scheme hasa notable drawback that similarity measured in the high-dimensionalparameter space is not meaningful due to the curse of dimensionality [4], and that the cost of calculating the similarity betweenparameters grows rapidly as the model size increases (See Figure 3).
Subgraph Similarity Estimation with Functional Embedding To tackle the limitations of usingparameter distance, we propose to measure the functional similarity of neural networks by feeding thesame input to every local client and then calculating the similarities using their outputs, inspired by awork for neural network search [17].
The main intuition is that we can consider the transformationdefined with a neural network as a function, and we measure the functional similarity of two networksby the distance of their outputs for the same input.
However, unlike the previous work [17] thattackles image classification, which uses Gaussian noises as inputs, we use random graphs as inputsas we work with GNNs.
Formally, let ˜ G =
(˜ V , ˜ E ) be a random community graph obtained from astochastic block model [15], where subgraphs within the community have more edges between themthan edges across the communities.
Further, ˜ V is randomly initialized from the normal distribution.
Then, the similarity between two functions defined by GNNs at clients i and j is defined as follows:
[Equation]
where ˜ h is the averaged output of all node embeddings for input ˜ G with AVG operation to reduce thedimensionality of the output from n × d to d , for n nodes with d -dimensional node features.
Personalized Weight Aggregation based on Subgraph Similarity With equation 3, the remainingstep is then to share the model weights between models working on similar subgraphs belonging tothe same community.
However, entirely ignoring the model parameters from different communitiesmay result in exploiting only the local objective while ignoring globally useful weights, which mayresult in performance degeneration.
Therefore, we perform weighted averaging of all the local modelsfrom the other clients based on their functional (subgraph) similarities, as follows (Figure 2 (B)):
[Equation]
where α ij is a normalized similarity between clients i and j , and τ is a hyperparameter for scalingthe unnormalized similarity score.
Note that increasing the value of τ (e.g., 10) will result in modelaveraging done almost exclusively among subgraphs detected as belonging to the same community.
This personalized scheme handles two challenges in subgraph FL.
First, in contrast to the globalweight aggregation scheme which easily collapses the knowledge from heterogeneous communitiesinto a single model, our subgraph FL allows the models belonging to different communities to obtainmodel weights that are beneficial for each community.
Also, the missing edges between subgraphsthat have been explicitly handled by previous works [42, 45] could be also implicitly considered byassigning larger weights to models within the same community (See Figure 10).
This also enhancesdata privacy while minimizing the communication costs between probably linked subgraphs.
4.2 Adaptive Weight Masking for Selecting Subgraph-Relevant Parameters
With the previous similarity matching scheme, we can effectively group GNN models that belong tothe same community, thus preventing the collapsing of irrelevant knowledge from other communities.
However, the scalar weighting scheme only considers how much each local model from other clientsis relevant for the subgraph task, but not which parameters are relevant.
Thus we propose a scheme toselect only the relevant parameters from the aggregated model weights transmitted from the server.
Personalized Parameter Masking We perform selective training and updating of the aggregatedparameters by modulating and shifting them, using sparse local masks.
Formally, let µ k be a localmask for a client k .
Then, our local model weight is obtained by modulating the weights from theserver, as follows:
θ k
= ¯ θ k ⊙ µ k , where ⊙ is an element-wise multiplication operation betweenthe globally given weight ¯ θ k and the local mask µ k .
Note that the local mask is a free variableand is not shared across clients.
Also, we initialize µ k as ones, in order to start training with theglobally initialized model parameters without modification.
We then further promote sparsity onthe mask, which brings two key advantages.
First, we can transmit only the partial parameters, thathave not been sparsified at the client to the server rather than sending all parameters, thus reducingthe communication costs.
Moreover, if local masks are sufficiently sparse, the local models can betrained faster, given that zero-skipping operations are supported (Figure 2 (C)).
To take these benefitsin sparsity, we use L 1 regularizer on µ k when performing local optimization, as shown in equation 5.
Preventing Local Divergence with Proximal Term As masks are trained only with limited localdata without parameter sharing, they may be easily overfitted to the training instances in each client.
To alleviate this issue, we adopt the proximal term proposed in Li et al.
[25] that regularizes the locallyupdated models θ k to be closer to the globally given model ¯ θ k , therefore, preventing the model fromextremely drifting to the local training distribution.
To sum up, at k -th client, our objective functionincluding sparsity and proximal terms with L 1 and L 2 losses is denoted as follows:
[Equation]
where L is the conventional cross-entropy loss function, and λ 1 and λ 2 are scaling hyper-parameters.
Experiments
We now experimentally validate our FED-PUB on six different datasets under both the overlappingand disjoint subgraph scenarios with varying client numbers, with node classification tasks.
5.1 Experimental Setups
Datasets Following the setup from Zhang et al.
[45], we construct the distributed subgraphs fromthe benchmark dataset by dividing it into the number of participants, each of which has a subgraphthat is a part of an original graph.
Specifically, we use six datasets: Cora, CiteSeer, Pubmed and
[Table]
[Figure]
ogbn-arxiv for citation graphs [39, 16]; Computer and Photo for product graphs [31, 40].
We thendivide the original graph into multiple subgraphs using the METIS graph partitioning algorithm [20].
Note that, unlike the Louvain algorithm [5] presented in Zhang et al.
[45] that requires to furthermerge partitioned subgraphs into particular numbers of subgraphs since it cannot specify the numberof subsets (i.e., clients for FL), the METIS algorithm can specify the number of subsets, thus makingmore reasonable experimental settings in subgraph
FL (See Section C.2 of the supplementary file).
For the non-overlapping scenario where there are no duplicate nodes between subgraphs, we use theoutput from the METIS as it provides the non-overlapping partitions.
Meanwhile, for the overlappingscenario where nodes are duplicated among subgraphs, we randomly sample the subgraphs multipletimes from the partitioned graph.
For more details, please see Section B of the supplementary file.
Baselines 1) FedAvg [32] and 2) FedProx [25]: The most popular FL baselines.
3) FedPer [2]: Apersonalized FL baseline without sharing personalized layers.
4) FedGNN [42] and 5) FedSage+ [45]:
Subgraph FL baselines which we mainly target.
6) GCFL [44]: A graph FL baseline which learnscompletely disjoint graphs as in clustered FL [37], adopted for subgraph FL. 7)
Local : A baselinewithout sharing weights with other clients.
8) FED-PUB :
Our personalized subgraph FL includingsubgraph similarity matching and weight masking.
See Section B of the supplementary file for details.
Implementation Details We set the GCN [22] with two layers as the base GNN for all models.
We perform federated learning over 100 communication rounds for Cora, CiteSeer and Pubmeddatasets, while 200 rounds for Computer, Photo and arxiv datasets, considering the size of datasets.
The local training epoch is selected in the range of { 1 , 2 , 3 } depending on the dataset size (e.g.,
Computer is three while CiteSeer is one) 2 .
We use the Adam optimizer [21] for model optimization.
We then measure the node classification accuracy on subgraphs at the client-side, and then averagethe performance across clients.
We provide further details in Section B of the supplementary file.
5.2 Experimental Results
Main Results Table 1 shows the node classification performance under the overlapping subgraphscenario, in which our FED-PUB statistically ( p > 0 . 05 ) significantly outperforms all the baselines.
In particular, while FedGNN and FedSage+ are two pioneer works for the subgraph FL problem,they significantly underperform personalized FL methods including ours, especially at the largernumber of clients.
This is even surprising as they share node information between clients for handlingthe missing edge problem, yet we suppose such inferior performance comes from naive averagingof local weights without consideration of community structures.
While personalized FL baselinesincluding FedPer and GCFL show decent performance by alleviating the knowledge collapse betweensubgraphs with local parameters or clustering, they still largely underperform ours as they are notconcerned with the aggregation between similar subgraphs that form a community (i.e., GCFL uses abi-partitioning scheme where it iteratively divides a group of subgraphs within the same community2 We found communication rounds and local epochs are important factors to prevent overfitting of all models.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
into two disjoint sets).
We then further conduct the experiments on the disjoint subgraph scenarios(non-overlapping scenario), where nodes are not overlapped between subgraphs, which makes thesubgraph FL problem more heterogeneous.
As shown in Table 2, FED-PUB consistently outperformsall existing baselines in such a challenging scenario, demonstrating the efficacy of ours.
Fast Local Convergence As shown in Figure 4 and 5, our FED-PUB converges rapidly, comparedagainst baselines including personalized FL models.
We conjecture that this is because, not only oursaccurately identifies subgraphs forming the community and then shares weights largely across themfor promoting the joint improvement of them, but also masking subgraph-irrelevant weights receivedfrom the server for localization to local subgraphs, demonstrated in the next two paragraphs.
Accurate Community Detection We aim to show whether FED-PUB accurately groups subgraphscomprising a community during weight aggregation.
If two different subgraphs have many missingedges or have similar label distributions, we usually regard those two as within the same community [35, 9, 34].
Thereby, as shown in Figure 6 (a) and (b), there are four different communities by theinterval of five, and the last two communities further comprise a larger community.
Then, as shownin Figure 6 (c) and (d), FED-PUB detects obvious four communities at the first few rounds, and thencaptures the larger yet somewhat less-obvious community consisting of two smaller communities.
Ablation Study To analyze the contribution of each component, we conduct the ablation studies.
As shown in Figure 7, we observe that each of our subgraph similarity matching and weight maskingsignificantly improves the performances from the naive FedAvg, while the performance is muchimproved when using both together.
However, the benefit from each component is different acrossoverlapping and non-overlapping scenarios.
In particular, in the former scenario where a group ofhighly overlapped subgraphs usually comprise a community, similarity matching for communitydetection is more beneficial since capturing the community would promote the joint improvement ofsubgraphs belonging to the same community.
However, in the non-overlapping scenarios, subgraphswithin the same community become lesser similar, thus selectively using the aggregated modelweights from the server with personalized weight masks improves the performance a lot.
Communication Efficiency
Another notable advantage of using the sparse masks is that we canreduce the communication costs at every FL round, as well as the model size for faster training, whichwe demonstrate in Table 8.
In particular, Table 8 shows that existing subgraph FL methods requiremore than two times larger communications costs, measured by adding both the client-to-server andserver-to-client costs, compared against the naive FedAvg, since they require to transfer additionalnode information between clients for estimating the probable nodes on the subgraphs.
Contrarily, our
FED-PUB has significantly lower communication costs and lower model sizes by using the sparsemasks on the model weights: transmitting and training with only the partial parameters not sparsifiedat the client.
Further, as shown in ours variants in Table 8, we can manage the trade-off between themodel sparsity and the performance by controlling the hyperparameter for sparsity regularization, λ 1 .
[Figure]
[Figure] [Figure]
[Figure] [Figure]
Varying Local Epochs As shown in Figure 9, when we increase the number of communicationrounds and the local steps, the model diverges to the local subgraphs (i
[Figure]
e., overfitting), due to thesmall number of training instances and the direct connection between training and test nodes: struggleto generalize to the test instances.
However, our model with the proximal term in equation 5 alleviatesthis issue, therefore, maintaining the highest local performance.
Notably, the performance with fivelocal epochs is inferior to the performance of one epoch, which indicates that increasing the localepochs does not always bring advantages and properly tuning them is important for subgraph FL.
Handling Missing Edges To measure whether FED-PUB can handle the missing edge problem:information is not shared between two neighboring subgraphs due to the missing edges, we usethe local model trained on the local subgraph for evaluating the performance on its neighboringsubgraph, in which the local subgraph has the most missing edges to its neighboring subgraph.
Specifically, in Figure 10, (Neighbor) denotes the subgraph performance evaluated by its neighbormodel, while (Local) denotes the subgraph performance from its own local model.
Then, the highperformance on (Neighbor) measure means two associated subgraphs share meaningful knowledgewithout having explicit edges between them, thereby solving the missing edge problem.
Note that,existing subgraph FL explicitly augments the nodes and edges for capturing the potential informationflow over the missing edges between subgraphs, while ours implicitly shares weights a lot acrosssimilar subgraphs within the same community.
Figure 10 shows that ours achieves the significantlysuperior performance on the neighboring subgraph problem against subgraph FL baselines, whichconfirms that ours has an advantage on the missing edge problem by meaningfully sharing knowledgebetween two subgraphs having potentially missing edges, without explicitly estimating them.
Conclusion
We introduced a novel problem of personalized subgraph FL, which focuses on the joint improvementof local GNNs working on interrelated subgraphs (e.g. subgraphs belonging to the same community),by selectively utilizing knowledge from other models.
The proposed personalized subgraph FL ishighly challenging due to 1) difficulty of computing similarities between local subgraphs that areonly locally accessible, and 2) knowledge collapse among local models that work on heterogeneoussubgraphs during weight aggregation.
To this end, we proposed a novel personalized subgraph FLframework, referred to as FEDerated Personalized sUBgraph learning (FED-PUB), which computesthe similarities across subgraphs using functional embeddings of their local GNNs on random graphs,and uses them to perform a weighted average of the local models for each client.
Further, we mask outglobally given weights to focus on only the relevant subnetwork for each client (or community).
Weextensively validated our framework on multiple benchmark datasets with both overlapping and nonoverlapping subgraphs, on which our FED-PUB significantly outperforms relevant baselines.
Furtheranalyses show the effectiveness of the subgraph similarity matching for detecting the communitystructures, as well as the weight masking for tackling the subgraph heterogeneity.
We provide thelimitations and potential societal impacts of our work in Section D of the supplementary file.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss them in Section D ofthe supplementary file.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discussthem in Section D of the supplementary file.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide thecode, data, and instructions in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] We specify all the training details in Section B of the supplementary
file.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] We report the main results with mean and standarddeviations, with multiple runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We include the computational costsand resources in Section B of the supplementary file.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]