Original: Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation
Revised: Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Federated Learning (FL) on knowledge graphs(KGs) has yet to be as well studied as otherdomains, such as computer vision and naturallanguage processing.
Revised: Federated Learning (FL) on knowledge graphs (KGs) has yet to be as well studied as other domains, such as computer vision and natural language processing.

Original: A recent study FedE firstproposes an FL framework that shares entityembeddings of KGs across all clients.
Revised: A recent study FedE first proposes an FL framework that shares entity embeddings of KGs across all clients.

Original: However, compared with model sharing in vanilla
Revised: However, compared with model sharing in vanilla

Original: FL, entity embedding sharing from FedE wouldincur severe privacy leakage.
Revised: FL, entity embedding sharing from FedE would incur severe privacy leakage.

Original: Specifically, theknown entity embedding can be used to inferwhether a specific relation between two entities exists in a private client.
Revised: Specifically, the known entity embedding can be used to infer whether a specific relation between two entities exists in a private client.

Original: In this paper,we first develop a novel attack that aims to recover the original data based on embeddinginformation, which is further used to evaluatethe vulnerabilities of FedE. Furthermore, wepropose a Fed erated learning paradigm withprivacy-preserving R elation embedding aggregation (F ED R) to tackle the privacy issue in
Revised: In this paper, we first develop a novel attack that aims to recover the original data based on embedding information, which is further used to evaluate the vulnerabilities of FedE. Furthermore, we propose a Fed erated learning paradigm with privacy-preserving R elation embedding aggregation (F ED R) to tackle the privacy issue in

Original: FedE.
Revised: FedE.

Original: Compared to entity embedding sharing,relation embedding sharing policy can significantly reduce the communication cost due toits smaller size of queries.
Revised: Compared to entity embedding sharing, relation embedding sharing policy can significantly reduce the communication cost due to its smaller size of queries.

Original: We conduct extensive experiments to evaluate F ED R with fivedifferent embedding learning models and threebenchmark KG datasets.
Revised: We conduct extensive experiments to evaluate F ED R with five different embedding learning models and three benchmark KG datasets.

Original: Compared to FedE,
Revised: Compared to FedE,

Original: F ED R achieves similar utility and significant(nearly 2 × ) improvements in both privacy andefficiency on link prediction task.
Revised: F ED R achieves similar utility and significant (nearly 2 × ) improvements in both privacy and efficiency on link prediction task.

Original:  Introduction
Revised:  Introduction

Original: Knowledge graphs (KGs) are critical data structures to represent human knowledge, and serveas resources for various real-world applications,such as recommendation (Gong et al., 2021), question answering (Liu et al., 2018), disease diagnosis (Chai, 2020), etc.
Revised: Knowledge graphs (KGs) are critical data structures to represent human knowledge, and serve as resources for various real-world applications, such as recommendation (Gong et al., 2021), question answering (Liu et al., 2018), disease diagnosis (Chai, 2020), etc.

Original: However, most KGs are usually incomplete and naturally distributed to different clients.
Revised: However, most KGs are usually incomplete and naturally distributed to different clients.

Original: Despite each client can explore themissing links with their own KGs by knowledgegraph embedding (KGE) models (Lin et al., 2015),exchanging knowledge with others can further enhance completion performance because the over-
Revised: Despite each client can explore the missing links with their own KGs by knowledge graph embedding (KGE) models (Lin et al., 2015),

Original: [Figure] lapping elements are usually involved in different [Figure] KGs (Chen et al., 2021; Peng et al., 2021).
Revised: Codes: https://github.com/taokz/FedR

Original: [Figure]
Revised: [Figure]

Original: Federated Learning (FL)(McMahan et al., 2017) [Figure] allows different clients to collaboratively learn a
Revised: Aggregator

Original: [Figure]
Revised: [Figure]

Original: global model without sharing their local data. The
Revised: FedR Aggregator

Original: [Figure]
Revised: [Figure]

Original: first FL framework for KG – FedE is recently pro- [Figure] posed, which jointly learns the global-view entity [Figure]
Revised: Since in F ED R, there would be infinite embedding pairs of head and tail given a relation embedding the ence attack would fail.

Original: embeddings from different KGs while the relation [Figure] embeddings can be updated locally. However, at
Revised: exchanging knowledge with others can further enhance completion performance because the overlapping elements are usually involved in different

Original: [Figure] the very beginning in FedE, the server will col- [Figure]
Revised: KGs (Chen et al., 2021; Peng et al., 2021).

Original: lect the entity sets of every client for entity align- [Figure] ment (Chen et al., 2021), which will lead to unin- [Figure]
Revised: Federated Learning (FL) allows different clients to collaboratively learn a global model without sharing their local data (McMahan et al., 2017).

Original: tentional privacy leakage.

Original: [Figure] Data leakage problem in FedE. Since the server [Figure] maintains a complete table of entity embeddings 
Revised: The first FL framework for KG – FedE is recently proposed, which learns the global-view and shared entity embeddings without collecting entire local

Original: [Figure] with the user IDs, it could easily (1) identify client’s [Figure]
Revised: KGs from different clients to a center server.

Original: users and (2) infer the relation embeddings by using [Figure]
Revised: However, at the very beginning in FedE, the server will collect the entity sets of every client for entity alignment (Chen et al., 2021), which will lead to unintentional privacy leakage.

Original:  the scoring function in KGE models, which could[Figure] be defined as f ( h, r, t ) ≤ 0 for all triples ( h, r, t ) .
Revised: Data leakage problem in FedE. Since the server maintains a complete table of entity embeddings with the user IDs, it could easily (1) identify client’s users and (2) infer the relation embeddings by using the scoring function in KGE models, which could be defined as f ( h, r, t ) ≤ 0 for all triples ( h, r, t ) .

Original: The details of scoring function are described in
Revised: The details of scoring function are described in

Original: Table 7.
Revised: Table 7.

Original: As shown in Figure 1, for the server in
Revised: As shown in Figure 1, for the server in

Original: FedE, the relation embeddings on each client can be obtained by calculating r ′ =
Revised: FedE, the relation embeddings on each client can be obtained by calculating r ′ =

Original: arg max r f ( h, r, t ) .
Revised: arg max r f ( h, r, t ) .

Original: Since the aligned local head and tail entity embeddings in different clients are identical with each other after aggregation.
Revised: Since the aligned local head and tail entity embeddings in different clients are identical with each other after aggregation.

Original: Once the server can access the name of entities and relations by colluding with a single client, the local data of any target clientincluding entities, relations, and the correspondingembeddings will be exposed.
Revised: Once the server can access the name of entities and relations by colluding with a single client, the local data of any target client including entities, relations, and the corresponding embeddings will be exposed.

Original: To tackle the privacy issue in FedE, we propose
Revised: To tackle the privacy issue in FedE, we propose

Original: F ED R based on relation embedding aggregationas illustrated in Figure 1.
Revised: F ED R based on relation embedding aggregation as illustrated in Figure 1.

Original: In F ED R, it would beimpossible for the server to infer local entity embeddings given only relation embeddings.
Revised: In F ED R, it would be impossible for the server to infer local entity embeddings given only relation embeddings.

Original: For example, we can not calculate t ′ =
Revised: For example, we can not calculate [Equation]

Original: arg max t f ( h, r, t )merely based on known r but without h .
Revised: merely based on known r but without h .

Original: Besides,the number of entities is usually much greaterthan the number of relations in real-world graphdatabases, so sharing relation embedding is muchmore communication-efficient.
Revised: Besides, the number of entities is usually much greater than the number of relations in real-world graph databases, so sharing relation embedding is much more communication-efficient.

Original: We summarize the following contributions ofour work.
Revised: We summarize the following contributions of our work.

Original: We present a KG reconstruction attack method and reveal that FedE suffers a potentialprivacy leakage from an honest-but-curious serverand its colluded clients.
Revised: We present a KG reconstruction attack method and reveal that FedE suffers a potential privacy leakage due to a malicious server and its colluded clients.

Original: We propose F ED R, anefficient and privacy-preserving FL framework on
Revised: We propose F ED R, an efficient and privacy-preserving FL framework on

Original: KGs.
Revised: KGs.

Original: Experimental results on three benchmarkdatasets demonstrate that F ED R has the competitive performance compared with FedE, but gainsnearly 2 × improvements in terms of privacy protection and communication efficiency.
Revised: Experimental results on three benchmark datasets demonstrate that F ED R has the competitive performance compared with FedE, but gains nearly 2 × improvements in terms of privacy protection and communication efficiency.

Original:  Methodology 
Revised:  Methodology 2. Knowledge Graph Reconstruction

Original: 2. Knowledge Graph Reconstruction Attack

Original: The purpose of this attack is to recover originalentities and relations in a KG given traitor’s information including parital or all triples and the corresponding embeddings, namely element-embeddingpairs.
Revised: The purpose of this attack is to recover original entities and relations in a KG given traitor’s information including parital or all triples and the corresponding embeddings, namely element-embedding pairs.

Original: We summarize the method into 4 steps:
Revised: We summarize the method into 4 steps:

Original: (1) The server colludes with one client C1 to obtain its element-embedding pairs ⟨ ( E, e ) , ( R, r ) ⟩ .
Revised: (1) The server colludes with one client C1 to obtain its element-embedding pairs ⟨ ( E, e ) , ( R, r ) ⟩ .

Original: (2) Infer the target client’s element embedding such as relation embedding by calculating r ′ =
Revised: (2) Infer the target client’s element embedding such as relation embedding by calculating r ′ =

Original: arg max r [Equation]
Revised: arg max r f ( h, r, t )

Original: , r, t ) where h, r ∈ e .(3)
Revised: where h, r ∈ e .

Original:  Measure the discrepancy between the inferredelement embedding such as relation embedding r ′and all known r with cosine similarity.(4)
Revised: (3) Measure the discrepancy between the inferred element embedding such as relation embedding r ′ and all known r with cosine similarity.

Original:  Infer the relation R ′ as R , E ′ as E with corresponding largest similarity scores.
Revised: (4) Infer the relation R ′ as R , E ′ as E with corresponding largest similarity scores.

Original: The whole attack progress in different cases areincluded in Appendix C.
Revised: The whole attack progress in different cases are included in Appendix B. Note that in the step (1) mentioned above, the server could also collude [Table]

Original: Privacy leakage quantization in FedE.
Revised: Privacy leakage quantization in FedE.

Original: We definetwo metrics: Triple Reconstruction Rate (TRR) and
Revised: We define two metrics: Triple Reconstruction Rate (TRR) and

Original: Entity Reconstruction Rate (ERR).
Revised: Entity Reconstruction Rate (ERR).

Original: TRR measuresthe ratio of correctly reconstructed triples by inferring relations between two entities as described[Table] above.
Revised: TRR measures the ratio of correctly reconstructed triples by inferring relations between two entities as described above.

Original: ERR measures the ratio of entities that the[Table] server can reveal their names to the whole num-
Revised: ERR measures the ratio of entities that the server can reveal their names to the whole number of entities.

Original: [Table]

Original: ber of entities. We let the server owns 30%, 50%,[Table] 100% trained element-embedding pairs from C1, the traitor, to reconstruct entities and triples of others.
Revised: We let the server owns 30%, 50%, 100% trained element-embedding pairs from C1, the traitor, to reconstruct entities and triples of others.

Original: In FedE, ERR simply reflects the portion of entities that C1 shares with the server.
Revised: In FedE, ERR simply reflects the portion of entities that C1 shares with the server.

Original: The results[Table] of privacy leakage on FB15k-237 (Toutanova et al., 2015) over three clients are summarized in Table 1.
Revised: The results of privacy leakage on FB15k-237 (Toutanova et al., 2015) over three clients are summarized in Table 1.

Original: LR in tables denotes information (entities, the corresponding relations and relation embeddings) leakage ratio from C1.
Revised: LR in tables denotes information (entities, the corresponding relations and relation embeddings) leakage ratio from C1.

Original: It is clear that the server only needs to collude with one client to obtain most of the information of KGs on other clients.
Revised: It is clear that the server only needs to collude with one client to obtain most of the information of KGs on other clients.

Original: In a word, FedE is not privacy-preserving.
Revised: In a word, FedE is not privacy-preserving.

Original: 2. F ED R
Revised: 2. F ED R

Original: Compared to single-silo learning, F ED R and FedE learn better representations by taking advantage of the complementary capabilities from cross-clients information.
Revised: Compared to single-silo learning, F ED R and FedE learn better representations by taking advantage of the complementary capabilities from cross-clients information.

Original: To guarantee the data privacy in the
Revised: To protect the data privacy in FL on

Original: FedE, F ED R adopts twomain strategies: (1) Before aggregation works, the server acquires all IDs of the unique relations from local clients and maintains a relation table via Private Set Union (PSU), which computes the union of relations, without revealing anything else, for relation alignmentbefore aggregation (Kolesnikov et al., 2019).
Revised: KGs, F ED R adopts two strategies: (1) Before aggregation works, the server acquires all IDs of the unique relations from local clients and maintains a relation table via Private Set Union (PSU), which computes the union of relations, without revealing anything else, for relation alignment (Kolesnikov et al., 2019).

Original: Therefore, although the server still maintains the relation table, the server does not know the relations each client holds.
Revised: Therefore, although the server still maintains the relation table, the server does not know the relations each client holds.

Original: (2) Unlike sharing entity embeddings, under the framework of F ED R, each client first trains its own entity and relation embeddings locally, and only sends relation embeddings to the server.
Revised: (2) Different from sharing entity embeddings, in F ED R, each client first trains its own entity and relation embeddings locally, and only sends relation embeddings to the server.

Original: The server will aggregate the aligned relation embeddings and dispense them to clients for further local updates.
Revised: The server will aggregate the aligned relation embeddings and dispense them to clients for further local updates.

Original: The client-side embedding training depends on the type of local KGE models such as translation distance models and semantic matching models (Sun et al., 2020).
Revised: The client-side embedding training depends on the type of local KGE models such as translation distance models and se- mantic matching models (Sun et al., 2020).

Original: More details are described in Appendix D.
Revised: More details are described in Appendix A.

Original: Privacy Enhancement.
Revised: Privacy Enhancement.

Original: Although the relation privacy could be achieved by PSU, the server still can roughly infer the relation by comparing the uploaded relation embedding with the one stored [Table] [Table] [Table]
Revised: Although the relation privacy could be achieved by PSU, the server still can roughly infer the relation by comparing the uploaded relation embedding with the one stored in the relation table.

Original: [Table]

Original: [Table] [Table] in the relation table. Therefore, to further guarantee no raw data leakage, Secure Aggregation(Bonawitz et al., 2017) is exploited to protect theprivacy of any individual relation embeddings.
Revised: Therefore, to further guarantee no raw data leakage, Secure Aggregation (Bonawitz et al., 2017) is exploited to protect the privacy of any individual relation embeddings.

Original: Thefundamental idea behind it is to mask the uploadedembeddings such that the server cannot obtain theactual ones from each client.
Revised: The fundamental idea behind it is to mask the uploaded embeddings such that the server cannot obtain the actual ones from each client.

Original: However, the sumof masks can be canceled out, so we still have thecorrect aggregation results.
Revised: However, the sum of masks can be canceled out, so we still have the correct aggregation results.

Original:  Experiments
Revised:  Experiments

Original: We carry out several experiments to explore
Revised: We carry out several experiments to explore

Original: F ED R’s performance in link prediction, in whichthe tail t is predicted given head h and relation r .
Revised: F ED R’s performance in link prediction, in which the tail t is predicted given head h and relation r .

Original: Datasets.
Revised: Datasets.

Original: We evaluate our framework through experiments on three public datasets, FB15k-237,
Revised: We evaluate our framework through experiments on three public datasets, FB15k-237,

Original: WN18RR (Dettmers et al., 2018) and a diseasedatabase – DDB14 (Wang et al., 2021).
Revised: WN18RR (Dettmers et al., 2018) and a disease database – DDB14 (Wang et al., 2021).

Original: To buildfederated datasets, we randomly split triples to eachclient without replacement, then divide the localtriples into the train, valid, and test sets with a ratioof 80/10/10.
Revised: To build federated datasets, we randomly split triples to each client without replacement, then divide the local triples into the train, valid, and test sets with a ratio of 80/10/10.

Original: The statistics of datasets after split isdescribed in Table 3.
Revised: The statistics of datasets after split is described in Table 2.

Original: KGE Algorithms.
Revised: KGE Algorithms.

Original: Four commonly-used KGE al- [Table] gorithms – TransE (Bordes et al., 2013), RotatE(Sun et al., 2019), DisMult (Yang et al., 2014) and
Revised: Four commonly-used KGE algorithms – TransE (Bordes et al., 2013), RotatE (Sun et al., 2019), DisMult (Yang et al., 2014) and

Original: ComplEx (Trouillon et al., 2016) are utilized inthe paper.
Revised: ComplEx (Trouillon et al., 2016) are utilized in the paper.

Original: We also implement federated NoGE(Nguyen et al., 2022), a GNN-based algorithm.
Revised: We also implement federated NoGE (Nguyen et al., 2022), a GNN-based algorithm.

Revised: Note that, although random split makes data homogeneous among all the clients to some extent, it ensures fair comparison between FedE and F ED R.

Revised: Otherwise, if we build subgraphs in terms of relation types using non-iid partition, FedE will definitely outperform F ED R because of less overlapping relations among clients.

Original: 3. Effectiveness Analysis
Revised: 3. Effectiveness Analysis

Original: The commonly-used metric for link prediction,mean reciprocal rank (MRR), is exploited to evaluate F ED R’s performance.
Revised: The commonly-used metric for link prediction, mean reciprocal rank (MRR), is exploited to evaluate F ED R’s performance.

Original: We take FedE and
Revised: We take FedE and

Original: Local , where embeddings are trained only on[Table] each client’s local KG, as the baselines.
Revised: Local , where embeddings are trained only on each client’s local KG, as the baselines.

Original: Table 2 shows the link prediction results under settings of different number of clients C .
Revised: Table 3 shows the link prediction results under settings of different number of clients C .

Original: We observe that
Revised: We observe that

Original:  F ED R comprehensively surpasses Local under all settings of the number of clients, which indicates that relation aggregation makes sense for learning better embeddings in FL.
Revised: [Table] F ED R comprehensively surpasses Local under all settings of the number of clients, which indicates that relation aggregation makes sense for learning better embeddings in FL.

Original: Take NoGE as an example, F ED R gains 29 .
Revised: Take NoGE as an example, F ED R gains 29 .

Original: 64 ± 0 .
Revised: 64 ± 0 .

Original: 037% , 22 . 13 ± 0 .
Revised: 037% , 22 . 13 ± 0 .

Original: 065% , and 11 .
Revised: 065% , and 11 .

Original: 84 ± 0 .
Revised: 84 ± 0 .

Original: 051% average improvement in MRR on three dataset.
Revised: 051% average improvement in MRR on three dataset.

Original: Compared with FedE, F ED R usually has the better or similar results with the
Revised: Compared with FedE, F ED R usually has the better or similar results with the

Original: KGE models of DistMult and its extensive version
Revised: KGE models of DistMult and its extensive version

Original: ComplEx on all datasets.
Revised: ComplEx on all datasets.

Original: We also observe that
Revised: We also observe that

Original: FedE fails to beat Local setting and even performs catastrophically with these two KGE models on both DDB14 and WN18RR.
Revised: FedE fails to beat Local setting and even performs catastrophically with these two KGE models on both DDB14 and WN18RR.

Original: Although FedE performs better than F ED R with TranE and RotatE, the absolute performance reductions between FedE and F ED R are mostly (13/16 = 81%) within 0.
Revised: Although FedE performs better than F ED R with TranE and RotatE, the absolute performance reductions between FedE and F ED R are mostly (13/16 = 81%) within 0.

Original: in MRR on both DDB14 and FB15k-237, which illustrates that F ED R is still effective.
Revised: in MRR on both DDB14 and FB15k-237, which illustrates that F ED R is still effective.

Original: The theoretical explanations behind these results w.r.t data

Original: GEE

Original: LEE

Original: GRE

Original: LRE

Original: FedE (cid:34) (cid:34) (cid:37) (cid:37)

Original: FedR (cid:37) (cid:34) (cid:34) (cid:34) [Table] heterogeneity, and characteristics of FL and KGEmodels need further studies.
Revised: The theoretical explanations behind these results w.r.t data heterogeneity, and characteristics of FL and KGE models need further studies.

Original: 3. Privacy Leakage Analysis
Revised: 3. Privacy Leakage Analysis

Original: Compared with entity aggregation, additionalknowledge is required to make privacy leakagein F ED R because it is almost impossible to inferany entity or triple from relation embeddings only.
Revised: Compared with entity aggregation, additional knowledge is required to make privacy leakage in F ED R because it is almost impossible to infer any entity or triple from relation embeddings only.

Original: Therefore, we assume the server can access partor all entity embeddings from clients.
Revised: Therefore, we assume the server can access part or all entity embeddings from clients.

Original: The information leakage ratio of local entity embeddings(LLR) set as 30% , 50% , 100% respectively in theexperiment.
Revised: The information leakage ratio of local entity embeddings (LLR) set as 30% , 50% , 100% respectively in the experiment.

Original: For simplicity, we let the server holdsall entity embeddings from C1 in Section 2.1, i.e.,
Revised: For simplicity, we let the server holds all entity embeddings from C1 in Section 2.1, i.e.,

Original: LLR=100%.
Revised: LLR=100%.

Original: Besides, for fair comparison, any encryption techniques mentioned in Section 2 are nottaken into account in this privacy analysis.
Revised: Besides, for fair comparison, any encryption techniques mentioned in Section 2 are not taken into account in this privacy analysis.

Original: Figure 2 presents the privacy leakage quantization in F ED R over three clients on FB15k237.
Revised: Figure 2 presents the privacy leakage quantiza- [Table] [Table] [Table]

Original: Notethat the scale of the Y-axis is in 10 − 4 .
Revised: represents “Global”, “L” represents “Local”. “EE” and “RE” represent entity and relation embeddings, respectively. tion in F ED R over three clients on FB15k237, in which the scale of the Y-axis is in 10 − 4 .

Original: The resultsdemonstrate that relation aggregation can guarantee both entity-level and graph-level privacy evenif providing additional local entity embeddings.
Revised: The results demonstrate that relation aggregation can guarantee both entity-level and graph-level privacy even if providing additional local entity embeddings.

Original: Inaddition, we summarize the difference of adversaryknowledge in FedE and F ED R in Table 4.
Revised: In addition, we summarize the difference of adversary knowledge in FedE and F ED R in Table 4.

Original: We observe that despite the relation embedding can beexploited directly in F ED R instead of inference,the privacy leakage rates in F ED R are still substantially lower than the ones in FedE.
Revised: We observe that despite the relation embedding can be exploited directly in F ED R instead of inference, the privacy leakage rates in F ED R are still substantially lower than the ones in FedE.

Original: For example,according to Table 1, for C2, F ED R obtains reduction (9690 − 145 .
Revised: For example, according to Table 1, for C2, F ED R obtains reduction (9690 − 145 .

Original: 43 = 9544 .
Revised: 43 = 9544 .

Original: 57) × 10 − 4 and(7378 − 35 .
Revised: 57) × 10 − 4 and (7378 − 35 .

Original: 04 =
Revised: 04 =

Original: 96) × 10 − 4 in ERR and
Revised: 96) × 10 − 4 in ERR and

Original: TRR (which is about 98.50% and 99.52% relativereduction) on FB15k237, respectively.
Revised: TRR (which is about 98.50% and 99.52% relative reduction) on FB15k237, respectively.

Original: To explainthe result intuitively, in F ED R, local entity embeddings of the same entity in each client usually vary.
Revised: To explain the result intuitively, in F ED R, local entity embeddings of the same entity in each client usually vary.

Original: Therefore, calculating the similarity between embeddings to reconstruct KGs does not work.
Revised: Therefore, calculating the similarity between embeddings to reconstruct KGs does not work.

Original: In this section, the product of data sizes and communication rounds is calculated to measure thecommunication cost.
Revised: Results of privacy leakage quantization (LLR = 100%) on other datasets are shown in Table 5. We observe that both ERRs and TRRs on two datasets are very low, which is consistent with the results on FB15k-237.

Original: Considering the performancedifference between F ED R  and FedE, for fair com-
Revised: Therefore, F ED R can successfully defense against KG reconstruction attack and gain

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Table]

Original: parison of communication efficiency, we count the communication rounds when the model reaches a pre-defined MRR target on the validation dataset, specifically, we set two different MRR targets: 0.
Revised: small privacy leakage rates. 3. Communication Efficiency Analysis In this section, the product of data sizes and communication rounds is calculated to measure the communication cost. Considering the performance difference between F ED R and FedE, for fair comparison of communication efficiency, we count the communication rounds when the model reaches a pre-defined MRR target on the validation dataset, specifically, we set two different MRR targets: 0.

Original: and 0.4.
Revised: and 0.4.

Original: Since all models perform well on DDB14, we take the setting with C = 5 on DDB14 as an example in this section.
Revised: Since all models perform well on DDB14, we take the setting with C = 5 on DDB14 as an example in this section.

Original: The required communication rounds for each model are depicted in Figure 3.
Revised: The required communication rounds for each model are depicted in Figure 3.

Original: We observe that F ED R reaches the target with much less communication rounds compared with FedE. For instance, F ED R-DistMult reaches the target MRR = 0.4 within 10 communication rounds while FedE uses 45 rounds.{+[Figure] C o mm un i c a t i o n R o un d s       5  4  
+}
Revised: We observe that F ED R reaches the target with much less communication rounds compared with FedE. For instance, F ED R-DistMult reaches {+[Figure] C o mm un i c a t i o n R o un d s       5  4  
+}

Revised: [Figure]

Revised: DistMult the target MRR = 0.4 within 10 communication rounds while FedE uses 45 rounds.

Original: Also, according to statistics of federated datasets in Table 3, the average of the number of unique entities in
Revised: Also, according to statistics of federated datasets in Table 2, the average of the number of unique entities in

Original: FedE and unique relations in F ED R are 4462.
Revised: FedE and unique relations in F ED R are 4462.

Original: and 12.8, respectively.
Revised: and 12.8, respectively.

Original: We ignore the embedding size and just use the number of entities/relations to reflect data size.
Revised: We ignore the embedding size and just use the number of entities/relations to reflect data size.

Original: By using relation aggregation, 99 . 89 ± 0 .
Revised: By using relation aggregation, 99 . 89 ± 0 .

Original: 029% of communication cost is reduced in average for all clients when the target MRR is 0.2, while 99 . 90 ± 0 .
Revised: 029% of communication cost is reduced in average for all clients when the target MRR is 0.2, while 99 . 90 ± 0 .

Original: 042% of communication cost is reduced in average when the target MRR is 0.4.
Revised: 042% of communication cost is reduced in average when the target MRR is 0.4.

Original: These results demonstrate that our proposed framework is much more communication-efficient.
Revised: These results demonstrate that our proposed framework is much more communication-efficient.

Original:  Conclusion
Revised:  Discussion

Revised: One potential concern will be discussed: can differential privacy (DP) used against KG reconstruction attack?

Revised: To answer this question, we employ the vanilla local Laplace DP to mask the uploaded entity embeddings in FedE, where the sensitivity is set as 2 because the value of components in an embedding ranges from -1 to 1.

Revised: In this experiment, we evaluate FedE-DP on

Revised: DDB14 dataset with TransE KGE model while we set LR = 100%.

Revised: The experimental results of C2 w.r.t

Revised: MRR and TRR based on different privacy budgets ϵ are shown in Table 6.

Revised: We can see that the TRRs drop from 0.5091 to approximately 0.04 when applying DP.

Revised: However, although DP can defend the

Revised: KG reconstruction, it also degrades the model performance on the link prediction task, where the

Revised: MRR drops from 0.5954 to less than 0.1.

Revised: In this case, we can conclude that vanilla local Laplace

Revised: DP is not a effective solution for reconstruction attack in FedE.

Revised: Besides, there are some interesting but unsolved problems left in our work, which provide future research opportunities: 1)

Original: In this paper, we propose F ED R – an FL framework on KGs with relation embedding aggregation.
Revised: What’s the explicit or implicit relations among data heterogeneity, aggre- [Table] gation stretegies, and KGE methods? 2)

Original: Experimental results show that F ED R outperforms
Revised: Could the success rate of the KG reconstruction attack transfer to the privacy level of any formal privacy guarantee such as differential privacy?  Conclusion In this paper, we conduct the first empirical quantization of privacy leakage to federated learning on knowledge graphs, which reveals that recent work FedE is susceptible to reconstruction attack based on shared element-embedding pairs when there are dishonest server and clients.

Original:  FedE w.r.t data privacy and communication efficiency and also keeps similar utility. 
Revised: Then we propose F ED R, a privacy-preserving FL framework on KGs with relation embedding aggregation that defenses against reconstruction attack effectively. Experimental results show that F ED R outperforms FedE w.r.t data privacy and communication efficiency but also keeps similar utility. A Federated Knowledge Graph Embedding Framework

Revised: The federated knowledge graph embedding (FKGE) framework consists of two processes in one communication round: 1) server-side aggregation and 2) client-side update, which are summarized in Algorithm 1, where { P c } denotes the set of permutation matrices and { v c } denotes existence vectors.

Revised: More specifically, P ci,j = 1 indicates that the i -th element in element table is aligned with the j -th element of client c , while v ci = 1 shows that the i -th element in element table exists in client c .

Revised: Besides, ⊘ is element-wide division, ⊗ is elementwide multiplication, and 1 is an all-one vector.

Original: A Implementation Details
Revised: B Knowledge Graph Reconstruction

Revised: We summarize the knowledge graph reconstruction attack in Algorithm 2. Note that in the algorithm, i) and ii) refer to different operations, and only one will be performed in FedE or F ED R. C Implementation Details

Original: For TransE, RotatE, DistMult, and ComplEx, wefollow the same setting as FedE ( ? ).
Revised: For TransE, RotatE, DistMult, and ComplEx, we follow the same setting as FedE (Chen et al., 2021).

Original: Specifically,the number of negative sampling, margin γ andthe negative sampling temperature α are set as 256,10 and 1, respectively.
Revised: Specifically, the number of negative sampling, margin γ and the negative sampling temperature α are set as 256, 10 and 1, respectively.

Original: Note that, we fix the issuein the FedE’s source code that local non-existententities will  be taken as negative samples.
Revised: Note that, we adopt a more conservative strategy for embedding aggregation where local non-existent entities will not be taken as negative samples compared to

Original:  For NoGE, we use GCN (Kipf and Welling,2016) as encoder and QuatE (Zhang et al., 2019)as decoder.
Revised: FedE. For NoGE, we use GCN (Kipf and Welling, 2016) as encoder and QuatE (Zhang et al., 2019) as decoder.

Original: Once local training is done in a communciation round, the embeddings are aggregatedand the triplet is scored by the decoder.
Revised: Once local training is done in a communciation round, the embeddings are aggregated and the triplet is scored by the decoder.

Original: The hiddensize of 1 hidden layer in NoGE is 128.
Revised: The hidden size of 1 hidden layer in NoGE is 128.

Original: Since the aggregated information is not exploitedin the local training in NoGE, we also implement
Revised: Since the aggregated information is not exploited in the local training in NoGE, we also implement

Original: KB-GAT (Nathani et al., 2019), the other GNNmodel
Revised: KB-GAT (Nathani et al., 2019), the other GNN model

Original: but it can take advantages of both graphstructure learning and global-view information aggregation.
Revised: but it can take advantages of both graph structure learning and global-view information aggregation.

Original: However, Fed-KB-GAT is memoryconsuming.
Revised: However, Fed-KB-GAT is memoryconsuming.

Original: For KB-GAT, we use GAT (Veliˇckovi´cet al., 2018) as encoder and ConvKB (Nguyen et al.,2017) as decoder.
Revised: For KB-GAT, we use GAT (Veliˇckovi´c et al., 2018) as encoder and ConvKB (Nguyen et al., 2017) as decoder.

Original: Although the input to KB-GATis the triple embedding, this model update neuralnetwork weights to obtain the final entity and relation embeddings.
Revised: Although the input to KB-GAT is the triple embedding, this model update neural network weights to obtain the final entity and relation embeddings.

Original: In each communication, we letthe aggregated embeddings be the new input to KBGAT, we find using small local epoches lead to badperformance because the model is not fully trainedto produce high-quality embeddings.
Revised: In each communication, we let the aggregated embeddings be the new input to KBGAT, we find using small local epoches lead to bad performance because the model is not fully trained to produce high-quality embeddings.

Original: Therefore,we set local epoch of GAT layers as 500, whilelocal epoch of convlutional layers as 150.
Revised: Therefore, we set local epoch of GAT layers as 500, while local epoch of convlutional layers as 150.

Original: Embedding size is 50 instead of 128 like others since wesuffers memory problem using this model.
Revised: Embedding size is 50 instead of 128 like others since we suffers memory problem using this model.

Original: If not specified, the local update epoch is 3, theembedding dimension of entities and relation is128.
Revised: If not specified, the local update epoch is 3, the embedding dimension of entities and relation is 128.

Original: Early stopping is utilized in experiments.
Revised: Early stopping is utilized in experiments.

Original: Thepatience, namely the number of epochs with no improvement in MRR on validation data after whichtraining will be stopped, is set as 5.
Revised: The patience, namely the number of epochs with no improvement in MRR on validation data after which training will be stopped, is set as 5.

Original: We use Adamwith learning rate 0 .
Revised: We use Adam with learning rate 0 .

Original: 001 for local model update.
Revised: 001 for local model update.

Revised: C. Scoring Function

Revised: [Table]

Original: B Additional Results 
Revised: D Additional Results D. Convergence Analysis

Original: B. Convergence Analysis

Original: The convergence curves considering four KGEmodels and three dataset are shown in Figure 4.
Revised: The convergence curves considering four KGE models and three dataset are shown in Figure 4.

Revised: The solid and dashed lines represent curves w.r.t

Revised: F ED R and FedE, respectively.

Original: We do not show the curves of NoGE because theaggregated embeddings does not influence localtraining.
Revised: We do not show the curves of NoGE because the aggregated embeddings does not influence local training.

Original: We observe that F ED R usually convergefaster than FedE.[Figure] [Figure] [Table]
Revised: We observe that F ED R usually converge faster than FedE.

Original: B. Privacy Quantization

Original: An extensive experiment is to quantify privacy leakage on DDB14 as shown in Table 5.

Original: B. Experiment result with KB-GAT
Revised: D. Experiment result with KB-GAT

Original: We conduct KB-GAT with both entity aggregationand relation aggregation on DDB14 with C = 3as shown in Table 6.
Revised: We conduct KB-GAT with both entity aggregation and relation aggregation on DDB14 with C = [Table] [Figure]

Original: Due to the good performanceof RotatE, we also compare KB-GAT with RotatE. Hit@N is also utilized in the evaluation. Fromthe table, KB-GAT beats RotatE in regard of allevaluation metrics in both FedE and FedR setting.
Revised: [Figure]

Original: However, how to implement federated KB-GAT ina memory-efficient way is still an open problem. C Knowledge Graph Reconstruction We summarize the knowledge graph reconstructionattack in Algorithm 1. Note that in the algorithm,i) and ii) refer to different operations, and only onewill be performed in FedE or F ED R.
Revised: as shown in Table 8.

Original: [Table]

Original: Entity reconstruction: embedding

Original: LEE do do greatest

Original: Triple one of i) and ii) will be implemented 6 i) embeddings ( do

Original: Calculate relation embedding ˆ r scoring function of used KGE model, e.g. ˆ r = ˆ t − ˆ h TransE for relation-embedding ( R, r
Revised: Due to the good performance of RotatE

Original: EEP do Calculate similarity between r r Update the inferred relation ˆ R = R the greatest similarity score 11 return the reconstructed relation set { ˆ R } 12 ii) for relation embedding ˆ r ∈ LRE do for relation-embedding
Revised: [Figure]

Original: ( R, r ) ∈ EEP do Calculate similarity between r and ˆ r Update the inferred relation ˆ R = R with the greatest similarity score 16 return the reconstructed relation set { ˆ R } 17 Utilize { ˆ E } and { ˆ R } to reconstruct triples. D Federated Knowledge Graph
Revised: we also compare KB-GAT with RotatE.

Original: Embedding Framework

Original: [-Input : local datasets T c , number of clients C , number of local epochs E , learning rate η Server excutes: 1 initialize E ∗ 
 * denotes r – use F ED R or e – use F ED E 2 for round t = 0 , 1 , ... do Sample a set of clients C t for c ∈ C t do-] in parallel
Revised: [-Input : local datasets T c , number of clients C , number of local epochs E , learning rate η Server excutes: 1 initialize E ∗ 
 * denotes r – use F ED R or e – use F ED E 2 for round t = 0 , 1 , ... do Sample a set of clients C t for c ∈ C t do-]Hit@N is also utilized in the evaluation.

Original: Send permuted relation embedding matrix to client c : E ∗ ,ct ← P c E t

Original: E ∗ ,ct +1 ← Update ( c, E t )

Original: E ∗ t +1 ← ( 1 ⊘ C t (cid:80) c

Original: =1 v c ) ⊗ C t (cid:80)

Original: c

Original: =1 P c E ∗ ,ct + 9 return E ∗ Client excutes Update ( c, E ) : 10 for each local epoch e = 1 , 2 , ..., E do for batch b , t ) T c do i)
Revised: From the table, KB-GAT beats RotatE in regard of all evaluation metrics in both FedE and FedR setting.

Original: E ← E − η ∇L ii) w w η ∇L only i) ii) will be implemented 14 return E ∗ ,c ∈ E := { E e,c , E r,c } The federated knowledge graph embedding (FKGE) framework consists of two processes in one communication round: 1) server-side aggregation and 2) client-side update, which are summarized in Algorithm 2, where { P c } denotes the set of permutation matrices and { v c } denotes existence vectors. More specifically, P ci,j = 1 indicates that the i -th element in element table is aligned with the j -th element of client c , while v ci = 1 shows that the i -th element in element table exists in client c .
Revised: However, how to implement federated KB-GAT in a memory-efficient way is still an open problem.

Original: Besides, ⊘ is element-wide division, ⊗ is elementwide multiplication, and 1 is an all-one vector.

Original: E Scoring Function

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]