Original: P RECISION C OLLABORATION FOR F EDERATED L EARNING
Revised: P RECISION C OLLABORATION FOR F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Inherent heterogeneity of local data distributions, which causes inefficient model learning and significant degradation of model performance, has been a key challenge in Federated Learning (FL).
Revised: Inherent heterogeneity of local data distributions, which causes inefficient model learning and significant degradation of model performance, has been a key challenge in Federated Learning (FL).

Original: So far, plenty of efforts have focused on addressing data heterogeneity by relying on a hypothetical clustering structure or a consistent information sharing mechanism.
Revised: So far, plenty of efforts have focused on addressing data heterogeneity by relying on a hypothetical clustering structure or a consistent information sharing mechanism.

Original: However, because of the diversity of the real-world local data, these assumptions may be largely violated.
Revised: However, because of the diversity of the real-world local data, these assumptions may be largely violated.

Original: In this work, we argue that information sharing is mostly fragmented in the federated network in reality.
Revised: In this work, we argue that information sharing is mostly fragmented in the federated network in reality.

Original: More specifically, the distribution overlaps are not consistent but scattered in local clients.
Revised: More specifically, the distribution overlaps are not consistent but scattered among local clients.

Original: In this work, we propose the concept “Precision Collaboration” which refers to learning from the informative overlaps precisely while avoiding the potential negative transfer induced by others.
Revised: We propose the concept “Precision Collaboration” which refers to learning from the informative overlaps precisely while avoiding the potential negative transfer induced by others.

Original: In particular, we propose to infer the local data manifolds and estimate the exact local data density simultaneously.
Revised: In particular, we propose to infer the local data manifolds and estimate the exact local data density simultaneously.

Original: The learned manifold aims to precisely identify the overlaps from other clients, and the estimated likelihood allows to generate samples from the manifold in an optimal sampling density.
Revised: The learned manifold aims to precisely identify the overlaps from other clients, and the estimated likelihood allows to generate samples from the manifold in an optimal sampling density.

Original: Experiments show that our proposed PCFL significantly overcomes baselines on benchmarks and a real-world clinical scenario.
Revised: Experiments show that our proposed PCFL significantly overcomes baselines on benchmarks and a real-world clinical scenario.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) has drawn considerable interest from a variety of disciplines in recent years.
Revised: Federated learning (FL) has drawn considerable interest from a variety of disciplines in recent years.

Original: FL enables collaborative model learning without the need to access the raw data across different clients, which facilitates real-world scenarios where privacy preservation is crucial, such as finance (Yang et al., 2019), healthcare (Xu et al., 2021) and criminal justice (Berk, 2012).
Revised: FL enables collaborative model learning without the need to access the raw data across different clients, which facilitates real-world scenarios where privacy preservation is crucial, such as finance (Yang et al., 2019), healthcare (Xu et al., 2021) and criminal justice (Berk, 2012).

Original: While it is common that the data samples in local clients are non-i.i.d., existing research reveals that data heterogeneity could lead to non-guaranteed convergence, inconsistent performance and catastrophic forgetting across different clients (Qu et al., 2022).
Revised: While it is common that the data samples in local clients are non-i.i.d., existing research reveals that data heterogeneity could lead to non-guaranteed convergence, inconsistent performance and catastrophic forgetting across different clients (Qu et al., 2022).

Original: Despite the promise of FL, an increasing concern is how to effectively handle data heterogeneity before FL is applied in real-world data scenarios.
Revised: Despite the promise of FL, an increasing concern is how to effectively handle data heterogeneity before FL is applied in real-world data scenarios.

Original: In view of this challenge, an important direction is personalization.
Revised: In view of this challenge, an important direction is personalization.

Original: A variety of efforts have been made to explore this direction.
Revised: A variety of efforts have been made to explore this direction.

Original: For example, Ghosh et al.
Revised: For example, Ghosh et al.

Original: (2020) proposed to cluster the clients according to their sample distributions and build a customized model for each cluster.
Revised: (2020) proposed to cluster the clients according to their sample distributions and build a customized model for each cluster.

Original: However, their hypothesis excludes the possibility of knowledge transfer across clusters.
Revised: However, their hypothesis excludes the possibility of knowledge transfer across clusters.

Original: Li et al.
Revised: Li et al.

Original: (2021b) enhanced personalized model learning by introducing a global regularization term, which assumed that the shared knowledge was consistent across all clients.
Revised: (2021b) enhanced personalized model learning by introducing a global regularization term, which assumed that the shared knowledge was consistent across all clients.

Original: Considering the diversity of local data, in this paper, we study a more flexible and general scenario where the distribution overlaps could be fragmented as shown in Figure 1 (a).
Revised: Considering the diversity of local data, in this paper, we study a more flexible and general scenario where the distribution overlaps could be fragmented as shown in Figure 1 (a).

Original: Since the informative and ambiguous data shards exist simultaneously in another client, collaborating with all data could do harm to the model learning.
Revised: Since the informative and ambiguous data shards exist simultaneously in another client, collaborating with all data could do harm to the model learning.

Original: An interesting and challenging problem is how to selectively collaborate with the favorable part of other clients in a privacy-preserving way.
Revised: An interesting and challenging problem is how to selectively collaborate with the favorable part of other clients in a privacy-preserving way.

Original: In this paper, we put forward the concept “Precision Collaboration” for fragmented information sharing.
Revised: In this paper, we put forward the concept “Precision Collaboration” for fragmented information sharing.

Original: To begin with, we argue that data heterogeneity comes from inconsistent local data manifolds.
Revised: To begin with, we argue that data heterogeneity comes from inconsistent local data manifolds.

Original: In particular, the data manifolds of different local clients could share different overlaps.
Revised: In particular, the data manifolds of different local clients could share different overlaps.

Original: Maximizing the benefit of collaboration requires a precise utilization of these overlaps.
Revised: Maximizing the benefit of collaboration requires a precise utilization of these overlaps.

Original: Moreover, local data are usually gathered from the manifold based on a particular density.
Revised: Moreover, local data are usually gathered from the manifold based on a particular density.

Original: If we want to generate data from the manifold, a precise distribution density approximation for each client could facilitate model learning.
Revised: If we want to generate data from the manifold, a precise distribution density approximation for each client could facilitate model learning.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /sLi0XLRXVhtplCVc1HnkR0nTY6nwZSjqSipfNONEsMDzxZk3ONDxsyuRpDIKT9UwFu2A9ULZlZwpojp2sRUw1efMHx2NL0Zy3LFLbtk1y/kJKjkoIV/Hkf2EFi4RgSNDAIEQirAPhpSec1TgIiaujRFxCSFp4gJjzJM3I5UgBSN2QN8e7c5zNqS9zpkaN6dTfHoTcjrYJE9EuoSwPs0x8cxk1uxvuUcmp77bkP5enisgVqFP7F++D+V/fboWhS72TA2SaooNo6vjeZbMdEXf3PlUlaIMMXEaX1I8IcyN86PPjvGkpnbdW2biL0apWb3nuTbDq74lDbjyfZw/QWO7XKmWd06qpdp+PupZrGMDWzTPXdRwiGPUzczv8YBHq2FdWzfW7bvUKuSeNXxZ1t0b6WOWBA==</latexit>
Revised: /sLi0XLRXVhtplCVc1HnkR0nTY6nwZSjqSipfNONEsMDzxZk3ONDxsyuRpDIKT9UwFu2A9ULZlZwpojp2sRUw1efMHx2NL0Zy3LFLbtk1y/kJKjkoIV/Hkf2EFi4RgSNDAIEQirAPhpSec1TgIiaujRFxCSFp4gJjzJM3I5UgBSN2QN8e7c5zNqS9zpkaN6dTfHoTcjrYJE9EuoSwPs0x8cxk1uxvuUcmp77bkP5enisgVqFP7F++D+V/fboWhS72TA2SaooNo6vjeZbMdEXf3PlUlaIMMXEaX1I8IcyN86PPjvGkpnbdW2biL0apWb3nuTbDq74lDbjyfZw/QWO7XKmWd06qpdp+PupZrGMDWzTPXdRwiGPUzczv8YBHq2FdWzfW7bvUKuSeNXxZ1t0b6WOWBA==</latexit>

Original: To realize our proposed precision collaboration, we develop a novel framework named PCFL shown in Figure 1.
Revised: To realize our proposed precision collaboration, we develop a novel framework named PCFL shown in Figure 1.

Original: We assert that the key to precisely collaborative model learning is identifying and utilizing the distribution overlaps scattered in other clients.
Revised: We assert that the key to precisely collaborative model learning is identifying and utilizing the distribution overlaps scattered in other clients.

Original: These overlaps between clients indeed correspond to a specific data manifold region.
Revised: These overlaps between clients indeed correspond to a specific data manifold region.

Original: We propose to infer the local data manifold to identify the overlaps.
Revised: We propose to infer the local data manifold to identify the overlaps.

Original: While it is hard to learn the local manifold from the insufficient data in local clients directly, we firstly infer the underlying manifold M g of the data from all clients, so that the data from all overlapped distributions are utilized for the manifold inference.
Revised: While it is hard to learn the local manifold from the insufficient data in local clients directly, we firstly infer the underlying manifold M g of the data from all clients, so that the data from all overlapped distributions are utilized for the manifold inference.

Original: Then the local manifold
Revised: Then the local manifold

Original: M i ⊂ M g of the i -th client could be determined by local data D i as shown in Figure 1 (b).
Revised: M i ⊂ M g of the i -th client could be determined by local data D i as shown in Figure 1 (b).

Original: From Figure 1 (c), the local data manifold M i is used to identify the beneficial overlaps from other clients.
Revised: From Figure 1 (c), the local data manifold M i is used to identify the beneficial overlaps from other clients.

Original: In particular, if a subset of the data from D j lies on M i , this subset is the overlaps between the i -th and j -th clients.
Revised: In particular, if a subset of the data from D j lies on M i , this subset is the overlaps between the i -th and j -th clients.

Original: To further boost the local model training, we suggest sampling from M i with an optimal sampling probability estimated from local data as shown in Figure 1 (d), which effectively mitigates the potential distribution discrepancy.
Revised: To further boost the local model training, we suggest sampling from M i with an optimal sampling probability estimated from local data as shown in Figure 1 (d), which effectively mitigates the potential distribution discrepancy.

Original: We highlight our key contributions as follows:
Revised: We highlight our key contributions as follows:

Original: • Compared with existing research  under certain assumptions about the information sharing mechanism, we study a more general learning scenario where the shared knowledge is fragmented among local clients; • We achieve a more precise collaboration for the federated network by proposing a framework PCFL .
Revised: • While existing research studies FL under certain assumptions about the information sharing, we investigate a more general learning scenario where the data sharing a common distribution is fragmented among local clients; • We achieve a more precise collaboration for the federated network by proposing a framework PCFL .

Original: Our frameworkencourages a novel efficient metric that identifies the meaningful overlaps and excludes ambiguous information from other clients; •Because of the adaptability of our proposed framework, PCFL could be used in any other SOTA algorithms.
Revised: Our framework identifies the meaningful overlaps and excludes ambiguous information from other clients, which avoids potential negative transfer; • PCFL could be used to improve other SOTA algorithms in a plug-and-play way.

Original: Empirical experiments corroborate that PCFL significantly outperforms all baselines on a series of benchmark data sets and a real-world clinical data set.
Revised: Empirical experiments corroborate that PCFL significantly outperforms all baselines on a series of benchmark data sets and a real-world clinical data set.

Original:  R ELATED W ORK 2. F EDERATED L EARNING AND D ATA H ETEROGENEITY
Revised:  R ELATED W ORK 2. F EDERATED L EARNING AND D ATA H ETEROGENEITY

Original: Recent years have witnessed growing attention to federated learning (McMahan et al., 2017), of which several challenges have been concerning topics including communication efficiency (Koneˇcn`y et al., 2016), privacy (Agarwal et al., 2018) and data heterogeneity (Karimireddy et al., 2020).
Revised: Recent years have witnessed growing attention to federated learning (McMahan et al., 2017), of which several challenges have been concerning topics including communication efficiency (Koneˇcn`y et al., 2016), privacy (Agarwal et al., 2018) and data heterogeneity (Karimireddy et al., 2020).

Original: While data heterogeneity could cause the lack of convergence and the potential of catastrophic forgetting (Qu et al., 2022), there are researchers aiming to tackle the heterogeneity by learning a global model.
Revised: While data heterogeneity could cause the lack of convergence and the potential of catastrophic forgetting (Qu et al., 2022), there are researchers aiming to tackle the heterogeneity by learning a global model.

Original: For example, Li et al.
Revised: For example, Li et al.

Original: (2020) propose a proximal term to restrict the local updates to be closer to the initial model.
Revised: (2020) propose a proximal term to restrict the local updates to be closer to the initial model.

Original: Mohri et al.
Revised: Mohri et al.

Original: (2019) seek a fair model performance distribution by maximizing the model performance on any arbitrary target distribution.
Revised: (2019) seek a fair model performance distribution by maximizing the model performance on any arbitrary target distribution.

Original: Li et al.
Revised: Li et al.

Original: (2021a) develop MOON that corrects local training by maximizing the agreements of representation between local and global models.
Revised: (2021a) develop MOON that corrects local training by maximizing the agreements of representation between local and global models.

Original: Instead of pursuing a balanced performance distribution, we are interested in achieving the best performance for each client by precisely learning the shared informative overlaps from others.
Revised: Instead of pursuing a balanced performance distribution, we are interested in achieving the best performance for each client by precisely learning the shared informative overlaps from others.

Original: 2. P ERSONALIZED F EDERATED L EARNING
Revised: 2. P ERSONALIZED F EDERATED L EARNING

Original: In addition to reaching a global consensus, personalized model learning also attracts widespread concern in FL community, which may boost the flexibility of learned models when adapting to local distributions (Cui et al., 2022; Li et al., 2021b).
Revised: In addition to reaching a global consensus, personalized model learning also attracts widespread concern in FL community, which may boost the flexibility of learned models when adapting to local distributions (Cui et al., 2022; Li et al., 2021b).

Original: Plentiful research have proposed techniques for a
Revised: Plentiful research have proposed techniques for a

Original: 𝐷shared 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷shared shared 𝐷𝐷𝐷(a) clustered sharing 𝐷shared 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷shared shared 𝐷𝐷𝐷(b) common partial sharing
Revised: 𝐷shared 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷shared shared 𝐷𝐷𝐷(a) clustered sharing 𝐷shared 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷shared shared 𝐷𝐷𝐷(b) common partial sharing

Original: D<latexit sha1_base64="JCv6WOhB15V2AUti8Rwvk8RYji8=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC3FVwbSFWiVJp3VoXkwmSind+ANu9cvEP9C/8M4YQS2iE5KcOfeeM3Pv9ZKAp9KyXgrGzOzc/
Revised: D<latexit sha1_base64="JCv6WOhB15V2AUti8Rwvk8RYji8=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC3FVwbSFWiVJp3VoXkwmSind+ANu9cvEP9C/8M4YQS2iE5KcOfeeM3Pv9ZKAp9KyXgrGzOzc/

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: /</latexit>
Revised: /</latexit>

Original: D<latexit sha1_base64="jL4AOAbe4zvnMokVpEY2w03ebOw=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC3FVwbSFWiVJp3VoXkwmSind+ANu9cvEP9C/8M4YQS2iE5KcOfeeM3Pv9ZKAp9KyXgrGzOzc/EJxsbS0vLK6Vl7faKZxJnzm+HEQi7bnpizgEXMklwFrJ4K5oRewljc8VvHWLRMpj6MLOUpYN3QHEe9z35VEOSdXY3tyXa5YVUsvcxrYOaggX424/IxL9BDDR4YQDBEk4QAuUno6sGEhIa6LMXGCENdxhglKpM0oi1GGS+yQvgPadXI2or3yTLXap1MCegUpTeyQJqY8QVidZup4pp0V+5v3WHuqu43o7+VeIbESN8T+pfvM/K9O1SLRx6GugVNNiWZUdX7ukumuqJubX6qS5JAQp3CP4oKwr5WffTa1JtW1q966Ov6qMxWr9n6em+FN3ZIGbP8c5zRo7lXtWnX/vFapH+WjLmIL29ileR6gjlM04JA3xwMe8WScGYlxZ4w+Uo1CrtnEt2XcvwMtbZEA</latexit>
Revised: D<latexit sha1_base64="jL4AOAbe4zvnMokVpEY2w03ebOw=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC3FVwbSFWiVJp3VoXkwmSind+ANu9cvEP9C/8M4YQS2iE5KcOfeeM3Pv9ZKAp9KyXgrGzOzc/EJxsbS0vLK6Vl7faKZxJnzm+HEQi7bnpizgEXMklwFrJ4K5oRewljc8VvHWLRMpj6MLOUpYN3QHEe9z35VEOSdXY3tyXa5YVUsvcxrYOaggX424/IxL9BDDR4YQDBEk4QAuUno6sGEhIa6LMXGCENdxhglKpM0oi1GGS+yQvgPadXI2or3yTLXap1MCegUpTeyQJqY8QVidZup4pp0V+5v3WHuqu43o7+VeIbESN8T+pfvM/K9O1SLRx6GugVNNiWZUdX7ukumuqJubX6qS5JAQp3CP4oKwr5WffTa1JtW1q966Ov6qMxWr9n6em+FN3ZIGbP8c5zRo7lXtWnX/vFapH+WjLmIL29ileR6gjlM04JA3xwMe8WScGYlxZ4w+Uo1CrtnEt2XcvwMtbZEA</latexit>

Original: D<latexit sha1_base64="hLCUDt5QxC368u6PLgQE21aup0g=">AAACyHicjVHLTsJAFD3UF+ILdemmkZi4Ii3B6JKoC+MKEwskiKYdBpxQ2qadaghh4w+41S8z/oH+hXfGkqjE6DRtz5x7z5m593qRLxJpWa85Y25+YXEpv1xYWV1b3yhubjWSMI0Zd1joh3HLcxPui4A7Ukift6KYu0PP501vcKLizTseJyIMLuUo4p2h2w9ETzBXEuWcXo8rk5tiySpbepmzwM5ACdmqh8UXXKGLEAwphuAIIAn7cJHQ04YNCxFxHYyJiwkJHeeYoEDalLI4ZbjEDujbp107YwPaK89Eqxmd4tMbk9LEHmlCyosJq9NMHU+1s2J/8x5rT3W3Ef29zGtIrMQtsX/pppn/1alaJHo40jUIqinSjKqOZS6p7oq6ufmlKkkOEXEKdykeE2ZaOe2zqTWJrl311tXxN52pWLVnWW6Kd3VLGrD9c5yzoFEp29XywUW1VDvORp3HDnaxT/M8RA1nqMMhb4FHPOHZODci494YfaYauUyzjW/LePgAL86RAQ==</latexit>
Revised: D<latexit sha1_base64="hLCUDt5QxC368u6PLgQE21aup0g=">AAACyHicjVHLTsJAFD3UF+ILdemmkZi4Ii3B6JKoC+MKEwskiKYdBpxQ2qadaghh4w+41S8z/oH+hXfGkqjE6DRtz5x7z5m593qRLxJpWa85Y25+YXEpv1xYWV1b3yhubjWSMI0Zd1joh3HLcxPui4A7Ukift6KYu0PP501vcKLizTseJyIMLuUo4p2h2w9ETzBXEuWcXo8rk5tiySpbepmzwM5ACdmqh8UXXKGLEAwphuAIIAn7cJHQ04YNCxFxHYyJiwkJHeeYoEDalLI4ZbjEDujbp107YwPaK89Eqxmd4tMbk9LEHmlCyosJq9NMHU+1s2J/8x5rT3W3Ef29zGtIrMQtsX/pppn/1alaJHo40jUIqinSjKqOZS6p7oq6ufmlKkkOEXEKdykeE2ZaOe2zqTWJrl311tXxN52pWLVnWW6Kd3VLGrD9c5yzoFEp29XywUW1VDvORp3HDnaxT/M8RA1nqMMhb4FHPOHZODci494YfaYauUyzjW/LePgAL86RAQ==</latexit>

Original: [Figure]
Revised: [Figure]

Original: trade-off between local and global models.
Revised: trade-off between local and global models.

Original: For example, Fallah et al.
Revised: For example, Fallah et al.

Original: (2020) proposed to train local models that can quickly adapt to local data starting from an initial shared model in a meta-learning way.
Revised: (2020) proposed to train local models that can quickly adapt to local data starting from an initial shared model in a meta-learning way.

Original: Some works train personalized models by interpolating between global and local models (Deng et al., 2020;
Revised: Some works train personalized models by interpolating between global and local models (Deng et al., 2020;

Original: Dinh et al., 2020).
Revised: Dinh et al., 2020).

Original: Li et al.
Revised: Li et al.

Original: (2021b)
Revised: (2021b)

Original: achieve such a trade-off through regularizing local models close to the global model.
Revised: achieve such a trade-off through regularizing local models close to the global model.

Original: There are other works suggesting a partially shared model structure for efficient information transferring (Liang et al., 2020; Collins et al., 2021).
Revised: There are other works suggesting a partially shared model structure for efficient information transferring (Liang et al., 2020; Collins et al., 2021).

Original: Nonetheless, we are concerned that a global model is hard to model the various shared information between clients.
Revised: Nonetheless, we are concerned that a global model is hard to model the various shared information between clients.

Original: The fragmented knowledge requires precise identification when collaboratively learning from others.
Revised: The fragmented knowledge requires precise identification when collaboratively learning from others.

Original: To mitigate the potential overfitting when learning from limited local data, some works also attempt generative methods to improve the model performance (Du & Wu, 2020; Zhu et al., 2021).
Revised: To mitigate the potential overfitting when learning from limited local data, some works also attempt generative methods to improve the model performance (Du & Wu, 2020; Zhu et al., 2021).

Original: In particular, Zhu et al.
Revised: In particular, Zhu et al.

Original: (2021) regulate local training with the distilled knowledge from all clients.
Revised: (2021) regulate local training with the distilled knowledge from all clients.

Original: Du & Wu (2020) lead into GAN for generating similar data for local clients.
Revised: Du & Wu (2020) lead into GAN for generating similar data for local clients.

Original: However, generating data in an arbitrary density could result in distribution discrepancy.
Revised: However, generating data in an arbitrary density could result in distribution discrepancy.

Original: An optimal sampling density may present more benefits for local learning tasks.
Revised: An optimal sampling density may present more benefits for local learning tasks.

Original:  N OTATIONS AND P ROBLEM D EFINITION 3. N OTATIONS
Revised:  N OTATIONS AND P ROBLEM D EFINITION 3. N OTATIONS

Original: Suppose there are N clients in a federated network, each client owns a private dataset D k with n k data samples.
Revised: Suppose there are N clients in a federated network, each client owns a private dataset D k with n k data samples.

Original: The dataset D k = (cid:8) X k , Y k (cid:9) consists of the input space X k and output space Y k .
Revised: The dataset D k = (cid:8) X k , Y k (cid:9) consists of the input space X k and output space Y k .

Original: We use z = { x, y } to denote a data point, and z ∈ M denotes the data manifold.
Revised: We use z = { x, y } to denote a data point, and z ∈ M denotes the data manifold.

Original: The input space and the output space are shared across all clients.
Revised: The input space and the output space are shared across all clients.

Original: In the following, we also use D i to denote the i -th client without causing further confusion.
Revised: In the following, we also use D i to denote the i -th client without causing further confusion.

Original: The goal of each client is to learn a best model to predict the label y by collaborating with others.
Revised: The goal of each client is to learn a best model to predict the label y by collaborating with others.

Original: For example, McMahan et al.
Revised: For example, McMahan et al.

Original: (2017) propose FedAvg , which learns a global model f for all clients by minimizing the empirical risk over the samples from all clients, i.e.,
Revised: (2017) propose FedAvg , which learns a global model f for all clients by minimizing the empirical risk over the samples from all clients, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where F is the hypothesis space and l denotes the loss objective of all clients.
Revised: where F is the hypothesis space and l denotes the loss objective of all clients.

Original: From Eq.1, FedAvg assumes that the data from different clients associate with a common data manifold M and sampling density p gz ( z ) , i.e.,
Revised: From Eq.1, FedAvg assumes that the data from different clients associate with a common data manifold M and sampling density p gz ( z ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: 3. A SSUMPTIONS ON D ATA H ETEROGENEITY
Revised: 3. A SSUMPTIONS ON D ATA H ETEROGENEITY

Original: However, i.i.d. assumption in Eq.(2) is largely violated as the local data distributions may be significantly distinctive.
Revised: However, i.i.d. assumption in Eq.(2) is largely violated as the local data distributions may be significantly distinctive.

Original: In this event, learning a consensus by averaging the local gradients could cause severe performance degradation on certain clients (Li et al., 2019b; Cui et al., 2021).
Revised: In this event, learning a consensus by averaging the local gradients could cause severe performance degradation on certain clients (Li et al., 2019b; Cui et al., 2021).

Original: There are research studying federated learning with non-i.i.d.
Revised: There are research studying federated learning with non-i.i.d.

Original: data and the assumptions on data heterogeneity are mainly from
Revised: data and the assumptions on data heterogeneity are mainly from

Original: Clustered sharing.
Revised: Clustered sharing.

Original: As shown Figure 2 (a), the clients partitioned in each cluster own a common data manifolds ( M j ) and sampling density ( p jz ( z ) ), i.e.,
Revised: As shown Figure 2 (a), the clients partitioned in each cluster own a common data manifolds ( M j ) and sampling density ( p jz ( z ) ), i.e.,

Original: ∀ i ∈ { 0 , 1 , ..., N − 1 } , ∃ j ∈ { 0 , 1 , ..., K } ( K < N ) , s.t., z ∈ D i ⊂ M j and z ∼ p
Revised: ∀ i ∈ { 0 , 1 , ..., N − 1 } , ∃ j ∈ { 0 , 1 , ..., K } ( K < N ) , s.t., z ∈ D i ⊂ M j and z ∼ p

Original: jz ( z ) .
Revised: jz ( z ) .

Original: From Eq.(3), clustered sharing requires that all message is shared within the clusters, and there is no knowledge transferring across clusters.
Revised: From Eq.(3), clustered sharing requires that all message is shared within the clusters, and there is no knowledge transferring across clusters.

Original: Common partial sharing.
Revised: Common partial sharing.

Original: From Figure 2 (b), a common distribution overlap is shared across all clients.
Revised: From Figure 2 (b), a common distribution overlap is shared across all clients.

Original: Meanwhile, each client owns specific knowledge that cannot be leveraged by others.
Revised: Meanwhile, each client owns specific knowledge that cannot be leveraged by others.

Original: Formally, each client associates with a specific data manifold M i , and the overlapped region of the manifold is shared across all clients, i.e.,
Revised: Formally, each client associates with a specific data manifold M i , and the overlapped region of the manifold is shared across all clients, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Compared with the previous assumptions above, we study a more general scenario fragmented sharing , where the shared distribution overlaps are scattered among the clients.
Revised: Compared with the previous assumptions above, we study a more general scenario fragmented sharing , where the shared distribution overlaps are scattered among the clients.

Original: Besides, these overlaps are inconsistent across all clients as shown in Figure 2 (c).
Revised: Besides, these overlaps are inconsistent across all clients as shown in Figure 2 (c).

Original: Fragmented sharing.
Revised: Fragmented sharing.

Original: The local data z ∈ D i are sampled from the local manifold M i in a particular density p iz ( z ) , and there exist overlaps among data manifolds, i.e.,
Revised: The local data z ∈ D i are sampled from the local manifold M i in a particular density p iz ( z ) , and there exist overlaps among data manifolds, i.e.,

Original: [Equation]
Revised: [Equation]

Original: 5b
Revised: 5b

Original: [Equation]
Revised: [Equation]

Original: where d in Eq.(5a) is the dimension of z . Eq.(5b) implies the shared overlaps may not be consistent across all clients, e.g., ∩ N − 1 i =

Revised: where d in Eq.(5a) is the dimension of z . Eq.(5b) implies the shared overlaps may not be consistent across all clients, e.g., ∩ N − 1 i =

Original: M i = ∅ .
Revised: M i = ∅ .

Original:  M ETHODOLOGY 4. P RELIMINARY : N ORMALIZING
Revised:  M ETHODOLOGY 4. P RELIMINARY : N ORMALIZING

Original: F LOW
Revised: F LOW

Original: Normalizing flow.
Revised: Normalizing flow.

Original: The generative method NF achieves exact likelihood estimation through an invertible transformation from a known distribution to a complex target distribution.
Revised: The generative method NF achieves exact likelihood estimation through an invertible transformation from a known distribution to a complex target distribution.

Original: Given a target dataset D = { z 0 , z 1 , ..., z n − 1 } , z i ∈ R d and a base variable e ∈ R d with a known density p e ( e ) , classic NF methods learn a diffeomorphism f : e i = g
Revised: Given a target dataset D = { z 0 , z 1 , ..., z n − 1 } , z i ∈ R d and a base variable e ∈ R d with a known density p e ( e ) , classic NF methods learn a diffeomorphism f : e i = g

Original: ( z i ) which maps p z to the density p e :
Revised: ( z i ) which maps p z to the density p e :

Original: [Equation]
Revised: [Equation]

Original: (6) where det J g ( g ( z ))
Revised: (6) where det J g ( g ( z ))

Original: ∈ R
Revised: ∈ R

Original: d × d denotes the Jacobian matrix evaluated at g ( z ) .
Revised: d × d denotes the Jacobian matrix evaluated at g ( z ) .

Original: Since g is bijective, it is trackable and Eq.(6) could be effectively computed.
Revised: Since g is bijective, it is trackable and Eq.(6) could be effectively computed.

Original: By fitting the dataset D , the approximated distribution p ′
Revised: By fitting the dataset D , the approximated distribution p ′

Original: z ( z ) is optimized through a pushforward operation.
Revised: z ( z ) is optimized through a pushforward operation.

Original: To enhance the scalability of g , one could compose several diffeomorphisms g
Revised: To enhance the scalability of g , one could compose several diffeomorphisms g

Original: = g n − 1 ◦ · · · ◦ g 1 ◦ g 0 for a larger model capacity.
Revised: = g n − 1 ◦ · · · ◦ g 1 ◦ g 0 for a larger model capacity.

Original: 4. A N O VERVIEW OF PCFL
Revised: 4. A N O VERVIEW OF PCFL

Original: M is <latexit sha1_base64="ODfW+TKQLjdq3OB/966CExorUh4=">AAAC2HicjVHLSsNAFD3Gd31Vu3QTLIKrkoqiy6IbN4KCrWKrZTKd1sG8SCZCCQV34tYfcKtfJP6B/oV3xhTUIjohyZlz7zkz91438mSiHOd1zBqfmJyanpktzM0vLC4Vl1caSZjGXNR56IXxmcsS4clA1JVUnjiLYsF81xOn7vW+jp/eiDiRYXCi+pG48FkvkF3JmSKqXSy1fKauOPOyw8FlJgftLBm0i2Wn4phlj4JqDsrI11FYfEELHYTgSOFDIIAi7IEhoaeJKhxExF0gIy4mJE1cYIACaVPKEpTBiL2mb492zZwNaK89E6PmdIpHb0xKG+ukCSkvJqxPs008Nc6a/c07M576bn36u7mXT6zCFbF/6YaZ/9XpWhS62DU1SKopMoyujucuqemKvrn9pSpFDhFxGncoHhPmRjnss200iald95aZ
Revised: M is <latexit sha1_base64="ODfW+TKQLjdq3OB/966CExorUh4=">AAAC2HicjVHLSsNAFD3Gd31Vu3QTLIKrkoqiy6IbN4KCrWKrZTKd1sG8SCZCCQV34tYfcKtfJP6B/oV3xhTUIjohyZlz7zkz91438mSiHOd1zBqfmJyanpktzM0vLC4Vl1caSZjGXNR56IXxmcsS4clA1JVUnjiLYsF81xOn7vW+jp/eiDiRYXCi+pG48FkvkF3JmSKqXSy1fKauOPOyw8FlJgftLBm0i2Wn4phlj4JqDsrI11FYfEELHYTgSOFDIIAi7IEhoaeJKhxExF0gIy4mJE1cYIACaVPKEpTBiL2mb492zZwNaK89E6PmdIpHb0xKG+ukCSkvJqxPs008Nc6a/c07M576bn36u7mXT6zCFbF/6YaZ/9XpWhS62DU1SKopMoyujucuqemKvrn9pSpFDhFxGncoHhPmRjnss200iald95aZ

Original: [Table]
Revised: [Table]

Original: M i
Revised: M i

Original: [Table]
Revised: [Table]

Original: +PupZrGMDWzTPXdRwiGPUzczv8YBHq2FdWzfW7bvUKuSeNXxZ1t0b6WOWBA==</latexit>
Revised: +PupZrGMDWzTPXdRwiGPUzczv8YBHq2FdWzfW7bvUKuSeNXxZ1t0b6WOWBA==</latexit>

Original: Learning an optimal personalized model f i for the i -th client expects a sufficient utilization of the shared overlaps with other clients.
Revised: Learning an optimal personalized model f i for the i -th client expects a sufficient utilization of the shared overlaps with other clients.

Original: However, due to privacy concerns, one cannot identify these overlaps with direct access to the raw data.
Revised: However, due to privacy concerns, one cannot identify these overlaps with direct access to the raw data.

Original: We suggest leveraging the overlaps via the learned data manifold to prevent privacy leakage.
Revised: We suggest leveraging the overlaps via the learned data manifold to prevent privacy leakage.

Original: As shown in Figure 3, in general, our proposed precisely collaborative learning scheme contains:
Revised: As shown in Figure 3, in general, our proposed precisely collaborative learning scheme contains:

Original: • for the shared overlaps in other clients, we aim to precisely learn from the shared overlaps identified by the local data manifold M i ; • for the remaining unshared region of M i , we expect to advance models with the generated synthetic data from M i in an optimal sampling density.
Revised: • for the shared overlaps in other clients, we aim to precisely learn from the shared overlaps identified by the local data manifold M i ; • for the remaining unshared region of M i , we expect to advance models with the generated synthetic data from M i in an optimal sampling density.

Original: 4. P RECISION C OLLABORATION I: L EARNING FROM THE S HARED O VERLAPS
Revised: 4. P RECISION C OLLABORATION I: L EARNING FROM THE S HARED O VERLAPS

Original: From Figure 1 (a), different clients could share different distribution overlaps, and the distribution overlaps associate with the overlapped region of the local data manifolds.
Revised: From Figure 1 (a), different clients could share different distribution overlaps, and the distribution overlaps associate with the overlapped region of the local data manifolds.

Original: While the data manifold of local clients is mostly agnostic and hardly be inferred by limited local data, we propose to learn the global data manifold with the data on all clients.
Revised: While the data manifold of local clients is mostly agnostic and hardly be inferred by limited local data, we propose to learn the global data manifold with the data on all clients.

Original: In this way, all data are utilized and contribute to the manifold inference.
Revised: In this way, all data are utilized and contribute to the manifold inference.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: <

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: base [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: d7jiCqGXcm42lxfNjJF6ySpZY5D2wNCtCrGuVfcIMuInhIMQRDCEE4gIOEnhZsWIiJa2NCHCfkqzjDFDnypqRipHCIHdC3T7uWZkPay5yJcnt0SkAvJ6eJA/JEpOOE5Wmmiqcqs2R/

Original: [Table] [Table]
Revised: [Table] [Table]

Original: LaMhw
Revised: +

Original: [Table]
Revised: [Table]

Original: </latexit>
Revised: /pRyS6Xjq/KhcqZHnUWe9hHkeZ5ggouUEVNzfERT3g2Lo3EGBv3n1Ijoz

Original: [Equation]
Revised: [Table]

Original:  ⇠ 
Revised: LaMhw+h4ZJf</latexit> z ⇠ p z ( z ) <latexit sha

Original: [Equation]
Revised: [Table]

Original: <
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: e
Revised: e

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: /0L74wlUYnRadqeOfeeM3PvtSPPTbhhvGa0ufmFxaXscm5ldW19I

Revised: /0L74wlUYnRadqeOfeeM3PvtSPPTbhhvGa0ufmFxaXscm5ldW19I

Original: +51UhCETus7oReGLdsK2GeG7A6d7nHWlHMLN/2WNMensp485bFiRsGl3wUsY5vDQK37zoWJ+oq6o5FcVISxf1uvmCUDbX0WWCmoIB01cL8C67RQwgHAj4YAnDCHiwk9LRhwkBEXAdj4mJCroozTJAjraAsRhkWsUP6DmjXTtmA9tIzUWqHTvHojUmpY480IeXFhOVpuooL5SzZ37zHylPebUR/O/XyieW4IfYv3TTzvzpZC0cfx6oGl2qKFCOrc1IXoboib65/qYqTQ0ScxD2Kx4QdpZz2WVeaRNUue2up+JvKlKzcO2muwLu8JQ3Y/DnOWdA4KJuV8uFFpVA9SUedxQ52UaJ5HqGKM9RQJ+8Aj3jCs3auCe1Ou/9M1TKpZhvflvbwAZR3krc=</latexit>
Revised: +51UhCETus7oReGLdsK2GeG7A6d7nHWlHMLN/2WNMensp485bFiRsGl3wUsY5vDQK37zoWJ+oq6o5FcVISxf1uvmCUDbX0WWCmoIB01cL8C67RQwgHAj4YAnDCHiwk9LRhwkBEXAdj4mJCroozTJAjraAsRhkWsUP6DmjXTtmA9tIzUWqHTvHojUmpY480IeXFhOVpuooL5SzZ37zHylPebUR/O/XyieW4IfYv3TTzvzpZC0cfx6oGl2qKFCOrc1IXoboib65/qYqTQ0ScxD2Kx4QdpZz2WVeaRNUue2up+JvKlKzcO2muwLu8JQ3Y/DnOWdA4KJuV8uFFpVA9SUedxQ52UaJ5HqGKM9RQJ+8Aj3jCs3auCe1Ou/9M1TKpZhvflvbwAZR3krc=</latexit>

Original: Split
Revised: Split

Original: Learn the global manifold.
Revised: Learn the global manifold.

Original: The data z ∈ D are usually supported on an unknown lower-dimensional manifold M .
Revised: The data z ∈ D are usually supported on an unknown lower-dimensional manifold M .

Original: In our realization, we propose to use a NF method to learn the global manifold M g .
Revised: In our realization, we propose to use a NF method to learn the global manifold M g .

Original: Given the data z , a bijective transformation g θ is used to obtain the latent representation e ∈ E , e = g θ ( z ) , where z = g − 1 θ ( e ) , (7) which avoids the risk of information loss during encoding.
Revised: Given the data z , a bijective transformation g θ is used to obtain the latent representation e ∈ E , e = g θ ( z ) , where z = g − 1 θ ( e ) , (7) which avoids the risk of information loss during encoding.

Original: While a classical NF requires the latent variable e ∈ E to have the same dimension with the data space
Revised: While a classical NF requires the latent variable e ∈ E to have the same dimension with the data space

Original: Z , following the design in (Brehmer & Cranmer, 2020), we separate the latent space E =
Revised: Z , following the design in (Brehmer & Cranmer, 2020), we separate the latent space E =

Original: U
Revised: U

Original: × V as shown in Figure 4, where U = R d ′ denotes the coordinates on the manifold.
Revised: × V as shown in Figure 4, where U = R d ′ denotes the coordinates on the manifold.

Original: V = 

Revised: V =

Original: d − d ′ denotes the remaining coordinates, which are the directions orthogonal to the manifold.
Revised: d − d ′ denotes the remaining coordinates, which are the directions orthogonal to the manifold.

Original: To model the density p u ( u ) , we transform the variable u to the variable u ′ with the given density p u ′ ( u ′ ) using a bijective model h ϕ : u ′ =
Revised: To model the density p u ( u ) , we transform the variable u to the variable u ′ with the given density p u ′ ( u ′ ) using a bijective model h ϕ : [Equation]

Original: h ϕ ( u ) , where u = Split ( e ) , (8) where Split ( e ) denotes deleting the d − d ′ dimensional 0 vector from e , and Pad ( u ) denotes the inverse operation.
Revised: where Split ( e ) denotes deleting the d − d ′ dimensional 0 vector from e , and Pad ( u ) denotes the inverse operation.

Original: Please note that in the rest of this paper we will use g ∗ θ to denote Split ◦ g θ and g ∗− 1 θ denotes g − 1 θ ◦ Pad.
Revised: Please note that in the rest of this paper we will use g ∗ θ to denote Split ◦ g θ and g ∗− 1 θ denotes g − 1 θ ◦ Pad.

Original: After the training of the model with the parameters θ and ϕ , we learn a diffeomorphism from the data z ∼ p
Revised: After the training of the model with the parameters θ and ϕ , we learn a diffeomorphism from the data z ∼ p

Original: z ( z ) to a lower dimension space u ′ ∼ p u ′ ( u ′ ) with the encoder h ϕ ◦ g ∗ θ .
Revised: z ( z ) to a lower dimension space u ′ ∼ p u ′ ( u ′ ) with the encoder h ϕ ◦ g ∗ θ .

Original: This means that we transform the original data manifold
Revised: This means that we transform the original data manifold

Original: M g to the projected data manifold U ′ .
Revised: M g to the projected data manifold U ′ .

Original: Note that the decoder is the inverse of the encoder.
Revised: Note that the decoder is the inverse of the encoder.

Original: The data is reconstructed given the latent variable u ′ ∈ U ′ , i.e., z = g ∗− 1 θ ◦ h − 1 ϕ ( u ′ ) .
Revised: The data is reconstructed given the latent variable u ′ ∈ U ′ , i.e., z = [Equation] ∗− [Equation]

Revised: Following the work in (Brehmer & Cranmer, 2020), we train g θ and h ϕ by a two-stage optimization framework.

Original:  Determine the local manifold.
Revised: In particular, we first train g θ to obtain the projection onto the manifold by minimizing the reconstruction error. Then, we optimize h ϕ to approximate the density by maximizing the likelihood (Brehmer & Cranmer, 2020). More implementation details could be found in Appendix due to the page limit. Determine the local manifold.

Original: A local data manifold M i should contain the local data D i .
Revised: A local data manifold M i should contain the local data D i .

Original: Considering that the original global manifold M g and the latent space U ′
Revised: Considering that the original global manifold M g and the latent space U ′

Original: ( U ′ =
Revised: ( U ′ =

Original: R d ′ ) is topologically equivalent, we propose to approximate the local manifold with the projected representation:
Revised: R d ′ ) is topologically equivalent, we propose to approximate the local manifold with the projected representation:

Original: [Equation]
Revised: [Equation]

Original: where U ′ i denotes the set of the samples transformed to U ′ from D i , and U ′ i is called the projected local data manifold, which is computed as the convex hull of U
Revised: where U ′ i denotes the set of the samples transformed to U ′ from D i , and U ′ i is called the projected local data manifold, which is computed as the convex hull of U

Original: ′ i .
Revised: ′ i .

Original: Note that U ′ i may have a clustered structure.
Revised: Note that U ′ i may have a clustered structure.

Original: In the realization, we could also firstly cluster the U
Revised: In the realization, we could also firstly cluster the U

Original: ′ i .
Revised: ′ i .

Original: The union of the convex hull of all clusters is the projected local data manifold, and the original local data manifold ( M i ) could be obtained by M i 
Revised: The union of the convex hull of all clusters is the projected local data manifold, and the original local data manifold ( M i ) could be obtained by M i =

Original: = g ∗− 1 θ ◦

Original: [Equation]
Revised: [Equation]

Revised: ∗−

Original: [Equation]
Revised: [Equation]

Original: Identify the data overlaps from other clients.
Revised: Identify the data overlaps from other clients.

Original: Since we cannot determine the data overlaps directly because of privacy concerns, we propose to identify the overlaps using the learned local manifolds.
Revised: Since we cannot determine the data overlaps directly because of privacy concerns, we propose to identify the overlaps using the learned local manifolds.

Original: Note that the data overlaps correspond to the overlaps of the data manifolds.
Revised: Note that the data overlaps correspond to the overlaps of the data manifolds.

Original: For example, suppose
Revised: For example, suppose

Original: D i,j is a subset of D i , if D i,j lies on M i , D i,j is the data overlap between D i and D j .
Revised: D i,j is a subset of D i , if D i,j lies on M i , D i,j is the data overlap between D i and D j .

Original: From Eq.(9),
Revised: From Eq.(9),

Original: M i is reconstructed by g ∗− 1 θ ◦ h − 1 ϕ with U
Revised: M i is reconstructed by g ∗− 1 θ ◦ h − 1 ϕ with U

Original: ′ i .
Revised: ′ i .

Original: Therefore, D i,j could be identified as follows:
Revised: Therefore, D i,j could be identified as follows:

Original: [Equation]
Revised: [Equation]

Original: From Eq.(10), the data overlap D i,j is the subset in which each sample is transformed in the projected local data manifold U
Revised: From Eq.(10), the data overlap D i,j is the subset in which each sample is transformed in the projected local data manifold U

Original: ′ i .
Revised: ′ i .

Original: By learning from the overlaps identified from other clients, we have the following objective,
Revised: By learning from the overlaps identified from other clients, we have the following objective,

Original: [Equation]
Revised: [Equation]

Original: where α > 0 is the regularization parameter, which controls the trade-off between the risk on the i -th client and other clients.
Revised: where α > 0 is the regularization parameter, which controls the trade-off between the risk on the i -th client and other clients.

Original: 4. P RECISION C OLLABORATION II: L EARNING WITH AN O PTIMAL S AMPLING D ENSITY
Revised: 4. P RECISION C OLLABORATION II: L EARNING WITH AN O PTIMAL S AMPLING D ENSITY

Original: In Sec.4.3, we learn personalized models from the data overlaps between clients.
Revised: In Sec.4.3, we learn personalized models from the data overlaps between clients.

Original: However, the model performance on the unshared data cannot be improved by collaborating with others.
Revised: However, the model performance on the unshared data cannot be improved by collaborating with others.

Original: The specific region M is has no overlap with others, which is formulated as
Revised: The specific region M is has no overlap with others, which is formulated as

Original: [Equation]
Revised: [Equation]

Original: We propose to advance the model by generating data sampled from the local manifold M i .
Revised: We propose to advance the model by generating data sampled from the local manifold M i .

Original: While an arbitrary sampling density could generate data D ′
Revised: While an arbitrary sampling density could generate data D ′

Original: i deviated from the local distribution d
Revised: i deviated from the local distribution d

Original: ( D ′ i , D i ) >
Revised: ( D ′ i , D i ) >

Original: ϵ , this could induce bias to the learned model.
Revised: ϵ , this could induce bias to the learned model.

Original: An optimal utilization of the synthetic data expects a sampling density close to p iz ( z ) .
Revised: An optimal utilization of the synthetic data expects a sampling density close to p iz ( z ) .

Original: Therefore, we propose to sample from M i with the exact estimation of p iz ( z ) .
Revised: Therefore, we propose to sample from M i with the exact estimation of p iz ( z ) .

Original: Exact likelihood estimation.
Revised: Exact likelihood estimation.

Original: Note that we learn the manifold by applying a normalizing flow framework, which achieves the exact likelihood estimation simultaneously.
Revised: Note that we learn the manifold by applying a normalizing flow framework, which achieves the exact likelihood estimation simultaneously.

Original: Since we learn the global data manifold, the global data density p gz ( z ) is transformed to p u ′ ( u ′ ) .
Revised: Since we learn the global data manifold, the global data density p gz ( z ) is transformed to p u ′ ( u ′ ) .

Original: For the local data density p iz ( z ) , we have the following proposition.
Revised: For the local data density p iz ( z ) , we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: (proof in Appendix)
Revised: (proof in Appendix)

Original: For any data point z ∈ M is , the local density p iz ( z ) satisfies
Revised: For any data point z ∈ M is , the local density p iz ( z ) satisfies

Original: [Equation]
Revised: [Equation]

Original:  where c is a proportionality constant, and J h ϕ and J g ∗ θ are the Jacobian matrix of h ϕ and g ∗ θ , respectively.
Revised: [Equation] where c is a proportionality constant, and J h ϕ and J g ∗ θ are the Jacobian matrix of h ϕ and g ∗ θ , respectively.

Original: From the Proposition 1, to sample z ∈ M is in the density p iz ( z ) , we could firstly sample u ′ ∼ p u ′ ( u ′ ) and choose u ′ ∈
Revised: From the Proposition 1, to sample z ∈ M is in the density p iz ( z ) , we could firstly sample u ′ ∼ p u ′ ( u ′ ) and choose u ′ ∈

Original: U ′ is defined in Eq.(12).
Revised: U ′ is defined in Eq.(12).

Original: Then we transform the sampled u ′ to the data space by z = g ∗− 1 θ ◦ h − 1 ϕ ( u ′ ) .
Revised: Then we transform the sampled u ′ to the data space by z = g ∗− 1 θ ◦ h − 1 ϕ ( u ′ ) .

Original: The final objective is as follows
Revised: The final objective is as follows

Original: [Equation]
Revised: [Equation]

Original: where the sampled ( x, y ) in the third term satisfies ( x, y ) ∈
Revised: where the sampled ( x, y ) in the third term satisfies ( x, y ) ∈

Original: M is .
Revised: M is .

Original: 4. A N EW M ETRIC OF C LIENT S IMILARITY

Original: Our framework inspires a novel metric for measuring the similarity between local clients.

Original: For example, suppose the i -th client and j -th client has the identical local manifold M i =

Original: M j , the similarity between clients is close to 1.

Original: On the contrary, if the two local manifolds are disjoint

Original: M i ∩ M j = ∅ , the measured similarity should be 0.

Original: In particular, we propose to measure the similarity as the Intersection of Union (IoU) of the projected local manifold,

Original: [Equation]

Original: A communication-efficient client-level collaboration.

Original: Our proposed metric allows an efficient collaborator identification which reduces the communication and computation overhead.

Original: For example, we require D i to collaborate with certain clients who have a higher client similarity:

Original: [Equation]

Original: where ϵ ≥ 0 is a pre-defined threshold.

Original: Note that the objective in Eq.(16) is different from clustered

Original: FL methods.

Original: Clustered FL methods learn a common model for each cluster while Eq.(16) learns a personalized model for each client.

Original: Experimental results shown in Sec.

Original: D.1 in Appendix verify that this method achieves a comparable performance while reducing communication and computation overhead.

Original: [Figure]

Original: 4. M ORE D ISCUSSION ON PCFL
Revised: 4. M ORE D ISCUSSION ON PCFL

Original: From the objective formulated in Eq.(14), PCFL advances a generative framework foran efficiently collaborative learning.
Revised: From the objective formulated in Eq.(14), PCFL advances a generative framework for efficiently collaborative learning.

Original: In addition to the properties summarized as follows, more discussions could be found in Appendix,
Revised: In addition to the properties summarized as follows, more discussions could be found in Appendix,

Original: • Scalability: Since PCFL achieves a smarter utilization of data from other clients, it is pluggable for other FL algorithms.
Revised: • Scalability: Since PCFL achieves a smarter utilization of data from other clients, it is pluggable for other FL algorithms.

Original: Experiments in Sec.
Revised: Experiments in Sec.

Original: 5 also verify that PCFL could benefit other SOTA baselines; • Optimization: PCFL approximates a transformed low-dimensional manifold, which significantly reduces the computational overhead.
Revised: 5 also verify that PCFL could benefit other SOTA baselines; • Similarity metric: PCFL identifies the overlaps among federated networks, which inspires a novel metric for measuring the similarity between clients.

Original: Experiments in Appendix indicate that PCFL does not bring much additional computation cost and has a similar time consumption compared with others;
Revised: More information about it could be found in Appendix.

Original: • Privacy: From Algorithm 2, PCFL learns models by communicating model parameters between clients and the server. There is no additional risk of privacy leakage compared with baselines. PCFL is also compatible with other privacy-preserving techniques, such as differential privacy, secure multi-party computation, etc.
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We intuitively show the motivation of our method by conducting experiments on synthetic data.
Revised: We intuitively show the motivation of our method by conducting experiments on synthetic data.

Original: We compare our method with various baselines on a wide range of benchmark datasets, including image and tabular datasets.
Revised: We compare our method with various baselines on a wide range of benchmark datasets, including image and tabular datasets.

Original: More importantly, the practicability of our method is validated in a real-world clinical federated scenario on eICU dataset (Pollard et al., 2018).
Revised: More importantly, the practicability of our method is validated in a real-world clinical federated scenario on eICU dataset (Pollard et al., 2018).

Original: The source codes are made publically available at https://github.com/pcfl/pcfl .
Revised: The source codes are made publically available at https://github.com/pcfl/pcfl .

Original: 5. S YNTHETIC E XPERIMENTS
Revised: 5. S YNTHETIC E XPERIMENTS

Original: Synthetic data.
Revised: Synthetic data.

Original: Suppose there are 96 clients: D i , i ∈ { 1 , 2 ..., 96 } .
Revised: Suppose there are 96 clients: D i , i ∈ { 1 , 2 ..., 96 } .

Original: The data points z = { x, y } is generated from two objectives
Revised: The data points z = { x, y } is generated from two objectives

Original: y = sin ( x ) + ϵ or y = − sin ( x ) + ϵ shown in Figure 5 (a), where ϵ ∼ N (0 , 0 . 1) denotes label noise.
Revised: y = sin ( x ) + ϵ or y = − sin ( x ) + ϵ shown in Figure 5 (a), where ϵ ∼ N (0 , 0 . 1) denotes label noise.

Original: Fragmented data overlaps.
Revised: Fragmented data overlaps.

Original: To generate heterogeneous and overlapped local data, we sample x from the overlapped ranges.
Revised: To generate heterogeneous and overlapped local data, we sample x from the overlapped ranges.

Original: In particular, we separate the input space X into four intervals [0 , π 2 ] , [ π 2 , π ] , [ π, 3 π 2 ] and [ 3 π 2 , 2 π ] , and each client randomly chooses two different intervals to sample data.
Revised: In particular, we separate the input space X into four intervals [0 , π 2 ] , [ π 2 , π ] , [ π, 3 π 2 ] and [ 3 π 2 , 2 π ] , and each client randomly chooses two different intervals to sample data.

Original: To create conflicting learning tasks, the label of the selected 48 clients is calculated by y = sin ( x ) + ϵ , and the label of the remaining 48 clients is calculated by y = − sin ( x ) + ϵ . 
Revised: To create conflicting learning tasks, the label of the selected 48 clients is calculated by y = sin ( x ) + ϵ , and the label of the remaining 48 clients is calculated by y = − sin ( x ) + ϵ . [Table]

Original:  In this setting, learning a global model for all clients could hurt the model performance as there are two conflicting learning tasks as shown in Figure 5 (c).
Revised: [Figure] In this setting, learning a global model for all clients could hurt the model performance as there are two conflicting learning tasks as shown in Figure 5 (c).

Original: The best way of collaborative learning for each client is identifying the data overlaps which are sampled from the identical objective with the same intervals.
Revised: The best way of collaborative learning for each client is identifying the data overlaps which are sampled from the identical objective with the same intervals.

Original: For example, D 0 consists of the data sampled from [0 , π 2 ] and [ π 2 , π ] , while Dconsists of the data sampled from [ π 2 , π ] and [ π, 3 π 2 ] shown in Figure 5 (a).
Revised: For example, D 0 consists of the data sampled from [0 , π 2 ] and [ π 2 , π ] , while D 1 consists of the data sampled from [ π 2 , π ] and [ π, 3 π 2 ] shown in Figure 5 (a).

Original: Learning an optimal model for D 0 needs to precisely identify the data overlap sampled from [ π 2 , π ] in D 1 .
Revised: Learning an optimal model for D 0 needs to precisely identify the data overlap sampled from [ π 2 , π ] in D 1 .

Original: From Figure(e), PCFL efficiently obtains local data manifolds and identifies the data overlaps between clients.
Revised: From Figure 5 (e), PCFL efficiently obtains local data manifolds and identifies the data overlaps between clients.

Original: Therefore, PCFL learns a better model by precision collaboration which maximizes the benefits and avoids potential negative transfer from other clients as shown inFigure 5 (d) [Figure]
Revised: Therefore, PCFL learns a better model by precision collaboration which maximizes the benefits and avoids potential negative transfer from other clients as shown in

Original: [Figure]
Revised: Figure 5 (d). Datasets.

Original: We adopt three benchmark image datasets: CIFAR10 (Krizhevsky et al., 2009), FEMNIST (Caldas et al., 2018), CelebA (Liu et al., 2015), and a tabular dataset Adult (Kohavi et al., 1996).
Revised: We adopt three benchmark image datasets: CIFAR10 (Krizhevsky et al., 2009), FEMNIST (Caldas et al., 2018), CelebA (Liu et al., 2015), and a tabular dataset Adult (Kohavi et al., 1996).

Original: We create the federated environment with data heterogeneity for CIFAR by randomly allocating several classes to each client following the work (McMahan et al., 2017).
Revised: We create the federated environment with data heterogeneity for CIFAR10 by randomly allocating several classes to each client following the work (McMahan et al., 2017).

Original: We use K to denote the number of clients and S to denote the number of classes in each client.
Revised: We use K to denote the number of clients and S to denote the number of classes in each client.

Original: For CIAFR10, K = 150 , S = 5 means there are 150 clients and each client containsclasses of images.
Revised: For CIAFR10, K = 150 , S = 5 means there are 150 clients and each client contains 5 classes of images.

Original: For FEMNIST which has 10 classes of handwritten letters, we consider the setting of K = 200 , S = 5 .
Revised: For FEMNIST which has 10 classes of handwritten letters, we consider the setting of K = 200 , S = 5 .

Original: The number of samples in each client is determined according to a log-normal distribution (Li et al., 2019a).
Revised: The number of samples in each client is determined according to a log-normal distribution (Li et al., 2019a).

Original: The task on CelebA is to classify whether the celebrity in the image is smiling (Li et al., 2021b).
Revised: The task on CelebA is to classify whether the celebrity in the image is smiling (Li et al., 2021b).

Original: There are 545 clients and 21 samples per client in average.
Revised: There are 545 clients and 21 samples per client in average.

Original: For the tabular dataset Adult , the task is to predict whether an individual’s income is beyond 50K/year based on some census features, including age, race, workclass, etc.
Revised: For the tabular dataset Adult , the task is to predict whether an individual’s income is beyond 50K/year based on some census features, including age, race, workclass, etc.

Original: Following the setting in (Mohri et al., 2019), all individuals are split into two clients.
Revised: Following the setting in (Mohri et al., 2019), all individuals are split into two clients.

Original: One is PhD client and the other is non-PhD client.
Revised: One is PhD client and the other is non-PhD client.

Original: [Table] [Figure] [Table] [Figure]
Revised: 5. B ENCHMARK E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: Baselines.

Original: We compare our method with various baseline, including global and personalized methods.
Revised: We compare our method with various baseline, including global and personalized methods.

Original: Global baselines include: 1) FedAvg (McMahan et al., 2017); 2)
Revised: Global baselines include: 1) FedAvg (McMahan et al., 2017); 2)

Original: FedProx (Li et al., 2020).
Revised: FedProx (Li et al., 2020).

Original: Personalized baselines include: 1) Fed-MTL (Smith et al., 2017); 2) PerFedAvg (Fallah et al., 2020); 3)
Revised: Personalized baselines include: 1) FedMTL (Smith et al., 2017); 2) PerFedAvg (Fallah et al., 2020); 3)

Original: LG-FedAvg (Liang et al., 2020); 4) FedPer (Arivazhagan et al., 2019); 5) FedRep (Collins et al., 2021); 6) 
Revised: LGFedAvg (Liang et al., 2020); 4) FedPer (Arivazhagan et al., 2019); 5) FedRep (Collins et al., 2021); 6) APFL (Deng et al., 2020); 7) L2GD

Original: APFL (Deng et al., 2020); 7) L2GD (Hanzely & Richtárik, 2020); 8) Ditto (Li et al., 2021b).[Figure]
Revised: (Hanzely & Richtárik, 2020); 8) Ditto (Li et al., 2021b).

Original: Experimental Results.
Revised: Experimental Results.

Original: The accuracy of all methods on CIFAR dataset are shown in Table 1 1 .
Revised: The accuracy of all methods on CIFAR10 dataset are shown in Table 1 1 .

Original: PCFL outperforms all baselines on this classification task.
Revised: PCFL outperforms all baselines on this classification task.

Original: Since each client has insufficient data samples ( n i = 333 ), FedAvg ( 72 . 3% ) learning from all data has a better performance compared with local ( 68 . 9 ).
Revised: Since each client has insufficient data samples ( n i = 333 ), FedAvg ( 72 . 3% ) learning from all data has a better performance compared with local ( 68 . 9 ).

Original: From Table 1, FedRep ( 82 . 2% ) surpasses other baselines by learning a global feature extractor.
Revised: From Table 1, FedRep ( 82 . 2% ) surpasses other baselines by learning a global feature extractor.

Original: As a pluggable method, PCFL could be used to enhance the performance of other art methods.
Revised: As a pluggable method, PCFL could be used to enhance the performance of other art methods.

Original: From Figure 6, PCFL improves the performace of FedRep by 5 . 1% , which indicates that PCFL effectively identifies the informative knowledge from others.
Revised: From Figure 6, PCFL improves the performance of FedRep by 5 . 1% , which indicates that PCFL effectively identifies the informative knowledge from others.

Original: Similar phenomena could also be found in the experimental results on FEMNIST and CelebA.
Revised: Similar phenomena could also be found in the experimental results on FEMNIST.

Original: On both datasets, FedAvg achieves a better performance compared with other baselines because of the relatively slighter heterogeneity.
Revised: FedAvg achieves a better performance compared with other baselines because of the relatively slighter heterogeneity.

Original: PCFL also outperforms all baselines on both learning tasks.
Revised: PCFL also outperforms all baselines on both learning tasks.

Original: Moreover, PCFL successifully boosts the models learning on the three baselines as shown inFigureand Figure 8.
Revised: Moreover, PCFL successfully boosts the models learning on the three baselines as shown in Figure 7. More experimental results could be found in Appendix. Table 3:

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We also evaluate the availability of our proposed method on a tabular data Adult
Revised: We also evaluate the availability of our proposed method on a tabular data Adult

Original: [Figure]
Revised: [Figure]

Original: and the results on the two clients are shown in Table 4.
Revised: and the results on the two clients are shown in Table 3.

Original: Since the classifier on Adult is one-layer MLP, LG-FedAvg,
Revised: Since the classifier on Adult is one-layer MLP, LG-FedAvg,

Original: FedPer and FedRep degrade into local training.
Revised: FedPer and FedRep degrade into local training.

Original: Compared with PhD client which has 413 training samples, non-PhD has more than 30000 training samples.
Revised: Compared with PhD client which has 413 training samples [Figure] non-PhD has more than 30000 training samples.

Original: From Table 4, all methods achieve similar performance on non-PhD client.
Revised: From Table 3, all methods achieve similar performance on non-PhD client.

Original: Because of the severe distribution discrepancy, naive averaging may not lead to optimal accuracy on non-PhD client.
Revised: Because of the severe distribution discrepancy, naive averaging may not lead to optimal accuracy on non-PhD client.

Original: From Figure 9, by leveraging the favorable data in non-PhD client and the learned manifold of PhD client, PCFL substantially improves the model performance of local training ( ↑ 7 . 1% ).
Revised: From Figure [Figure] by leveraging the favorable data in non-PhD client and the learned manifold of PhD client, PCFL substantially improves the model performance of local training ( ↑ 7 . 1% ).

Original: 5. R EAL D ATA E XPERIMENTS
Revised: 5. R EAL D ATA E XPERIMENTS

Original: To further verify the practicability of PCFL , we conduct experiments on a real-world clinical dataset eICU
Revised: To further verify the practicability of PCFL , we conduct experiments on a real-world clinical dataset eICU

Original: (Pollard et al., 2018).
Revised: (Pollard et al., 2018).

Original: eICU contains the patients to ICUs with hospital information.
Revised: eICU contains the patients to ICUs with hospital information.

Original: Naturally, hospitals located in different areas are local clients as in (Cui et al., 2022), where the patient data are kept confidential.
Revised: Naturally, hospitals located in different areas are local clients as in (Cui et al., 2022), where the patient data are kept confidential.

Original: We preprocess the data following the work (Sheikhalishahi et al., 2019) and each data spans a 1-hour window.
Revised: We preprocess the data following the work (Sheikhalishahi et al., 2019) and each data spans a 1-hour window.

Original: The task is to predict in-hospital mortality of each instance using the 48-hour monitoring data.
Revised: The task is to predict in-hospital mortality of each instance using the 48-hour monitoring data.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In the experiments, we randomly select 14 hospitals in the federated network.
Revised: [Table] [Figure] [Table] [Figure]

Original: We use a Bi-LSTM to implement this binary classification.
Revised: We use a Bi-LSTM to implement this binary classification.

Original: We use AUC as the metric due to the severe label imbalance (more than 90% samples have negative labels).
Revised: We use AUC as the metric due to the severe label imbalance (more than 90% samples have negative labels).

Original: From Table 5, PCFL still maintains the best model utility.
Revised: From Table 4, PCFL still maintains the best model utility.

Original: FedAvg achieves a comparable performance compared with local.
Revised: FedAvg achieves a comparable performance compared with local.

Original: While different hospitals own different populations which could result in data heterogeneity, Ditto learns a robust personalized model and achieves better performance ( 78 . 3% ).
Revised: While different hospitals own different populations which could result in data heterogeneity, Ditto learns a robust personalized model and achieves better performance ( 78 . 3% ).

Original: The results shown in Figure 10 prove that PCFL could also benefit the baselines in real-world scenarios. 
Revised: The results shown in Figure 9 prove that PCFL could also benefit the baselines in real-world scenarios. 5. A BLATION S TUDIES

Revised: While PCFL formulated in Eq.(14) consists three terms: 1).

Revised: loss of local training; 2).

Revised: loss on the identified overlapped data from other clients; 3).

Revised: loss on sampled data from the manifold, to analysis the effect of each component, we conduct ablation studies on several datasets.

Original: C ONCLUSION
Revised: More ablation studies and implementation details could be found in Appendix. When α = 0 , only the local data and sampled data from local manifolds are used for model learning. When β = 0 , only the local data and the identified overlapped data from other clients are utilized for model learning. Experiments in Table 5 demonstrate that 1).

Revised: both the identified distributional overlaps ( β = 0) and the data sampled with a learned distribution density ( α = 0) facilitate the model learning; 2).

Revised: the identified overlaps ( β = 0) could achieve more performance gain than the generated data ( α = 0) .

Original:  In this paper, we propose a precise collaboration framework PCFL for a more general FL learning scenario, where the fragmented and shared knowledge is distributed among other clients.
Revised: [Table]  C ONCLUSION In this paper, we propose a precise collaboration framework PCFL for a more general FL learning scenario, where the fragmented and shared knowledge is distributed among other clients.

Original: Experiments on benchmark datasets and a real-world clinical dataset verify the superiority of our method because of optimal and precise utilization of the shared information.
Revised: Experiments on benchmark datasets and a real-world clinical dataset verify the superiority of our method because of optimal and precise utilization of the shared information.

Original: Our framework determines the overlaps between clients, which suggests several attractive topics, such as identifying malicious clients or noisy data in the federated network.
Revised: Our framework determines the overlaps between clients, which suggests several attractive topics, such as identifying malicious clients or noisy data in the federated network.

Original: Moreover, PCFL encourages a novel similarity metric stated in Sec. 4.5.
Revised: Moreover, PCFL encourages a novel similarity metric stated in Sec. C.1.

Original: This metric could be used to provide incentives or impose charges on each client, to promote the practicality of FL in real-world applications.
Revised: This metric could be used to provide incentives or impose charges on each client, to promote the practicality of FL in real-world applications.

Original: A T HEORETICAL P ROOFS A. P ROOF OF P ROPOSITION
Revised: A T HEORETICAL P ROOFS A. P ROOF OF P ROPOSITION

Original: Suppose there is a smooth and injective mapping g ∗ θ : R d → R d ′ with d ′ ≤ d , U ∈ R d
Revised: Suppose there is a smooth and injective mapping g ∗ θ : R d → R d ′ with d ′ ≤ d , U ∈ R d

Original: ′ is the latent variable and has Z :=
Revised: ′ is the latent variable and has Z :=

Original: g ∗− 1 θ ( U ) .
Revised: g ∗− 1 θ ( U ) .

Original: From differential geometry (Krantz & Parks, 2008), we have
Revised: From differential geometry (Krantz & Parks, 2008), we have

Original: [Equation]
Revised: [Equation]

Original: Suppose there is a smooth and bijective mapping h ϕ : R d ′ → R d ′ , U
Revised: Suppose there is a smooth and bijective mapping h ϕ : R d ′ → R d ′ , U

Original: ′ ∈
Revised: ′ ∈

Original: R d ′ is the latent variable which has U :=
Revised: R d ′ is the latent variable which has U :=

Original: h ϕ ( U ′ ) .
Revised: h ϕ ( U ′ ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: According to the chain rule, combining Eq.(17) and Eq.(18), we have
Revised: According to the chain rule, combining Eq.(15) and Eq.(16), we have

Original: [Equation]
Revised: [Equation]

Original: Since we learn a global manifold M g with the data from all clients, the density of the data from all clients p gz ( z ) is approximated in Eq.(19).
Revised: Since we learn a global manifold M g with the data from all clients [Equation] the density of the data from all clients p gz ( z ) is approximated in Eq.(17).

Original: From the definition of M is in Eq.(12), if there is a data point z ∈ M is , z will cannot be sampled from any other manifolds M j ( j ̸ = i ) but M i , i.e.,
Revised: From the definition of M is in Eq.(12), if there is a data point z ∈ M is , z will cannot be sampled from any other manifolds M j ( j ̸ = i ) but M i , i

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have [Equation]
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: Combining with Eq.(19), for z ∈ M is , we have[Equation] [Equation] h ϕ
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: h ϕ [Equation] θ [Equation]
Revised: Combining with Eq.(17)

Original: det

Original: [Equation]
Revised: [Equation]

Original: ⊤ g

Original: [Equation]

Original: [Equation]

Original:  z [Equation] g ∗ θ [Equation]
Revised: for z ∈ M is , we have

Original: g

Original: [Equation]
Revised: [Equation]

Original: θ

Original: [Equation] z ) [Equation] (cid:12)(cid:12)(cid:12) − 1 / 2 , (22) and Proposition 1 holds.
Revised: and Proposition 1 holds.

Original: B P IPELINE OF OUR FRAMEWORK PCFL
Revised: B P IPELINE OF OUR FRAMEWORK PCFL

Original: The pipeline of the global manifold learning M g is elaborated in Algorithm 1.
Revised: The pipeline of the global manifold learning M g is elaborated in Algorithm 1.

Original: We learn a global manifold model in the federated learning setting.
Revised: We learn a global manifold model in the federated learning setting.

Original: There are two phases of training.
Revised: There are two phases of training.

Original: Firstly, only the parameters of g θ are updated as in Line 5-7.
Revised: Firstly, only the parameters of g θ are updated as in Line 5-7.

Original: Then the parameters of h ϕ are updated as in Line 9-10.
Revised: Then the parameters of h ϕ are updated as in Line 9-10.

Original: The learned manifold model g ∗ θ ◦ h ϕ is utilized in our framework PCFL , whose pipeline is elaborated in Algorithm 2.
Revised: The learned manifold model g ∗ θ ◦ h ϕ is utilized in our framework PCFL , whose pipeline is elaborated in Algorithm 2.

Original: To begin with, the local manifolds of clients are extracted based on Eq.(9) and the distribution overlaps are calculated based on Eq.(10).
Revised: To begin with, the local manifolds of clients are extracted based on Eq.(9) and the distribution overlaps are calculated based on Eq.(10).

Original: Since only the borders of convex hulls are exchanged, there is no leakage of sensitive information.
Revised: Since only the borders of convex hulls are exchanged, there is no leakage of sensitive information.

Original: The data from the overlapped distribution of other clients are used to train the models.
Revised: The data from the overlapped distribution of other clients are used to train the models.

Original: They are utilized by transmitting the average gradients through the server as in Line 7-14.
Revised: They are utilized by transmitting the average gradients through the server as in Line 7-14.

Revised: det do 7: draw mini-batch ( x i , y i ) ∼ D

Revised: i ; 8: calculate the loss E ( x i ,y i )

Revised: ∈

Original: B. O PTIMIZATION E FFICIENCY
Revised: D i ( ℓ ( f i ( x k ) , y k )) + β · E ( x,y ) ∈ p iz ( z ) ℓ ( f i ( x ) , y ) , and update the model f i using the gradients of loss; 9: for k = 0 , ..., N, k ̸

Revised: = i do 10: draw mini-batch ( x k , y k )

Revised: ∼ D

Revised: i,k 11: calculate the loss α · E ( x k ,

Revised: y k ) ∈

Revised: D i,k ℓ ( f i ( x k ) , y k ) , and update the model f

Revised: i using the gradients of loss; 12: end for 13: the Server aggregates the parameters of f i from other clients and send the average to the i -th client; 14: then the i -th client D i updates the model f i with the received parameters and local gradients.

Revised: 15: end for 16: end for 17: Output: the learned personalized models (cid:8) f 0 , ..., f N − 1 (cid:9) .

Revised: C M ORE D ISCUSSIONS ABOUT PCFL C. A N EW M ETRIC OF C LIENT S IMILARITY

Revised: Our framework PCFL inspires a novel metric for measuring the similarity between local clients.

Revised: For example, suppose the i -th client and j -th client has the identical local manifold M i =

Revised: M j , the similarity between clients is close to 1.

Revised: On the contrary, if the two local manifolds are disjoint

Revised: M i ∩ M j = ∅ , the measured similarity should be 0.

Revised: In particular, we propose to measure the similarity as the Intersection of Union (IoU) of the projected local manifold,

Revised: S (

Revised: [Equation]

Revised: A communication-efficient client-level collaboration.

Revised: Our proposed metric allows efficient collaborator identification which reduces the communication and computation overhead.

Revised: For example, we require D i to collaborate with certain clients who have a higher client similarity:

Revised: [Equation]

Revised: where ϵ ≥ 0 is a pre-defined threshold.

Revised: Note that the objective in Eq.(22) is different from clustered

Revised: FL methods.

Revised: Clustered FL methods learn a common model for each cluster while Eq.(22) learns a personalized model for each client.

Revised: Experimental results shown in Sec.

Revised: E.2 verify that this method achieves a comparable performance while reducing communication and computation overhead.

Revised: Previous work has explored the problem of identifying similar datasets in a graph network for downstream learning tasks (Hallac et al., 2015).

Revised: In particular, Jung (2020) formulate the learning from distributed local datasets as a convex optimization problem, and proposes to cluster the local datasets according to the learned parameters.

Revised: Jung & Tran (2019) extend network lasso methods in regression tasks under a clustering assumption.

Revised: These cluster-based methods could be applied in federated learning with a proper design for privacy-preserving.

Revised: In our experiments, we use network lasso to cluster the local datasets under the federated setting.

Revised: In Table 6, we show the comparison of PCFL and the clustered methods.

Revised: Our method outperforms all cluster-based methods, which demonstrates that a precision identification of overlaps in other clients facilitates model learning.

Revised: Moreover, an interesting direction is the application of our proposed similarity metric in the graph network.

Revised: For example, the manifold learning of local datasets in the graph network may also be used for similarity measurement.

Revised: [Table]

Revised: C. C OMPUTATION C OMPLEXITY AND O PTIMIZATION E FFICIENCY

Revised: From Algorithm 1 and Algorithm 2, PCFL is realized by a two-staged optimization framework.

Revised: For the training of the normalizing flow in the first stage, PCFL learns a global model for all clients, which has the computation complexity as FedAvg.

Revised: For the identification of the manifold overlaps, it has O (1) time complexity as the server only computes it once.

Revised: For the training of local models in the second stage, PCFL learns a personalized model for each client, which has the computation complexity as other personalized methods.

Revised: From the above all, PCFL achieves a similar computation complexity as baselines.

Original: A classical NF method requires a fixed dimensionality of the latent space, which is the same as the dimension of the data.
Revised: A classical NF method requires a fixed dimensionality of the latent space, which is the same as the dimension of the data.

Original: In this case, learning such a NF model could bring a huge computation overhead when the data is high-dimensional.
Revised: In this case, learning such a NF model could bring a huge computation overhead when the data is high-dimensional.

Original: In the first phase of our framework, we learn a low-dimensional manifold in a NF method, which significantly reduces the computation overhead.
Revised: In the first phase of our framework, we learn a low-dimensional manifold in a NF method, which significantly reduces the computation overhead.

Original: Our method learns from local data, data overlaps of other clients, and sampled data in the manifold.
Revised: Our method learns from local data, data overlaps of other clients, and sampled data in the manifold.

Original: By precision collaboration, we avoid learning from all data.
Revised: By precision collaboration, we avoid learning from all data.

Original: We make comparisons of run-time consumption with the baselines.
Revised: We make comparisons of run-time consumption with the baselines.

Original: The experiments are conducted on the same device NVIDIA GeForce
Revised: The experiments are conducted on the same device NVIDIA GeForce

Original: RTX 2080 Ti.
Revised: RTX 2080 Ti.

Original: The results on eICU dataset are displayed in Table 6.
Revised: The results on eICU dataset are displayed in Table 7.

Original: As a pluggable method, the time consumption of PCFL is comparable to the corresponding baselines.
Revised: As a pluggable method, the time consumption of PCFL is comparable to the corresponding baselines.

Original: Fed-MTL involves computing the correlation of the parameters among all client models, which could result in more computation overhead. 
Revised: Fed-MTL involves computing the correlation of the parameters among all client models, which could result in more computation overhead. [Table]

Original: B. P RIVACY P RESERVING
Revised: C. P RIVACY P RESERVING

Original: PCFL maintains data confidentiality as baselines because there is no shared data between locao clients.
Revised: PCFL maintains data confidentiality as baselines because there is no shared data between local clients.

Original: For privacy preserving, PCFL achieves privacy preserving as baselines because our framework learns[Table] models by communicating model parameters only.
Revised: PCFL achieves privacy-preserving as baselines because our framework learns models by communicating model parameters only.

Original: Federated learning may need further exploration to maintain data privacy.
Revised: Federated learning may need further exploration to maintain data privacy.

Original: Some researchers claim there is information leakage when sharing models or gradients (Zhu et al., 2019).
Revised: Some researchers claim there is information leakage when sharing models or gradients (Zhu et al., 2019).

Original: To alleviate this issue, there are research proposing to apply other techniques to FL methods, such as differential privacy (Wei et al., 2020).
Revised: To alleviate this issue, there are research proposing to apply other techniques to FL methods, such as differential privacy (Wei et al., 2020), secure multi-party computation, etc.

Original: PCFL is also compatible with these techniques.
Revised: PCFL is also compatible with these techniques.

Original: C A BLATION S TUDIESC. D ISCUSSIONS ABOUTTHE O BJECTIVE OF PCFL
Revised: D A BLATION S TUDIES D. M ORE A BLATION S TUDIES ABOUT PCFL

Original: There are three terms in our learning objective as in Eq.14: 1).

Original: loss of local data, 2).

Original: loss of overlapped data from other clients, 3).

Original: loss of sampled data in the manifold.

Original: When α = β = 0 in the objective, our method degrades to local training.

Original: We perform experiments on several datasets to study the impact of the last two terms in our objective.

Original: The results are displayed in Table 7.

Original: The CIFAR10 dataset contains 50000 training images ofclasses.

Original: We set K = 150 and S = 5 , then each client contains around 333 samples of 5 classes.

Original: Local training is difficult to perform well due to the lack of local data.

Original: When α = 0 , only the local data and sampled data from local manifolds are used in our method.

Original: The local manifolds are extracted from global manifolds.

Original: Sampled data from local manifolds help local models learn the decision boundary more precisely.

Original: Thus the performance exceeds local training.

Original: When β = 0 , only the local data and overlapped data from other clients are utilized.

Original: Data from other clients is selected by manifold overlaps.

Original: It also achieves higher accuracy than local training.

Original: A similar phenomenon could also be discovered on FEMNIST, CelebA, and eICU.

Original: Using only one of the last two terms in our objective function could boost the performance of local training, and the combination of both achieves the best performance.

Original: [Table]

Original: C. M ORE A BLATION S TUDIES ABOUT PCFL

Original: PCFL is pluggable for other FL algorithms.
Revised: PCFL is pluggable for other FL algorithms.

Original: We test local, FedAvg, FedRep and Ditto which are implemented with/without our method as in Table 8 and Table 9.
Revised: We test local, FedAvg, FedRep and Ditto which are implemented with/without our method as in Table 8 and Table 9.

Original: In the datset Adult, all individuals are split into two clients, one of which is PhD client and the other is non-PhD client.
Revised: In the datset Adult, all individuals are split into two clients, one of which is PhD client and the other is non-PhD client.

Original: The non-PhD client contains 32148 training samples while the PhD client contains 413 samples.
Revised: The non-PhD client contains 32148 training samples while the PhD client contains 413 samples.

Original: Therefore the non-PhD client of Adult can not benefit much from federated learning methods.
Revised: Therefore the non-PhD client of Adult can not benefit much from federated learning methods.

Original: In other datasets, our method boosts the baselines by large margins.
Revised: In other datasets, our method boosts the baselines by large margins.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E  XPERIMENTS AND I MPLEMENTATION D ETAILSD. E VALUATION ABOUT THE P ROPOSED N OVEL M ETRIC
Revised: E E XPERIMENTS AND I MPLEMENTATION D ETAILS

Revised: M ORE E XPERIMENTS

Revised: The experimental results on CelebA are shown in Table 11 and Figure 10.

Revised: Moreover, we conduct experiments on FEMNIST on more heterogeneous settings with more clients.

Revised: We partition the dataset into 400 clients with the Dirichlet distribution Dir 400 (0 . 1) and Dir 400 (0 . 5) following the work in (Wang et al., 2019).

Revised: We compare our method with the baselines, and the results are shown in

Original: In Sec. 4.5, we propose a novel metric for measuring the distance between local clients, which could be used for a communication-efficient client-level collaboration.
Revised: Table 10. With more clients, each client has fewer training samples. Local method shows poor performance ( 63 . 6% in Dir 400 (0 . 5) ).

Revised: Global methods (FedAvg and FedProx) achieve better performance under a less heterogeneous setting ( Dir 400 (0 . 5) ), while the performance of personalized methods degrades.

Revised: Under two settings ( Dir 400 (0 . 1) and Dir 400 (0 . 5) ), PCFL outperforms all baselines by identifying the informative overlaps for each client

Revised: [Table]

Revised: [Table]

Revised: E.

Revised: [Table]

Revised: [Figure]

Revised: In Sec.

Revised: C.1, we propose a novel metric for measuring the distance between local clients, which could be used for a communicationefficient client-level collaboration.

Original: We conduct experiments on eICU dataset, in which we select the most similar 7 clients for each client to learn a personalized model.
Revised: We conduct experiments on eICU dataset, in which we select the most similar 7 clients for each client to learn a personalized model.

Original: The experimental results are shown in Table 10. [Table]
Revised: The experimental results are shown in Table 12.

Original: From Table 10, our method for identifying the collaborators achieves a comparable performance compared with baselines and reduces computation and communication overhead by collaborating with a subset of local clients.
Revised: From Table 12, our method for identifying the collaborators achieves a comparable performance compared with baselines and reduces computation and communication overhead by collaborating with a subset of local clients.

Revised: To explore the effect of ϵ on the performance of the learned models, we set ϵ by controlling the number of clients to collaborate for each client.

Revised: There are 14 clients in eICU dataset.

Revised: We test the number of the collaborator ( C ) to be 1, 3, 5, ... etc.

Original: D. I MPLEMENTATION D ETAILS
Revised: The results are shown in Table ?? . When C = 7 , the learned model achieves the highest AUC ( 78 . 0 ). When C > 7 , the performance tends to remain unchanged. E. I MPLEMENTATION D ETAILS

Original: Our method is implemented with Pytorch and all experiments are run 5 times to calculate the average results with stds.
Revised: Our method is implemented with Pytorch and all experiments are run 5 times to calculate the average results with stds.

Original: We use a four-layer MLP for the synthetic experiment, three-layer MLP for
Revised: We use a four-layer MLP for the synthetic experiment, three-layer MLP for

Original: FEMNIST, two-layer CNNs for CIFAR10 and CelebA, and a one-layer MLP for Adult.
Revised: FEMNIST, two-layer CNNs for CIFAR10 and CelebA, and a one-layer MLP for Adult.

Original: Following the work (Collins et al., 2021), for all the methods we sample 10% of the clients in every global epoch.
Revised: Following the work (Collins et al., 2021), for all the methods we sample 10% of the clients in every global epoch.

Original: We train the models for 200 global epochs on FEMNIST, CIFAR10 and CelebA, 50 on
Revised: We train the models for 200 global epochs on FEMNIST, CIFAR10 and CelebA, 50 on

Original: Adult.
Revised: Adult.

Original: And we train 15 local epochs for FEMNIST, CIFAR10 and Adult in every global epoch,for CelebA. All models are trained with stochastic gradient descent.
Revised: And we train 15 local epochs for FEMNIST, CIFAR10 and Adult in every global epoch,for CelebA. All models are trained with stochastic gradient descent.

Original: We use grid search to find the optimal hyperparameters α and β in the validation set of each dataset.
Revised: We use grid search to find the optimal hyperparameters α and β in the validation set of each dataset.

Original: We set α = 0 .
Revised: We set α = 0 .

Original: 5 , β = 0 .
Revised: 5 , β = 0 .

Original: 5 for
Revised: 5 for

Original: CIFAR10, CelebA, Adult; and set α = 1 , β = 0 .
Revised: CIFAR10, CelebA, Adult; and set α = 1 , β = 0 .

Original: 5 for FEMNIST and eICU.
Revised: 5 for FEMNIST and eICU.

Original: Besides, we test different  manifold dimensions d ′ for each benchmark dataset.
Revised: Besides, we test different [Table] [Table] manifold dimensions d ′ for each benchmark dataset.

Original: We keep d ′ as small as possible while ensuring reconstruction quality on the validation set.
Revised: We keep d ′ as small as possible while ensuring reconstruction quality on the validation set.

Original: We d ′ = 256 for CIFAR10 and CelebA, d ′ = 12 for
Revised: We d ′ = 256 for CIFAR10 and CelebA, d ′ = 12 for

Original: FEMNIST, d ′ = 32 for Adult and eICU.
Revised: FEMNIST, d ′ = 32 for Adult and eICU.

Original: For synthetic experiment, the data dimension d = 3 and manifold dimension d ′ = 2 since one element of data z identically equals to 0.
Revised: For synthetic experiment, the data dimension d = 3 and manifold dimension d ′ = 2 since one element of data z identically equals to 0.

Original: The source codes are made publically available at https://github.com/pcfl/pcfl .
Revised: The source codes are made publically available at https://github.com/pcfl/pcfl .

Original: D. D ATASETS
Revised: E. D ATASETS

Original: In our experiments, CIFAR10, FEMNIST, CelebA and Adult are all public dataset.
Revised: In our experiments, CIFAR10, FEMNIST, CelebA and Adult are all public dataset.

Original: For the synthetic experiment, the data point z
Revised: For the synthetic experiment, the data point z

Original: = { x, 0 , y } has three elements.
Revised: = { x, 0 , y } has three elements.

Original: We add a zero element to data so that the manifold dimension is smaller than the data dimension, which simulates the situation in real-world datasets.
Revised: We add a zero element to data so that the manifold dimension is smaller than the data dimension, which simulates the situation in real-world datasets.

Original: We create the federated environment with data heterogeneity for CIFAR10 and FEMNSIT by randomly allocating several classes to each client following the work (McMahan et al., 2017).
Revised: We create the federated environment with data heterogeneity for CIFAR10 and FEMNSIT by randomly allocating several classes to each client following the work (McMahan et al., 2017).

Original: For the dataset eICU, we follow the procedure on the website https://eicu-crd.mit.edu and got the approval for the dataset.
Revised: For the dataset eICU, we follow the procedure on the website https://eicu-crd.mit.edu and got the approval for the dataset.

Original: We follow the data preprocessing as in Sheikhalishahi et al.
Revised: We follow the data preprocessing as in Sheikhalishahi et al.

Original: (2019) and randomly select 14 hospitals as introduced in the main text.
Revised: (2019) and randomly select 14 hospitals as introduced in the main text.

Original: D. C OMPUTING R ESOURCES
Revised: E. C OMPUTING R ESOURCES

Original: Part of the experiments is conducted on a local server with Ubuntu 16.04 system.
Revised: Part of the experiments is conducted on a local server with Ubuntu 16.04 system.

Original: It has two physical
Revised: It has two physical

Original: CPU chips which are Intel(R)
Revised: CPU chips which are Intel(R)

Original: Xeon(R) CPU E5-2667 v4 @ 3.20GHz with 32 logical kernels.
Revised: Xeon(R) CPU E5-2667 v4 @ 3.20GHz with 32 logical kernels.

Original: The other experiments are conducted on a remote server.
Revised: The other experiments are conducted on a remote server.

Original: It has 8 GPUs which are GeForce RTX 2080 Ti.
Revised: It has 8 GPUs which are GeForce RTX 2080 Ti.