Original: T HE B EST OF B OTH W ORLDS : A CCURATE G LOBAL AND P ERSONALIZED M ODELS THROUGH F EDERATED L EARNING WITH D ATA -F REE H YPER -K NOWLEDGE
Revised: T HE B EST OF B OTH W ORLDS : A CCURATE G LOBAL AND P ERSONALIZED M ODELS THROUGH F EDERATED L EARNING WITH D ATA -F REE H YPER -K NOWLEDGE

Original: D ISTILLATION
Revised: D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Heterogeneity of data distributed across clients limits the performance of global models trained through federated learning, especially in the settings with highly imbalanced class distributions of local datasets.
Revised: Heterogeneity of data distributed across clients limits the performance of global models trained through federated learning, especially in the settings with highly imbalanced class distributions of local datasets.

Original: In recent years, personalized federated learning (pFL) has emerged as a potential solution to the challenges presented by heterogeneous data.
Revised: In recent years, personalized federated learning (pFL) has emerged as a potential solution to the challenges presented by heterogeneous data.

Original: However, existing pFL methods typically enhance performance of local models at the expense of the global model’s accuracy.
Revised: However, existing pFL methods typically enhance performance of local models at the expense of the global model’s accuracy.

Original: We propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL algorithm in which clients rely on knowledge distillation (KD) to train local models.
Revised: We propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL algorithm in which clients rely on knowledge distillation (KD) to train local models.

Original: In particular, each client extracts and sends to the server the means of local data representations and the corresponding soft predictions – information that we refer to as “hyper-knowledge”.
Revised: In particular, each client extracts and sends to the server the means of local data representations and the corresponding soft predictions – information that we refer to as “hyper-knowledge”.

Original: The server aggregates this information and broadcasts it to the clients in support of local training.
Revised: The server aggregates this information and broadcasts it to the clients in support of local training.

Original: Notably, unlike other KD-based pFL methods, FedHKD does not rely on a public dataset nor it deploys a generative model at the server.
Revised: Notably, unlike other KD-based pFL methods, FedHKD does not rely on a public dataset nor it deploys a generative model at the server.

Original: We analyze convergence of FedHKD and conduct extensive experiments on visual datasets in a variety of scenarios, demonstrating that
Revised: We analyze convergence of FedHKD and conduct extensive experiments on visual datasets in a variety of scenarios, demonstrating that

Original: FedHKD provides significant improvement in both personalized as well as global model performance compared to state-of-the-art FL methods designed for heterogeneous data settings.
Revised: FedHKD provides significant improvement in both personalized as well as global model performance compared to state-of-the-art FL methods designed for heterogeneous data settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL), a communication-efficient and privacy-preserving alternative to training on centrally aggregated data, relies on collaboration between clients who own local data to train a global machine learning model.
Revised: Federated learning (FL), a communication-efficient and privacy-preserving alternative to training on centrally aggregated data, relies on collaboration between clients who own local data to train a global machine learning model.

Original: A central server coordinates the training without violating clients’ privacy – the server has no access to the clients’ local data.
Revised: A central server coordinates the training without violating clients’ privacy – the server has no access to the clients’ local data.

Original: The first ever such scheme, Federated Averaging (FedAvg) (McMahan et al., 2017), alternates between two steps: (1) randomly selected client devices initialize their local models with the global model received from the server, and proceed to train on local data; (2) the server collects local model updates and aggregates them via weighted averaging to form a new global model.
Revised: The first ever such scheme, Federated Averaging (FedAvg) (McMahan et al., 2017), alternates between two steps: (1) randomly selected client devices initialize their local models with the global model received from the server, and proceed to train on local data; (2) the server collects local model updates and aggregates them via weighted averaging to form a new global model.

Original: As analytically shown in (McMahan et al., 2017), FedAvg is guaranteed to converge when the client data is independent and identically distributed (iid).
Revised: As analytically shown in (McMahan et al., 2017), FedAvg is guaranteed to converge when the client data is independent and identically distributed (iid).

Original: A major problem in FL systems emerges when the clients’ data is heterogeneous (Kairouz et al., 2021).
Revised: A major problem in FL systems emerges when the clients’ data is heterogeneous (Kairouz et al., 2021).

Original: This is a common setting in practice since the data owned by clients participating in federated learning is likely to have originated from different distributions.
Revised: This is a common setting in practice since the data owned by clients participating in federated learning is likely to have originated from different distributions.

Original: In such settings, the FL procedure may converge slowly and the resulting global model may perform poorly on the local data of an individual client.
Revised: In such settings, the FL procedure may converge slowly and the resulting global model may perform poorly on the local data of an individual client.

Original: To address this challenge, a number of FL methods aiming to enable learning on non-iid data has recently been proposed (Karimireddy et al., 2019; Li et al., 2020; 2021a;
Revised: To address this challenge, a number of FL methods aiming to enable learning on non-iid data has recently been proposed (Karimireddy et al., 2019; Li et al., 2020; 2021a;

Original: Acar et al., 2021; Liu et al., 2021; Yoon et al., 2021; Chen & Vikalo, 2022).
Revised: Acar et al., 2021; Liu et al., 2021; Yoon et al., 2021; Chen & Vikalo, 2022).

Original: Unfortunately, these methods struggle to train a global model that performs well when the clients’ data distributions differ significantly.
Revised: Unfortunately, these methods struggle to train a global model that performs well when the clients’ data distributions differ significantly.

Original: Difficulties of learning on non-iid data, as well as the heterogeneity of the clients’ resources (e.g., compute, communication, memory, power), motivated a variety of personalized FL (pFL) techniques (Arivazhagan et al., 2019; T Dinh et al., 2020; Zhang et al., 2020; Huang et al., 2021; Collins et al.,
Revised: Difficulties of learning on non-iid data, as well as the heterogeneity of the clients’ resources (e.g., compute, communication, memory, power), motivated a variety of personalized FL (pFL) techniques (Arivazhagan et al., 2019; T Dinh et al., 2020; Zhang et al., 2020; Huang et al., 2021; Collins et al.,

Original: 2021; Tan et al., 2022).
Revised: 2021; Tan et al., 2022).

Original: In a pFL system, each client leverages information received from the server and utilizes a customized objective to locally train its personalized model.
Revised: In a pFL system, each client leverages information received from the server and utilizes a customized objective to locally train its personalized model.

Original: Instead of focusing on global performance, a pFL client is concerned with improving the model’s local performance empirically evaluated by running the local model on data having distribution similar to the distribution of local training data.
Revised: Instead of focusing on global performance, a pFL client is concerned with improving the model’s local performance empirically evaluated by running the local model on data having distribution similar to the distribution of local training data.

Original: Since most personalized FL schemes remain reliant upon on gradient or model aggregation, they are highly susceptible to ’stragglers’ that slow down the training convergence process.
Revised: Since most personalized FL schemes remain reliant upon on gradient or model aggregation, they are highly susceptible to ’stragglers’ that slow down the training convergence process.

Original: FedProto (Tan et al., 2021) is proposed to address high communication cost and limitations of homogeneous models in federated learning.
Revised: FedProto (Tan et al., 2021) is proposed to address high communication cost and limitations of homogeneous models in federated learning.

Original: Instead of model parameters, in FedProto each client sends to the server only the class prototypes – the means of the representations of the samples in each class.
Revised: Instead of model parameters, in FedProto each client sends to the server only the class prototypes – the means of the representations of the samples in each class.

Original: Aggregating the prototypes rather than model updates significantly reduces communication costs and lifts the requirement of FedAvg that clients must deploy the same model architecture.
Revised: Aggregating the prototypes rather than model updates significantly reduces communication costs and lifts the requirement of FedAvg that clients must deploy the same model architecture.

Original: However, note that even though FedProto improves local validation accuracy by utilizing aggregated class prototypes, it leads to barely any improvement in the global performance.
Revised: However, note that even though FedProto improves local validation accuracy by utilizing aggregated class prototypes, it leads to barely any improvement in the global performance.

Original: Motivated by the success of Knowledge Distillation (KD) (Hinton et al., 2015) which infers soft predictions of samples as the ’knowledge’ extracted from a neural network, a number of FL methods that aim to improve global model’s generalization ability has been proposed (Jeong et al., 2018b; Li & Wang, 2019;
Revised: Motivated by the success of Knowledge Distillation (KD) (Hinton et al., 2015) which infers soft predictions of samples as the ’knowledge’ extracted from a neural network, a number of FL methods that aim to improve global model’s generalization ability has been proposed (Jeong et al., 2018b; Li & Wang, 2019;

Original: Lin et al., 2020; Zhang et al., 2021).
Revised: Lin et al., 2020; Zhang et al., 2021).

Original: However, most of the existing KD-based FL methods require that a public dataset is provided to all clients, limiting the feasibility of these methods in practical settings.
Revised: However, most of the existing KD-based FL methods require that a public dataset is provided to all clients, limiting the feasibility of these methods in practical settings.

Original: In this paper we propose FedHKD (F ederated Hyper-Knowledge Distillation), a novel FL framework that relies on prototype learning and knowledge distillation to facilitate training on heterogeneous data.
Revised: In this paper we propose FedHKD (F ederated Hyper-Knowledge Distillation), a novel FL framework that relies on prototype learning and knowledge distillation to facilitate training on heterogeneous data.

Original: Specifically, the clients in FedHKD compute mean representations and the corresponding mean soft predictions for the data classes in their local training sets; this information, which we refer to as “hyper-knowledge,” is endued by differential privacy via the Gaussian mechanism and sent for aggregation to the server.
Revised: Specifically, the clients in FedHKD compute mean representations and the corresponding mean soft predictions for the data classes in their local training sets; this information, which we refer to as “hyper-knowledge,” is endued by differential privacy via the Gaussian mechanism and sent for aggregation to the server.

Original: The resulting globally aggregated hyper-knowledge is used by clients in the subsequent training epoch and helps lead to better personalized and global performance.
Revised: The resulting globally aggregated hyper-knowledge is used by clients in the subsequent training epoch and helps lead to better personalized and global performance.

Original: A number of experiments on classification tasks involving SVHN (Netzer et al., 2011), CIFAR10 and
Revised: A number of experiments on classification tasks involving SVHN (Netzer et al., 2011), CIFAR10 and

Original: CIFAR100 datasets demonstrate that FedHKD consistently outperforms state-of-the-art approaches in terms of both local and global accuracy.
Revised: CIFAR100 datasets demonstrate that FedHKD consistently outperforms state-of-the-art approaches in terms of both local and global accuracy.

Original:  R ELATED W ORK 2. H ETEROGENEOUS F EDERATED L EARNING
Revised:  R ELATED W ORK 2. H ETEROGENEOUS F EDERATED L EARNING

Original: Majority of the existing work on federated learning across data-heterogeneous clients can be organized in three categories.
Revised: Majority of the existing work on federated learning across data-heterogeneous clients can be organized in three categories.

Original: The first set of such methods aims to reduce variance of local training by introducing regularization terms in local objective (Karimireddy et al., 2019;
Revised: The first set of such methods aims to reduce variance of local training by introducing regularization terms in local objective (Karimireddy et al., 2019;

Original: Li et al., 2020; 2021a; Acar et al., 2021).
Revised: Li et al., 2020; 2021a; Acar et al., 2021).

Original: Mendieta et al. (2022) analyze regularization-based FL algorithms and, motivated by the regularization technique GradAug in centralized learning (Yang et al., 2020), propose FedAlign.
Revised: (Mendieta et al., 2022) analyze regularization-based FL algorithms and, motivated by the regularization technique GradAug in centralized learning (Yang et al., 2020), propose FedAlign.

Original: Another set of techniques for FL on heterogeneous client data aims to replace the naive model update averaging strategy of FedAvg by more efficient aggregation schemes.
Revised: Another set of techniques for FL on heterogeneous client data aims to replace the naive model update averaging strategy of FedAvg by more efficient aggregation schemes.

Original: To this end, PFNM (Yurochkin et al., 2019) applies a Bayesian non-parametric method to select and merge multi-layer perceptron (MLP) layers from local models into a more expressive global model in a layer-wise manner.
Revised: To this end, PFNM (Yurochkin et al., 2019) applies a Bayesian non-parametric method to select and merge multi-layer perceptron (MLP) layers from local models into a more expressive global model in a layer-wise manner.

Original: FedMA (Wang et al. (2020a)) proceeds further in this direction and extends the same principle to CNNs and LSTMs.
Revised: FedMA ((Wang et al., 2020a)) proceeds further in this direction and extends the same principle to CNNs and LSTMs.

Original: Wang et al. (2020b) analyze convergence of heterogeneous federated learning and propose a novel normalized averaging method.
Revised: (Wang et al., 2020b) analyze convergence of heterogeneous federated learning and propose a novel normalized averaging method.

Original: Finally, the third set of methods utilize either the mixup mechanism (Zhang et al., 2017) or generative models to enrich diversity of local datasets (Yoon et al., 2021; Liu et al., 2021; Chen & Vikalo, 2022).
Revised: Finally, the third set of methods utilize either the mixup mechanism (Zhang et al., 2017) or generative models to enrich diversity of local datasets (Yoon et al., 2021; Liu et al., 2021; Chen & Vikalo, 2022).

Original: However, these methods introduce additional memory/computation costs and increase the required communication resources.
Revised: However, these methods introduce additional memory/computation costs and increase the required communication resources.

Original: 2. P ERSONALIZED F EDERATED L EARNING
Revised: 2. P ERSONALIZED F EDERATED L EARNING

Original: Motivated by the observation that a global model collaboratively trained on highly heterogeneous data may not generalize well on clients’ local data, a number of personalized federated learning (pFL) techniques aiming to train customized local models have been proposed (Tan et al., 2022).
Revised: Motivated by the observation that a global model collaboratively trained on highly heterogeneous data may not generalize well on clients’ local data, a number of personalized federated learning (pFL) techniques aiming to train customized local models have been proposed (Tan et al., 2022).

Original: They can be categorized into two groups depending on whether or not they also train a global model.
Revised: They can be categorized into two groups depending on whether or not they also train a global model.

Original: The pFL techniques focused on global model personalization follow a procedure similar to the plain vanilla FL – clients still need to upload all or a subset of model parameters to the server to enable global model aggregation.
Revised: The pFL techniques focused on global model personalization follow a procedure similar to the plain vanilla FL – clients still need to upload all or a subset of model parameters to the server to enable global model aggregation.

Original: The global model is personalized by each client via local adaptation steps such as fine-tuning (Wang et al., 2019; Hanzely et al., 2020; Schneider & Vlachos, 2021), creating a mixture of global and local layers (Arivazhagan et al., 2019; Mansour et al., 2020;
Revised: The global model is personalized by each client via local adaptation steps such as fine-tuning (Wang et al., 2019; Hanzely et al., 2020; Schneider & Vlachos, 2021), creating a mixture of global and local layers (Arivazhagan et al., 2019; Mansour et al., 2020;

Original: Deng et al., 2020; Zec et al., 2020; Hanzely & Richt´arik, 2020; Collins et al.,  2021), regularization (T Dinh et al., 2020; Li et al., 2021b) and meta learning (Jiang et al., 2019; Fallah et al., 2020).
Revised: Deng et al., 2020; Zec et al., 2020; Hanzely & Richt´arik, 2020; Collins et al., 2021; Chen & Chao, 2021), regularization (T Dinh et al., 2020; Li et al., 2021b) and meta learning (Jiang et al., 2019; Fallah et al., 2020).

Original: However, when the resources available to different clients vary, it is impractical to require that all clients train models of the same size and type.
Revised: However, when the resources available to different clients vary, it is impractical to require that all clients train models of the same size and type.

Original: To address this, some works waive the global model by adopting multi-task learning (Smith et al., 2017) or hyper-network frameworks (Shamsian et al., 2021).
Revised: To address this, some works waive the global model by adopting multi-task learning (Smith et al., 2017) or hyper-network frameworks (Shamsian et al., 2021).

Original: Inspired by prototype learning (Snell et al., 2017; Hoang et al., 2020; Michieli & Ozay, 2021), FedProto (Tan et al., 2021) utilizes aggregated class prototypes received from the server to align clients’ local objectives via a regularization term; since there is no transmission of model parameters between clients and the server, this scheme requires relatively low communication resources.
Revised: Inspired by prototype learning (Snell et al., 2017; Hoang et al., 2020; Michieli & Ozay, 2021), FedProto (Tan et al., 2021) utilizes aggregated class prototypes received from the server to align clients’ local objectives via a regularization term; since there is no transmission of model parameters between clients and the server, this scheme requires relatively low communication resources.

Original: Although FedProto improves local test accuracy of the personalized models, it does not benefit the global performance.
Revised: Although FedProto improves local test accuracy of the personalized models, it does not benefit the global performance.

Original: 2. F EDERATED LEARNING WITH K NOWLEDGE D ISTILLATION
Revised: 2. F EDERATED LEARNING WITH K NOWLEDGE D ISTILLATION

Original: Knowledge Distillation (KD) Hinton et al. (2015), a technique capable of extracting knowledge from a neural network by exchanging soft predictions instead of the entire model, has been introduced to federated learning to aid with the issues that arise due to variations in resources (computation, communication and memory) available to the clients (Jeong et al., 2018a;
Revised: Knowledge Distillation (KD) (Hinton et al., 2015), a technique capable of extracting knowledge from a neural network by exchanging soft predictions instead of the entire model, has been introduced to federated learning to aid with the issues that arise due to variations in resources (computation, communication and memory) available to the clients (Jeong et al., 2018a;

Original: Chang et al., 2019; Itahara et al., 2020).
Revised: Chang et al., 2019; Itahara et al., 2020).

Original: FedMD (Li & Wang, 2019), FedDF (Lin et al., 2020) and FedKT-pFL (Zhang et al., 2021) transmit only soft-predictions as the knowledge between the server and clients, allowing for personalized/heterogeneous client models.
Revised: FedMD (Li & Wang, 2019), FedDF (Lin et al., 2020) and FedKTpFL (Zhang et al., 2021) transmit only soft-predictions as the knowledge between the server and clients, allowing for personalized/heterogeneous client models.

Original: However, these KD-based federated learning methods require that a public dataset is made available to all clients, presenting potential practical challenges.
Revised: However, these KD-based federated learning methods require that a public dataset is made available to all clients, presenting potential practical challenges.

Original: Recent studies (Zhu et al., 2021; Zhang et al., 2022) explored using GANs (Goodfellow et al., 2014) to enable data-free federated knowledge distillation in the context of image classification tasks; however, training GANs incurs considerable additional computation and memory requirements.
Revised: Recent studies (Zhu et al., 2021; Zhang et al., 2022) explored using GANs (Goodfellow et al., 2014) to enable data-free federated knowledge distillation in the context of image classification tasks; however, training GANs incurs considerable additional computation and memory requirements.

Original: In summary, most of the existing KD-based schemes require a shared dataset to help align local models; others require costly computational efforts to synthesize artificial data or deploy a student model at the server and update it using local gradients computed when minimizing the divergence of soft prediction on local data between clients’ teacher model and the student model (Lin et al., 2020).
Revised: In summary, most of the existing KD-based schemes require a shared dataset to help align local models; others require costly computational efforts to synthesize artificial data or deploy a student model at the server and update it using local gradients computed when minimizing the divergence of soft prediction on local data between clients’ teacher model and the student model (Lin et al., 2020).

Original: In our framework, we extend the concept of knowledge to ’hyper-knowledge’, combining class prototypes and soft predictions on local data to improve both the local test accuracy and global generalization ability of federated learning.
Revised: In our framework, we extend the concept of knowledge to ’hyper-knowledge’, combining class prototypes and soft predictions on local data to improve both the local test accuracy and global generalization ability of federated learning.

Original:  M ETHODOLOGY 3. P ROBLEM F ORMULATION
Revised:  M ETHODOLOGY 3. P ROBLEM F ORMULATION

Original: Consider a federated learning system where m clients own local private dataset D 1 , . .
Revised: Consider a federated learning system where m clients own local private dataset D 1 , . .

Original: , D m ; the distributions of the datasets may vary across clients, including the scenario in which a local dataset contains samples from only a fraction of classes.
Revised: , D m ; the distributions of the datasets may vary across clients, including the scenario in which a local dataset contains samples from only a fraction of classes.

Original: In such an FL system, the clients communicate locally trained models to the server which, in turn, sends the aggregated global model back to the clients.
Revised: In such an FL system, the clients communicate locally trained models to the server which, in turn, sends the aggregated global model back to the clients.

Original: The plain vanilla federated learning (McMahan et al., 2017)
Revised: The plain vanilla federated learning (McMahan et al., 2017)

Original: implements aggregation as
Revised: implements aggregation as

Original: [Equation]
Revised: [Equation]

Original: where w t denotes parameters of the global model at round t ; w t − 1 i denotes parameters of the local model of client i at round t − 1 ; m is the number of participating clients; and M = (cid:80) mi =1 |D i | .
Revised: where w t denotes parameters of the global model at round t ; w t − 1 i denotes parameters of the local model of client i at round t − 1 ; m is the number of participating clients; and M = (cid:80) mi =1 |D i | .

Original: The clients are typically assumed to share the same model architecture.
Revised: The clients are typically assumed to share the same model architecture.

Original: Our aim is to learn a personalized model w i for each client i which not only performs well on data generated from the distribution of the i th client’s local training data, but can further be aggregated into a global model w that performs well across all data classes (i.e., enable accurate global model performance).
Revised: Our aim is to learn a personalized model w i for each client i which not only performs well on data generated from the distribution of the i th client’s local training data, but can further be aggregated into a global model w that performs well across all data classes (i.e., enable accurate global model performance).

Original: This is especially difficult when the data is heterogenous since straightforward aggregation in such scenarios likely leads to inadequate performance of the global model.
Revised: This is especially difficult when the data is heterogenous since straightforward aggregation in such scenarios likely leads to inadequate performance of the global model.

Original: 3. U TILIZING H YPER -K NOWLEDGE
Revised: 3. U TILIZING H YPER -K NOWLEDGE

Original: Knowledge distillation (KD) based federated learning methods that rely on a public dataset require clients to deploy local models to run inference / make predictions for the samples in the public dataset; the models’ outputs are then used to form soft predictions according to
Revised: Knowledge distillation (KD) based federated learning methods that rely on a public dataset require clients to deploy local models to run inference / make predictions for the samples in the public dataset; the models’ outputs are then used to form soft predictions according to

Original: [Equation]
Revised: [Equation]

Original: where z i denotes the i th element in the model’s output z for a given data sample; q i is the i th element in the soft prediction q ; and T is the so-called ”temperature” parameter.
Revised: where z i denotes the i th element in the model’s output z for a given data sample; q i is the i th element in the soft prediction q ; and T is the so-called ”temperature” parameter.

Original: The server collects soft predictions from clients (local knowledge), aggregates them into global soft predictions (global knowledge), and sends them to clients to be used in the next training round.
Revised: The server collects soft predictions from clients (local knowledge), aggregates them into global soft predictions (global knowledge), and sends them to clients to be used in the next training round.

Original: Performing inference on the public dataset introduces additional computations in each round of federated learning, while sharing and locally storing public datasets consumes communication and memory resources.
Revised: Performing inference on the public dataset introduces additional computations in each round of federated learning, while sharing and locally storing public datasets consumes communication and memory resources.

Original: It would therefore be beneficial to develop KD-based methods that do not require use of public datasets; synthesizing artificial data is an option, but one that is computationally costly and thus may be impractical.
Revised: It would therefore be beneficial to develop KD-based methods that do not require use of public datasets; synthesizing artificial data is an option, but one that is computationally costly and thus may be impractical.

Original: To this end, we extend the notion of distilled knowledge to include both the averaged representations and the corresponding averaged soft predictions, and refer to it as “hyperknowledge”; the “hyper-knowledge” is protected via the Gaussian differential privacy mechanism and shared between clients and server.
Revised: To this end, we extend the notion of distilled knowledge to include both the averaged representations and the corresponding averaged soft predictions, and refer to it as “hyperknowledge”; the “hyper-knowledge” is protected via the Gaussian differential privacy mechanism and shared between clients and server.

Original: Feature Extractor and Classifier.
Revised: Feature Extractor and Classifier.

Original: We consider image classification as an illustrative use case.
Revised: We consider image classification as an illustrative use case.

Original: Typically, a deep network for classification tasks consists of two parts (Kang et al., 2019): (1) a feature extractor translating the input raw data (i.e., an image) into latent space representation; (2) a classifier mapping representations into categorical vectors.
Revised: Typically, a deep network for classification tasks consists of two parts (Kang et al., 2019): (1) a feature extractor translating the input raw data (i.e., an image) into latent space representation; (2) a classifier mapping representations into categorical vectors.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where x i denotes raw data of client i , R ϕ
Revised: where x i denotes raw data of client i , R ϕ

Original: i ( · ) and G ω i ( · ) are the embedding functions of feature extractor and classifier with model parameters
Revised: i ( · ) and G ω i ( · ) are the embedding functions of feature extractor and classifier with model parameters

Original: ϕ
Revised: ϕ

Original: i and ω i , respectively; h i is the representation vector of x i ; and z i is the categorical vector.
Revised: i and ω i , respectively; h i is the representation vector of x i ; and z i is the categorical vector.

Original: Evaluating and Using Hyper-Knowledge.
Revised: Evaluating and Using Hyper-Knowledge.

Original: The mean latent representation of class j in the local dataset of client i is computed as
Revised: The mean latent representation of class j in the local dataset of client i is computed as

Original: [Equation]
Revised: [Equation]

Original: k [Equation] (4) where N ji is the number of samples with label j in client i ’s dataset; Q ( · , T ) is the soft target function; h j,ki and z j,ki are the data representation and prediction of the i th client’s k th sample with label j .
Revised: where N ji is the number of samples with label j in client i ’s dataset; Q ( · , T ) is the soft target function; h j,ki and z j,ki are the data representation and prediction of the i th client’s k th sample with label j .

Original: The mean latent data representation ¯ h ji and soft prediction ¯ q ji are the hyper-knowledge of class j in client i ; for convenience, we denote K ji
Revised: The mean latent data representation ¯ h ji and soft prediction ¯ q ji are the hyper-knowledge of class j in client i ; for convenience, we denote K ji

Original: = ( ¯ h ji , ¯ q ji ) .
Revised: = ( ¯ h ji , ¯ q ji ) .

Original: If there are n classes, then the full hyper-knowledge of client i is K i
Revised: If there are n classes, then the full hyper-knowledge of client i is K i

Original: = {K 1 i , . .
Revised: = {K 1 i , . .

Original: , K ni } .
Revised: , K ni } .

Original: As a comparison, FedProto Tan et al. (2021) only utilizes means of data representations and makes no use of soft predictions.
Revised: As a comparison, FedProto (Tan et al., 2021) only utilizes means of data representations and makes no use of soft predictions.

Original: Note that to avoid the situations where K ji = ∅ , which may happen when data is highly heterogeneous, FedHKD sets a threshold (tunable hyper-parameter) ν which is used to decided whether or not a client should share its hyper-knowledge; in particular, if the fraction of samples with label j in the local dataset of client i is below ν , client i is not allowed to share the hyper-knowledge K ji .
Revised: Note that to avoid the situations where K ji = ∅ , which may happen when data is highly heterogeneous, FedHKD sets a threshold (tunable hyper-parameter) ν which is used to decided whether or not a client should share its hyper-knowledge; in particular, if the fraction of samples with label j in the local dataset of client i is below ν , client i is not allowed to share the hyper-knowledge K ji .

Original: If there is no participating client sharing hyper-knowledge for class j , the server sets K j = ∅ .
Revised: If there is no participating client sharing hyper-knowledge for class j , the server sets K j = ∅ .

Revised: A flow diagram illustrating the computation of hyper-knowledge is given in Appendix.

Revised: A.3.

Original: Differential Privacy Mechanism.
Revised: Differential Privacy Mechanism.

Original: It has previously been argued that communicating averaged data representation promotes privacy (Tan et al., 2021); however, hyper-knowledge exchanged between server and clients may still be exposed to differential attacks (Dwork, 2008; Geyer et al., 2017).
Revised: It has previously been argued that communicating averaged data representation promotes privacy (Tan et al., 2021); however, hyper-knowledge exchanged between server and clients may still be exposed to differential attacks (Dwork, 2008; Geyer et al., 2017).

Original: A number of studies (Geyer et al., 2017; Sun et al., 2021; Gong et al., 2021; Ribero et al., 2022; Chen & Vikalo, 2022) that utilize differential privacy to address security concerns in federated learning have been proposed.
Revised: A number of studies (Geyer et al., 2017; Sun et al., 2021; Gong et al., 2021; Ribero et al., 2022; Chen & Vikalo, 2022) that utilize differential privacy to address security concerns in federated learning have been proposed.

Original: The scheme presented in this paper promotes privacy by protecting the shared means of data representations through a differential privacy (DP) mechanism (Dwork et al., 2006a;b) defined below.
Revised: The scheme presented in this paper promotes privacy by protecting the shared means of data representations through a differential privacy (DP) mechanism (Dwork et al., 2006a;b) defined below.

Original: Definition 1 ( ( ε, δ ) -Differential
Revised: Definition 1 ( ( ε, δ ) -Differential

Original: Privacy) A randomized function F : D → R provides ( ε, δ ) - differential privacy if for all adjacent datasets d , d ′ ∈ D differing on at most one element, and all S ∈ range ( F ) , it holds that
Revised: Privacy) A randomized function F : D → R provides ( ε, δ ) - differential privacy if for all adjacent datasets d , d ′ ∈ D differing on at most one element, and all S ∈ range ( F ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: where ϵ denotes the maximum distance between the range of F ( d ) and F ( d ′ ) and may be thought of as the allotted privacy budget, while δ is the probability that the maximum distance is not bounded by ε .
Revised: where ϵ denotes the maximum distance between the range of F ( d ) and F ( d ′ ) and may be thought of as the allotted privacy budget, while δ is the probability that the maximum distance is not bounded by ε .

Original: Any deterministic function f : D → R can be endued with arbitrary ( ϵ, δ ) -differential privacy via the Gaussian mechanism, defined next.
Revised: Any deterministic function f : D → R can be endued with arbitrary ( ϵ, δ ) -differential privacy via the Gaussian mechanism, defined next.

Original: Theorem 1 (Gaussian mechanism)
Revised: Theorem 1 (Gaussian mechanism)

Original: A randomized function F derived from any deterministic function f : D → R perturbed by Gaussian noise N 
Revised: A randomized function F derived from any deterministic function f : D → R perturbed by Gaussian noise N (0 , S 2 f · σ 2 ) ,

Original: [Equation]
Revised: [Equation]

Original:  achieves ( ε, δ ) -differential privacy for any σ > (cid:113) 2 log 54 δ /ε .
Revised: ) achieves ( ε, δ ) -differential privacy for any σ > (cid:113) 2 log 54 δ /ε .

Original: Here S f denotes the sensitivity of function f defined as the maximum of the absolute distance | f ( d ) − f ( d ′ ) | .
Revised: Here S f denotes the sensitivity of function f defined as the maximum of the absolute distance | f ( d ) − f ( d ′ ) | .

Original: We proceed by defining a deterministic function f l ( d ji ) ≜ ¯ h ji ( l ) = 1 N ji (cid:80) N ji k =1 h j,ki ( l ) which evaluates the l th element of ¯ h ji , where d ji is the subset of client i ’s local dataset including samples with label j only; h j,ki denotes the representation of the k th sample in d ji while h j,ki ( l ) is the l th element of h j,ki .
Revised: We proceed by defining a deterministic function f l ( d ji ) ≜ ¯ h ji ( l ) = 1 N ji (cid:80) N ji k =1 h j,ki ( l ) which evaluates the l th element of ¯ h ji , where d ji is the subset of client i ’s local dataset including samples with label j only; h j,ki denotes the representation of the k th sample in d ji while h j,ki ( l ) is the l th element of h j,ki .

Original: In our proposed framework, client i transmits noisy version of its hyper-knowledge to the server,
Revised: In our proposed framework, client i transmits noisy version of its hyper-knowledge to the server,

Original: [Equation]
Revised: [Equation]

Original: where χ ji ( l ) ∼ N (0 , ( S if ) 2 · σ 2 ) ; σ 2 denotes a hyper-parameter shared by all clients.
Revised: where χ ji ( l ) ∼ N (0 , ( S if ) 2 · σ 2 ) ; σ 2 denotes a hyper-parameter shared by all clients.

Original: ( S if ) 2 is the sensitive of function f l ( · ) with client i ’s local dataset.
Revised: ( S if ) 2 is the sensitive of function f l ( · ) with client i ’s local dataset.

Original: Lemma 

Revised: Lemma

Original: If | h j,ki ( l ) | is bounded by ζ > 0 for any k , then
Revised: If | h j,ki ( l ) | is bounded by ζ > 0 for any k , then

Original: [Equation]
Revised: [Equation]

Original: Therefore, S if = 2 ζ N ji .
Revised: Therefore, S if = 2 ζ N ji .

Original: Note that ( S if ) 2 depends on N ji , the number of samples in class j , and thus differs across clients in the heterogeneous setting.
Revised: Note that ( S if ) 2 depends on N ji , the number of samples in class j , and thus differs across clients in the heterogeneous setting.

Original: A discussion on the probability that differential privacy is broken can be found in the Section 4.3.
Revised: A discussion on the probability that differential privacy is broken can be found in the Section 4.3.

Original: Proof of Lemma 1 is provided in Appendix A.3.
Revised: Proof of Lemma 1 is provided in Appendix A.5.

Original: 3.  G LOBAL H YPER -K NOWLEDGE A GGREGATION
Revised:  G LOBAL H YPER -K NOWLEDGE A GGREGATION

Original: After the server collects hyper-knowledge from participating clients, the global hyper-knowledge for class j at global round t + 1 , K j,t +1 = (cid:0) H j,t +1 , Q j,t +1 (cid:1) , is formed as
Revised: After the server collects hyper-knowledge from participating clients, the global hyper-knowledge for class j at global round t + 1 , K j,t +1 = (cid:0) H j,t +1 , Q j,t +1 (cid:1) , is formed as

Original: [Equation]
Revised: [Equation]

Original: j [Equation] t
Revised: where p i = N ji /N

Original: [Equation] where p i = N ji /N j , N ji denotes the number of samples in class j owned by client i , and N j = (cid:80) mi
Revised: j , N ji denotes the number of samples in class j owned by client i , and N j = (cid:80) mi

Original: =1 N ji .
Revised: =1 N ji .

Original: For clarity, we emphasize that ˜ h
Revised: For clarity, we emphasize that ˜ h

Original: j,ti denotes the local hyper-knowledge about class j of client i at global round t .
Revised: j,ti denotes the local hyper-knowledge about class j of client i at global round t .

Original: Since the noise is drawn from N (cid:16) 0 , ( S if ) 2 · σ 2 (cid:17) , its effect on the quality of hyper-knowledge is alleviated during aggregation assuming sufficiently large number of participating clients, i.e.,
Revised: Since the noise is drawn from N (cid:16) 0 , ( S if ) 2 · σ 2 (cid:17) , its effect on the quality of hyper-knowledge is alleviated during aggregation assuming sufficiently large number of participating clients, i.e.,

Original: [Equation]
Revised: [Equation]

Original: with variance σ 2 (cid:80) mi =1 ( S if ) 2 .
Revised: with variance σ 2 (cid:80) mi =1 ( S if ) 2 .

Original: In other words, the additive noise is “averaged out” and effectively near-eliminated after aggregating local hyper-knowledge.
Revised: In other words, the additive noise is “averaged out” and effectively near-eliminated after aggregating local hyper-knowledge.

Original: For simplicity, we assume that in the above expressions N ji ̸ = 0 .
Revised: For simplicity, we assume that in the above expressions N ji ̸ = 0 .

Original: 3. L OCAL T RAINING O BJECTIVE
Revised: 3. L OCAL T RAINING O BJECTIVE

Original: Following the aggregation at the server, the global hyper-knowledge is sent to the clients participating in the next FL round to assist in local training.
Revised: Following the aggregation at the server, the global hyper-knowledge is sent to the clients participating in the next FL round to assist in local training.

Original: In particular, given data samples ( x , y ) ∼ D i , the loss function of client i is formed as
Revised: In particular, given data samples ( x , y ) ∼ D i , the loss function of client i is formed as

Original: [Equation]
Revised: [Equation]

Original: where B i denotes the number of samples in the dataset owned by client i , n is the number of classes,
Revised: where B i denotes the number of samples in the dataset owned by client i , n is the number of classes,

Original: CELoss ( · , · ) denotes the cross-entropy loss function, ∥ · ∥ 2 denotes Euclidean norm, Q ( · , T ) is the soft target function with temperature T , and λ and γ are hyper-parameters.
Revised: CELoss ( · , · ) denotes the cross-entropy loss function, ∥ · ∥ 2 denotes Euclidean norm, Q ( · , T ) is the soft target function with temperature T , and λ and γ are hyper-parameters.

Revised: Note that the loss function in (11) consists of three terms: the empirical risk formed using predictions and ground-truth labels, and two regularization terms utilizing hyper-knowledge.

Revised: Essentially, the second and third terms in the loss function are proximity/distance functions.

Revised: The second term is to force the local classifier to output similar soft predictions when given global data representations while the third term is to force the features extractor to output similar data representations when given local data samples.

Revised: For both, we use Euclidean distance because it is non-negative and convex.

Original: 3. F ED HKD: S UMMARY OF THE F RAMEWORK
Revised: 3. F ED HKD: S UMMARY OF THE F RAMEWORK

Original: The training starts at the server by initializing the global model θ
Revised: The training starts at the server by initializing the global model θ

Original: 1 = ( ϕ 1 , ω 1 ) , where ϕ 1 and ωdenote parameters of the global feature extractor and global classifier, respectively.
Revised: 1 = ( ϕ 1 , ω 1 ) , where ϕ 1 and ωdenote parameters of the global feature extractor and global classifier, respectively.

Original: At the beginning of each global epoch, the server sends the global model and global hyper-knowledge to clients selected for training.
Revised: At the beginning of each global epoch, the server sends the global model and global hyper-knowledge to clients selected for training.

Original: In turn, each client initializes its local model with the received global model, and performs updates by minimizing the objective in Eq.
Revised: In turn, each client initializes its local model with the received global model, and performs updates by minimizing the objective in Eq.

Original: 11; the objective consists of three terms: (1) prediction loss in a form of the cross-entropy between prediction and ground-truth; (2) classifier loss reflective of the Euclidean norm distance between the output of the classifier and the corresponding global soft predictions; and (3) feature loss given by the Euclidean norm distance between representations extracted from raw data by a local feature extractor and global data representations.
Revised: 11; the objective consists of three terms: (1) prediction loss in a form of the cross-entropy between prediction and ground-truth; (2) classifier loss reflective of the Euclidean norm distance between the output of the classifier and the corresponding global soft predictions; and (3) feature loss given by the Euclidean norm distance between representations extracted from raw data by a local feature extractor and global data representations.

Original: Having completed local updates, clients complement their local hyper-knowledge by performing inference on local data, and finally send local model as well as local hyper-knowledge to the server for aggregation.
Revised: Having completed local updates, clients complement their local hyper-knowledge by performing inference on local data, and finally send local model as well as local hyper-knowledge to the server for aggregation.

Original: The method outlined in this section is formalized as Algorithm 1.Algorithm
Revised: The method outlined in this section is formalized as Algorithm 1.

Original: Input: Datasets distributed across m clients, D = {D 1 , D 2 , . . .
Revised: For convenience, we provided a visualization of the FedHKD procedure in Appendix.

Original: D m } ; client participating rate µ ; hyper-parameters λ and γ ; the sharing threshold ν ; variance σ 2 characterizing differential privacy noise; temperature T ; the number of global epochs T r .
Revised: A.4.

Original: Output:
Revised: [Table]

Original: The global model θ T r +1 =
Revised: global model θ T r +1 =

Original: ( ϕ T r +1 , ω T r +1 ) 1: Server executes: 2: randomly initialize ( ϕ 1 , ω 1 ) , K = {} 3: for t = 1 , . . .
Revised: ( ϕ T r +1 , ω T r +1 ) 1: Server executes: 2: randomly initialize ( ϕ 1 , ω 1 ) , K = {} 3: for t = 1 , . . .

Original: , T r do 4:
Revised: , T r do 4:

Original: S t ←− ⌊ mµ ⌋ clients selected at random 5: send the global model ϕ t , ω t , K to clients in S t 6: for i ∈ S t do 7: ϕ ti , ω ti , K i ←−
Revised: S t ←− ⌊ mµ ⌋ clients selected at random 5: send the global model ϕ t , ω t , K to clients in S t 6: for i ∈ S t do 7: ϕ ti , ω ti , K i ←−

Original: LocalUpdate ( ϕ t , ω t , K , D i , σ 2 , ν, i ) 8: end for 9:
Revised: LocalUpdate ( ϕ t , ω t , K , D i , σ ν i ) 8: end for 9:

Original: Aggregate global hyper-knowledge K by
Revised: Aggregate global hyper-knowledge K by

Original: Eq. 9.
Revised: Eq. 9.

Original: Aggregate global model θ t + = ( ϕ t+1 , ω t +1 ) 11: end for 12: return θ T r +1 =
Revised: Aggregate global model θ t + = ( ϕ t ω t 1 ) 11: end for 12: return θ T r +1 =

Original: ( ϕ T r +1 , ω T r +1 ) 13: 14: LocalUpdate( ϕ t , ω t , K , D i , σ 2 s , i ): 15: ϕ ti ←− ϕ t , ω ti ←− ω t , ( x, y ) ∼ D i 16: for each local epoch do 17: ϕ ti , ω ti ←− OptimAlg ( L ( x, y, K , λ, γ )) 18: end for 19: update local hyper-knowledge K i 20: return ϕ ti , ω ti , K
Revised: ( ϕ T r +1 , ω T r +1 ) 13: 14: LocalUpdate( ϕ t , ω t , K , D i , σ 2 s , i ): 15: ϕ ti ←− ϕ t , ω ti ←− ω t , ( x, y ) ∼ D i 16: for each local epoch do 17: ϕ ti , ω ti ←− OptimAlg ( L ( x, y, K , λ, γ )) 18: end for 19: update local hyper-knowledge K i 20: return ϕ ti , ω ti , K

Original: i
Revised: i

Original: 3. C ONVERGENCE A NALYSIS
Revised: 3. C ONVERGENCE A NALYSIS

Original: To facilitate the convergence analysis of FedHKD, we make thefollowing assumptions commonly encountered in literature (Li et al., 2019; 2020; Tan et al., 2021).The proofs are in Appendix A.4. Assumption 1. (Lipschitz Continuity).
Revised: To facilitate the convergence analysis of FedHKD, we make the assumptions commonly encountered in literature (Li et al., 2019; 2020; Tan et al., 2021).

Original: The gradient of the local loss function L ( · ) is L 1 -Lipschitz continuous, the embedding function of the local feature extractor R ϕ ( · ) is L 2 -Lipschitz continuous, and the embedding function of the local classifier Q ( G ω ( · )) is L 3 -Lipschitz continuous, i.e.,

Original: [Equation]

Original: [Equation]

Original: Inequality 30 also implies that

Original: [Equation]

Original: Assumption 2. (Unbiased Gradient and Bounded Variance). The stochastic gradients on a batch of client i ’s data ξ i , denoted by g ti = ∇L ( θ ti , ξ ti ) [Equation]
Revised: The details in assumptions and proof are in

Original: is an unbiased estimator of the local gradient for each client i with the variance bounded by σ 2 ,

Original: i

Original: [Equation] e., [Equation] Assumption 3.
Revised: Appendix A.6.

Original: (Bounded Expectation of Gradients).

Original: The expectation of the stochastic gradient is bounded by V ,

Original: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Instate Assumptions 1-3.
Revised: Instate Assumptions 1-3.

Original: For an arbitrary client, after each communication round the loss function is bounded as
Revised: For an arbitrary client, after each communication round the loss function is bounded as

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: (FedHKD convergence rate)
Revised: (FedHKD convergence rate)

Original: Instate Assumptions 1-3 hold and define regret ∆ =
Revised: Instate Assumptions 1-3 hold and define regret ∆ =

Original: L 12 , 1 − L ∗ .
Revised: L 12 , 1 − L ∗ .

Original: If the learning rate is set to η , for an arbitrary client after
Revised: If the learning rate is set to η , for an arbitrary client after

Original: [Equation]
Revised: [Equation]

Original: global rounds ( ϵ > 0 ), it holds that
Revised: global rounds ( ϵ > 0 ), it holds that

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS

Original: In this section, we present extensive benchmarking results comparing the performance of FedHKD and the competing FL methods designed to address the challenge of learning from non-iid data.
Revised: In this section, we present extensive benchmarking results comparing the performance of FedHKD and the competing FL methods designed to address the challenge of learning from non-iid data.

Original: All the methods were implemented and simulated in Pytorch Paszke et al. (2019), with models trained using Adam optimizer Kingma & Ba (2014).
Revised: All the methods were implemented and simulated in Pytorch (Paszke et al., 2019), with models trained using Adam optimizer (Kingma & Ba, 2014).

Original: Details of the implementation and the selection of hyper-parameters are provided in Appendix.
Revised: Details of the implementation and the selection of hyper-parameters are provided in Appendix.

Original: Below we describe the datasets, models and baselines used in the experiments.
Revised: Below we describe the datasets, models and baselines used in the experiments.

Original: Datasets.
Revised: Datasets.

Original: Three benchmark datasets are used in the experiments: SVHN Netzer et al.
Revised: Three benchmark datasets are used in the experiments: SVHN (Netzer et al., 2011),

Original: CIFAR10 and CIFAR100 Krizhevsky et al.
Revised: CIFAR10 and CIFAR100 (Krizhevsky et al., 2009).

Original: To generate heterogeneous partitions of local training data, we follow the strategy in Yoon et al.
Revised: To generate heterogeneous partitions of local training data, we follow the strategy in (Yoon et al., 2021;

Original: ; Yurochkin et al. ; Li et al. (2021a)
Revised: Yurochkin et al., 2019; Li et al., 2021a)

Original: and utilize Dirichlet distribution with varied concentration parameters β which controls the level of heterogeneity.
Revised: and utilize Dirichlet distribution with varied concentration parameters β which controls the level of heterogeneity.

Original: Since our focus is on understanding and addressing the impact of class heterogeneity in clients data on the performance of trained models, we set equal the size of clients’ datasets.
Revised: Since our focus is on understanding and addressing the impact of class heterogeneity in clients data on the performance of trained models, we set equal the size of clients’ datasets.

Original: Furthermore, to evaluate both personalized as well as global model performance, each client is allocated a local test dataset (with the same class distribution as the corresponding local training dataset) and a global test dataset with uniformly distributed classes (shared by all participating clients); this allows computing both the average local test accuracy of the trained local models as well as the global test accuracy of the global model aggregated from the clients’ local models.
Revised: Furthermore, to evaluate both personalized as well as global model performance, each client is allocated a local test dataset (with the same class distribution as the corresponding local training dataset) and a global test dataset with uniformly distributed classes (shared by all participating clients); this allows computing both the average local test accuracy of the trained local models as well as the global test accuracy of the global model aggregated from the clients’ local models.

Original: Models.
Revised: Models.

Original: Rather than evaluate the performance of competing schemes on a simple CNN network as in McMahan et al. ; Li et al. (2020; 2021a), we apply two widely used benchmarking models better suited to practical settings.
Revised: Rather than evaluate the performance of competing schemes on a simple CNN network as in (McMahan et al., 2017; Li et al., 2020; 2021a), we apply two widely used benchmarking models better suited to practical settings.

Original: Specifically, we deploy ShuffleNetV2 Ma et al. (2018) on
Revised: Specifically, we deploy ShuffleNetV2 (Ma et al., 2018) on

Original: SVHN and ResNet18 He et al. (2016) on CIFAR10/100.
Revised: SVHN and ResNet18 (He et al., 2016) on CIFAR10/100.

Original: As our results show, FedHKD generally outperforms competing methods on both (very different) architectures, demonstrating remarkable consistency and robustness.
Revised: As our results show, FedHKD generally outperforms competing methods on both (very different) architectures, demonstrating remarkable consistency and robustness.

Original: Baselines.
Revised: Baselines.

Original: We compare the test accuracy of FedHKD with seven state-of-the-art federated learning methods including FedAvg McMahan et al. FedMD Li & Wang (2019), FedProx Li et al. (2020), Moon Li et al. (2021a)
Revised: We compare the test accuracy of FedHKD with seven state-of-the-art federated learning methods including FedAvg (McMahan et al., 2017), FedMD (Li & Wang, 2019), FedProx (Li et al., 2020), Moon (Li et al., 2021a), FedProto (Tan et al., 2021), FedGen (Zhu et al., 2021) and FedAlign (Mendieta et al., 2022).

Original: , FedProto Tan et al.

Original: FedGen Zhu et al. (2021) and FedAlign Mendieta et al. We emphasize that the novelty of FedHKD lies in data-free knowledge distillation that requires neither a public dataset nor a generative model; this stands in contrast to FedMD which relies on a public dataset and FedGen which deploys a generative model.
Revised: We emphasize that the novelty of FedHKD lies in data-free knowledge distillation that requires neither a public dataset nor a generative model; this stands in contrast to FedMD which relies on a public dataset and FedGen which deploys a generative model.

Original: Like
Revised: Like

Original: FedHKD, FedProto shares means of data representations but uses different regularization terms in the loss functions and does not make use of soft predictions.
Revised: FedHKD, FedProto shares means of data representations but uses different regularization terms in the loss functions and does not make use of soft predictions.

Original: When discussing the results, we will particularly analyze and compare the performance of FedMD, FedGen and FedProto with the performance of FedHKD.
Revised: When discussing the results, we will particularly analyze and compare the performance of FedMD, FedGen and FedProto with the performance of FedHKD.

Original: 4. P ERFORMANCE A NALYSIS
Revised: 4. P ERFORMANCE A NALYSIS

Original: Table 1 shows that FedHKD generally outperforms other methods across various settings and datasets.
Revised: Table 1 shows that FedHKD generally outperforms other methods across various settings and datasets.

Original: For each dataset, we ran experiments with 10, 20 and 50 clients, with local data generated from a Dirichlet distribution with fixed concentration parameter β
Revised: For each dataset, we ran experiments with 10, 20 and 50 clients, with local data generated from a Dirichlet distribution with fixed concentration parameter β

Original: = 0 .
Revised: = 0 .

Original: As previously stated, 
Revised: As previously stated, we focus on the heterogeneity in class distribution of local dataset rather than the heterogeneity in the number of samples. To this end, an increasing fraction of data is partitioned and allocated to the clients in the experiments, maintaining the size of local datasets as the number of clients increases. A single client’s averaged training time per global round is computed across different settings to characterize the required training time. To provide a more informative comparison with FedProto (Tan

Original: [Table]
Revised: [Table]

Original: No FedHKD 0.
Revised: No FedHKD 0.

Original: [Table]

Original: [Table]

Original: No
Revised: No

Original: CIFAR
Revised: CIFAR

Original: FedAvg [Table] [Table] [Table]
Revised: FedAvg 0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: No FedProx [Table] [Table] [Table]
Revised: No FedProx 0.

Original: [Table]

Original: [Table] [Table] No Moon [Table]
Revised: No Moon 0.

Original: [Table]

Original: [Table] [Table] No FedAlign [Table]
Revised: No FedAlign 0.

Original: [Table] No FedGen [Table] [Table]
Revised: No FedGen 0.

Original: [Table] No FedMD [Table] [Table]
Revised: No FedMD 0.

Original: Yes FedProto [Table] [Table] [Table]
Revised: Yes FedProto 0.

Original: [Table] No FedHKD* [Table] [Table]
Revised: No FedHKD* 0.

Original: [Table]

Original: No FedHKD [Table] [Table] [Table]
Revised: No FedHKD 0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: No
Revised: No

Original: CIFAR
Revised: CIFAR

Original: FedAvg [Table]
Revised: FedAvg 0.

Original: [Table] [Table] [Table] [Table]
Revised: No FedProx

Original: [Table]
Revised: [Table]

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: we focus on the heterogeneity in class distribution of local dataset rather than the heterogeneity in the number of samples. To this end, an increasing fraction of data is partitioned and allocated to the clients in the experiments, maintaining the size of local datasets as the number of clients increases.
Revised: 0.7128 FedAlign 0. 0.7919 FedGen 0.

Original: A single client’s averaged training time per global round is computed across different settings to characterize the required training time.
Revised: 0.7251 FedMD 0.

Original: To provide a more informative comparison with FedProto 
Revised: 0.8144 FedProto 0. 0.8055 FedHKD* 0. 0.8044 FedHKD 0.

Original: Tan et al. (2021), we ran two setting of our proposed method, labeled as FedHKD and FedHKD*: (1)
Revised: et al., 2021), we ran two setting of our proposed method, labeled as FedHKD and FedHKD*: (1)

Original: FedHKD deploys the second and third term in Eq.
Revised: FedHKD deploys the second and third term in Eq.

Original: 11 using λ = 0 .
Revised: 11 using λ = 0 .

Original: 05 and γ = 0 .
Revised: 05 and γ = 0 .

Original: 05 ; (2) FedHKD* excludes the constraint on Feature Extractor R ϕ by setting λ = 0 .
Revised: 05 ; (2) FedHKD* excludes the constraint on Feature Extractor R ϕ by setting λ = 0 .

Original: 05 and γ = 0 .
Revised: 05 and γ = 0 .

Original: Accuracy comparison.
Revised: Accuracy comparison.

Original: The proposed method, FedHKD, generally ranks as either the best or the second best in terms of both local and global accuracy, competing with FedMD without using public data.
Revised: The proposed method, FedHKD, generally ranks as either the best or the second best in terms of both local and global accuracy, competing with FedMD without using public data.

Original: On SVHN, FedHKD significantly improves the local test accuracy over FedAvg (by 19.5%, 14.3% and 20.6%) as well as the global test accuracy (by 37.0%, 15.6% and 39.5%) in experiments involving 10 , 20 and 50 clients, respectively.
Revised: On SVHN, FedHKD significantly improves the local test accuracy over FedAvg (by 19.5%, 14.3% and 20.6%) as well as the global test accuracy (by 37.0%, 15.6% and 39.5%) in experiments involving 10 , 20 and 50 clients, respectively.

Original: The improvement over FedAvg carry over to the experiments on CIFAR10, with 5.1%, 8.9% and 14.5% increase in local accuracy and 14.5%, 9.9% and 45.6% increase in global accuracy in the experiments involving 10, 20 and 50 clients, respectively.
Revised: The improvement over FedAvg carry over to the experiments on CIFAR10, with 5.1%, 8.9% and 14.5% increase in local accuracy and 14.5%, 9.9% and 45.6% increase in global accuracy in the experiments involving 10, 20 and 50 clients, respectively.

Original: On CIFAR100, the improvement of global accuracy is somewhat more modest, but the improvement in local accuracy is still remarkable, outperforming FedAvg by 26.3%, 23.6% and 26.9% in the experiments involving 10 , 20 and 50 clients, respectively.
Revised: On CIFAR100, the improvement of global accuracy is somewhat more modest, but the improvement in local accuracy is still remarkable, outperforming FedAvg by 26.3%, 23.6% and 26.9% in the experiments involving 10 , 20 and 50 clients, respectively.

Original: The local test accuracies of FedHKD* and
Revised: The local test accuracies of FedHKD* and

Original: FedProto are comparable, but FedHKD* outperforms FedProto in terms of global test accuracy (as expected, following the discussion in Section 3.2).
Revised: FedProto are comparable, but FedHKD* outperforms FedProto in terms of global test accuracy (as expected, following the discussion in Section 3.2).

Original: FedAlign outperforms the other two regularization methods, FedProx and Moon, both locally and globally; however, but is not competitive with the other methods in which clients’ local training is assisted by additional information provided by the server.
Revised: FedAlign outperforms the other two regularization methods, FedProx and Moon, both locally and globally; however, but is not competitive with the other methods in which clients’ local training is assisted by additional information provided by the server.

Original: While it has been reported that FedGen performs well on simpler datasets such as
Revised: While it has been reported that FedGen performs well on simpler datasets such as

Original: MNIST LeCun et al. and EMNIST Cohen et al. (2017), it appears that its MLP-based generative model is unable to synthesize data of sufficient quality to assist in KD-based FL on SVHN and CIFAR10/100 – on the former dataset, FedGen actually leads to performance deterioration as compared to FedAvg.
Revised: MNIST (LeCun et al., 1998) and EMNIST (Cohen et al., 2017), it appears that its MLP-based gen- erative model is unable to synthesize data of sufficient quality to assist in KD-based FL on SVHN and CIFAR10/100 – on the former dataset, FedGen actually leads to performance deterioration as compared to FedAvg.

Original: Training time comparison.
Revised: Training time comparison.

Original: We compare training efficiency of different methods in terms of the averaged training time (in second) per round/client.
Revised: We compare training efficiency of different methods in terms of the averaged training time (in second) per round/client.

Original: For fairness, all the experiments were conducted on the same machine with 8 AMD Vega20 GPUs.
Revised: For fairness, all the experiments were conducted on the same machine with 8 AMD Vega20 GPUs.

Original: As shown in Table 1, the training time of FedHKD, FedHKD*, FedProto and FedGen is slightly higher than the training time of FedAvg.
Revised: As shown in Table 1, the training time of FedHKD, FedHKD*, FedProto and FedGen is slightly higher than the training time of FedAvg.

Original: The additional computational burden of FedHKD is due to evaluating two extra regularization terms and calculating local hyper-knowledge.
Revised: The additional computational burden of FedHKD is due to evaluating two extra regularization terms and calculating local hyper-knowledge.

Original: The extra computations of FedGen are primarily due to training a generative model; the MLP-based generator leads to minor additional computations but clearly limits the performance of FedGen.
Revised: The extra computations of FedGen are primarily due to training a generative model; the MLP-based generator leads to minor additional computations but clearly limits the performance of FedGen.

Original: FedMD relies on a public dataset of the same size as the clients’ local datasets, thus approximately doubling the time FedAvg needs to complete the forward and backward pass during training.
Revised: FedMD relies on a public dataset of the same size as the clients’ local datasets, thus approximately doubling the time FedAvg needs to complete the forward and backward pass during training.

Original: Finally, the training efficiency of Moon and FedAlign is inferior to the training efficiency of other methods.
Revised: Finally, the training efficiency of Moon and FedAlign is inferior to the training efficiency of other methods.

Original: Moon is inefficient as it requires more than double the training time of FedAvg.
Revised: Moon is inefficient as it requires more than double the training time of FedAvg.

Original: FedAlign needs to pass forward the network multiple times and runs large matrix multiplications to estimate second-order information (Hessian matrix).
Revised: FedAlign needs to pass forward the network multiple times and runs large matrix multiplications to estimate second-order information (Hessian matrix).

Original: Effect of class heterogeneity.
Revised: Effect of class heterogeneity.

Original: We compare the performance of the proposed method, FedHKD, and other techniques as the data heterogeneity is varied by tuning the parameter β .
Revised: We compare the performance of the proposed method, FedHKD, and other techniques as the data heterogeneity is varied by tuning the parameter β .

Original: When β
Revised: When β

Original: = 0 .
Revised: = 0 .

Original: 2 , the heterogeneity is severe and the local datasets typically contain only one or two classes; when β = 5 , the local datasets are nearly homogeneous.
Revised: 2 , the heterogeneity is severe and the local datasets typically contain only one or two classes; when β = 5 , the local datasets are nearly homogeneous.

Original: Data distributions are visualized in Appendix A.2.
Revised: Data distributions are visualized in Appendix A.2.

Original: As shown in Table 2, FedHKD improves both local and global accuracy in all settings, surpassing other methods except FedMD on SVHN dataset for β = 5 .
Revised: As shown in Table 2, FedHKD improves both local and global accuracy in all settings, surpassing other methods except FedMD on SVHN dataset for β = 5 .

Original: FedProto exhibits remarkable improvement on local accuracy with either extremely heterogeneous ( β = 0 . 2 ) or homogeneous ( β = 5 ) local data but its global performance deteriorates when β
Revised: FedProto exhibits remarkable improvement on local accuracy with either extremely heterogeneous ( β = 0 . 2 ) or homogeneous ( β = 5 ) local data but its global performance deteriorates when β

Original: = 0 .
Revised: = 0 .

Original: 4. P RIVACY A NALYSIS
Revised: 4. P RIVACY A NALYSIS

Original: In our experimental setting, clients share the same network architecture (either ShuffleNetV2 or
Revised: In our experimental setting, clients share the same network architecture (either ShuffleNetV2 or

Original: ResNet18).
Revised: ResNet18).

Original: In both network architectures, the outermost layer in the feature extractor is a batch normalization (BN) layer Ioffe & Szegedy (2015).
Revised: In both network architectures, the outermost layer in the feature extractor is a batch normalization (BN) layer (Ioffe & Szegedy, 2015).

Original: For a batch of vectors B = { v 1 , . . .
Revised: For a batch of vectors B = { v 1 , . . .

Original: , v b } at the input of the BN layer, the operation of the BN layer is specified by
Revised: , v b } at the input of the BN layer, the operation of the BN layer is specified by

Original: [Equation]
Revised: [Equation]

Original: Assuming b is sufficiently large, the law of large numbers implies ˜ v i ∼ N (0 , 1) .
Revised: Assuming b is sufficiently large, the law of large numbers implies ˜ v i ∼ N (0 , 1) .

Original: Therefore, − 3 ≤ v i ≤ 3 with probability 99 .
Revised: Therefore, − 3 ≤ v i ≤ 3 with probability 99 .

Original: 73% (almost surely).
Revised: 73% (almost surely).

Original: Consider the experimental scenarios where client i contains N i = 1024 samples in its local dataset, the sharing threshold is ν = 0 .
Revised: Consider the experimental scenarios where client i contains N i = 1024 samples in its local dataset, the sharing threshold is ν = 0 .

Original: 25 , N ji > νN i = 256 , δ = 0 .
Revised: 25 , N ji > νN i = 256 , δ = 0 .

Original: 01 , and ϵ = 0 .
Revised: 01 , and ϵ = 0 .

Original: According to Theorem 1, to obtain 0 .
Revised: According to Theorem 1, to obtain 0 .

Original: 5 -differential privacy with confidence 1 − δ = 99% we set σ > (cid:113) 2 log 54 δ /ε
Revised: 5 -differential privacy with confidence 1 − δ = 99% we set σ > (cid:113) 2 log 54 δ /ε

Original: ≈ 6 . 215 .
Revised: ≈ 6 . 215 .

Original: According to Lemma 1, ( S i f ) 2 = (cid:16) 2 ζ
Revised: According to Lemma 1, ( S if ) 2 = (cid:16) 2 ζ

Original: N ji (cid:17) 2 < ( 6256 ) 2 .
Revised: N ji (cid:17) 2 < ( 6256 ) 2 .

Original: Setting σ = 7 (large privacy budget), the variance of noise added to the hyperknowledge K ji of client i should be ( S if ) 2 σ 2 < 0 .
Revised: Setting σ = 7 (large privacy budget), the variance of noise added to the hyperknowledge K ji of client i should be ( S if ) 2 σ 2 < 0 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented FedHKD, a novel FL algorithm that relies on knowledge distillation to enable efficient learning of personalized and global models in data heterogeneous settings; FedHKD requires neither a public dataset nor a generative model and therefore addresses the data heterogeneity challenge without a need for significantly higher resources.
Revised: We presented FedHKD, a novel FL algorithm that relies on knowledge distillation to enable efficient learning of personalized and global models in data heterogeneous settings; FedHKD requires neither a public dataset nor a generative model and therefore addresses the data heterogeneity challenge without a need for significantly higher resources.

Original: By introducing and utilizing the concept of “hyper-knowledge”, information that consists of the means of data representations and the corresponding means of soft predictions, FedHKD enables clients to train personalized models that perform well locally while allowing the server to aggregate a global model that performs well across all data classes.
Revised: By introducing and utilizing the concept of “hyper-knowledge”, information that consists of the means of data representations and the corresponding means of soft predictions, FedHKD enables clients to train personalized models that perform well locally while allowing the server to aggregate a global model that performs well across all data classes.

Original: To address privacy concerns, FedHKD deploys a differential privacy mechanism.
Revised: To address privacy concerns, FedHKD deploys a differential privacy mechanism.

Original: We conducted extensive experiments in a variety of setting on several benchmark datasets, and provided a theoretical analysis of the convergence of FedHDK.
Revised: We conducted extensive experiments in a variety of setting on several benchmark datasets, and provided a theoretical analysis of the convergence of FedHKD.

Original: The experimental results demonstrate that FedHKD outperforms state-of-the-art federated learning schemes in terms of both local and global accuracy while only slightly increasing the training time.
Revised: The experimental results demonstrate that FedHKD outperforms state-of-the-art federated learning schemes in terms of both local and global accuracy while only slightly increasing the training time.

Original: A A PPENDIX A. E XPERIMENTAL D ETAILS
Revised: A A PPENDIX A. E XPERIMENTAL D ETAILS

Original: General setting.
Revised: General setting.

Original: We implemented all the models and ran the experiments in Pytorch (Paszke et al., 2019)
Revised: We implemented all the models and ran the experiments in Pytorch (Paszke et al., 2019)

Original: (Ubuntu 18.04 operating system, 8 AMD Vega20 GPUs).
Revised: (Ubuntu 18.04 operating system, 8 AMD Vega20 GPUs).

Original: Adam (Kingma & Ba, 2014) optimizer was used for model training in all the experiments; learning rate was initialized to 0.001 and decreased every 10 iterations with a decay factor 0.5, while the hyper-parameter γ in Adam was set to 0 .
Revised: Adam (Kingma & Ba, 2014) optimizer was used for model training in all the experiments; learning rate was initialized to 0.001 and decreased every 10 iterations with a decay factor 0.5, while the hyper-parameter γ in Adam was set to 0 .

Original: The number of global communication rounds was set to 50 while the number of local epochs was set to 5.
Revised: The number of global communication rounds was set to 50 while the number of local epochs was set to 5.

Original: The size of a data batch was set to 64 and the participating rate of clients was for simplicity set to 1.
Revised: The size of a data batch was set to 64 and the participating rate of clients was for simplicity set to 1.

Original: For SVHN (Netzer et al., 2011) dataset, the latent dimension of data representation was set to 32; for CIFAR10/100 (Krizhevsky et al., 2009), the latent dimension was set to 64.
Revised: For SVHN (Netzer et al., 2011) dataset, the latent dimension of data representation was set to 32; for CIFAR10/100 (Krizhevsky et al., 2009), the latent dimension was set to 64.

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: In all experiments, the FedProx (Li et al., 2020) hyper-parameter µ prox was set to 0.5; the Moon (Li et al., 2021a) hyper-parameter µ moon in the proximTal term was set to 1.
Revised: In all experiments, the FedProx (Li et al., 2020) hyper-parameter µ prox was set to 0.5; the Moon (Li et al., 2021a) hyper-parameter µ moon in the proximTal term was set to 1.

Original: In
Revised: In

Original: FedAlign (Mendieta et al., 2022), the fractional width of the sub-network was set to 0.25, and the balancing parameter µ align was set to 0.45.
Revised: FedAlign (Mendieta et al., 2022), the fractional width of the sub-network was set to 0.25, and the balancing parameter µ align was set to 0.45.

Original: The generative model required by FedGen (Zhu et al., 2021) is the MLP-based architecture proposed in (Zhu et al., 2021).
Revised: The generative model required by FedGen (Zhu et al., 2021) is the MLP-based architecture proposed in (Zhu et al., 2021).

Original: The hidden dimension of the generator was set to 512 ; the latent dimension, noise dimension, and input/output channels were adapted to the datasets.
Revised: The hidden dimension of the generator was set to 512 ; the latent dimension, noise dimension, and input/output channels were adapted to the datasets.

Original: The number of epochs for training the generative model in each global round was set to 5, and the ratio of the generating batch-size and the training batch-size was set to 0.5 (i.e, the generating batch-size was set to 32).
Revised: The number of epochs for training the generative model in each global round was set to 5, and the ratio of the generating batch-size and the training batch-size was set to 0.5 (i.e, the generating batch-size was set to 32).

Original: Parameters α generative and β generative were initialized to 10 with a decay factor 0 .
Revised: Parameters α generative and β generative were initialized to 10 with a decay factor 0 .

Original: 98 in each global round.
Revised: 98 in each global round.

Original: In FedMD (Li & Wang, 2019), we set the regularization hyper-parameter λ md to 0.05; the size of the public dataset was set equal to the size of the clients’ local training dataset.
Revised: In FedMD (Li & Wang, 2019), we set the regularization hyper-parameter λ md to 0.05; the size of the public dataset was set equal to the size of the clients’ local training dataset.

Original: In FedProto Tan et al. (2021), the regularization hyper-parameter λ proto was set to 0.05.
Revised: In FedProto (Tan et al., 2021), the regularization hyper-parameter λ proto was set to 0.05.

Original: The hyper-parameters λ and γ in our proposed method FedHKD* were set to 0.05 and 0, respectively; as for FedHKD, the two hyper-parameters λ and γ were set to 0.05 and 0.05, respectively.
Revised: The hyper-parameters λ and γ in our proposed method FedHKD* were set to 0.05 and 0, respectively; as for FedHKD, the two hyper-parameters λ and γ were set to 0.05 and 0.05, respectively.

Original: Variance σ of the Gaussian noise added to the generated hyper-knowledge was set to 7 ; threshold ν that needs to be met to initiate computation of hyper-knowledge was set to 0 .
Revised: Variance σ of the Gaussian noise added to the generated hyper-knowledge was set to 7 ; threshold ν that needs to be met to initiate computation of hyper-knowledge was set to 0 .

Original: Temperature for FedHKD and Moon algorithm was set to 0.5.
Revised: Temperature for FedHKD and Moon algorithm was set to 0.5.

Original: A. D ATA P ARTITIONING
Revised: A. D ATA P ARTITIONING

Original: For convenience, we used datasets encapsulated by Torchvision To obtain the global test dataset, we directly load SVHN, CIFAR10 and CIFAR100 test set in Torchvision without any sampling.
Revised: For convenience, we used datasets encapsulated by Torchvision To obtain the global test dataset, we directly load SVHN, CIFAR10 and CIFAR100 test set in Torchvision without any sampling.

Original: For the local training and test sets, we first utilized Dirichlet distribution to sample m partitions as m local datasets from the encapsulated set ( m denotes the number of clients).
Revised: For the local training and test sets, we first utilized Dirichlet distribution to sample m partitions as m local datasets from the encapsulated set ( m denotes the number of clients).

Original: Then we divided the local dataset into a training and test set in 75%/25% proportion.
Revised: Then we divided the local dataset into a training and test set in 75%/25% proportion.

Original: Figures 1, 2 and 3 visualize the class distribution of local clients by showing the number of samples belonging to different classes at each client (colors distinguish the magnitude – the darker the color, the more samples are in the corresponding class).
Revised: Figures 1, 2 and 3 visualize the class distribution of local clients by showing the number of samples belonging to different classes at each client (colors distinguish the magnitude – the darker the color, the more samples are in the corresponding class).

Original: [Figure]
Revised: [Figure]

Revised: A. F LOW D IAGRAM I LLUSTRATING C OMPUTATION OF H YPER -K NOWLEDGE

Revised: Figure 4 illustrates computation of local hyper-knowledge by a client.

Revised: At the end of local training, each participating client obtains a fine-tuned local model consisting of a feature extractor R ϕ ( · ) and a classifier G ω ( · ) .

Revised: There are three steps in the process of obtaining local hyper-knowledge for class j of client k : (1) Representations of data samples in class j , generated by the feature extractor, are used to compute the mean of data representations for that class; (2) A classifier generates soft predictions for the obtained data representations, thus enabling computation of the mean of soft predictions for class j ; (3) After adding Gaussian noise to the mean of data representations, the noisy mean of data representations and mean of soft predictions are packaged into local hyper-knowledge for class j .

Original: A. P ROOF OF L EMMA
Revised: A. D ETAILS OF THE F ED HKD ALGORITHM

Revised: Figure. 5 illustrates the iterative training procedure of FedHKD. At the start of training, global hyper-knowledge is initialized to an empty set and thus in round 1 each client trains its local model without global hyper-knowledge.

Revised: Following local training, each client extracts representations from local data samples via a feature extractor and finds soft predictions via a classifier, computing local hyper-knowledge as shown in Figure.

Revised: The server collects local hyper-knowledge and model updates from clients, aggregates them into global hyper-knowledge and model, and then sends the results back to the clients.

Revised: From this point on, clients perform local training aided by the global knowledge.

Revised: Alternating local training and aggregation lasts for T − 1 rounds where T denotes the number of global epochs.

Revised: A. P ROOF OF L EMMA

Original: To compute i th client’s mean of class j representation, ¯ h ji , we consider the deterministic function (averaging in an element-wise manner)
Revised: To compute i th client’s mean of class j representation, ¯ h ji , we consider the deterministic function (averaging in an element-wise manner)

Original: f l ( d ji ) ≜ ¯ h ji ( l ) =
Revised: f l ( d ji ) ≜ ¯ h ji ( l ) =

Original: 1 N ji (cid:80) N ji k =1 ¯ h j,ki ( l ) where d ji is the subset of the i th client’s local dataset collecting samples with label j ; h j,ki denotes the data representation of the k th sample in d ji while h j,ki ( l ) is the l th element of h j,ki .
Revised: 1 N ji (cid:80) N ji k =1 ¯ h j,ki ( l ) where d ji is the subset of the i th client’s local dataset collecting samples with label j ; h j,ki denotes the data representation of the k th sample in d ji while h j,ki ( l ) is the l th element of h j,ki .

Original: Lemma 1.
Revised: Lemma 1.

Original: If | h j,ki ( l ) | is bounded by ζ > 0 for any k , then
Revised: If | h j,ki ( l ) | is bounded by ζ > 0 for any k , then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof: Without a loss of generality, specify
Revised: Proof: Without a loss of generality, specify

Original: [Equation]
Revised: [Equation]

Original: and e 
Revised: and e ′

Original: [Equation]
Revised: [Equation]

Original: 24) where e and e ′ denote adjacent sets differing in at most one element.
Revised: 18) where e and e ′ denote adjacent sets differing in at most one element.

Original: Define 1 = { 1 , . . .
Revised: Define 1 = { 1 , . . .

Original: , 1 } with | 1 | =
Revised: , 1 } with | 1 | =

Original: N ji − 1 .
Revised: N ji − 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: i ( l ) − 1 T e ′
Revised: i ( l ) − 1 T e ′

Original: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:16) N ji −(cid:17)
Revised: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:16) N ji −(cid:17)

Original: h
Revised: h

Original: N ji
Revised: N ji

Original: i
Revised: i

Original: ( l )
Revised: ( l )

Original: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 T e ′
Revised: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 T e ′

Original: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:16) N ji − 1 (cid:17)
Revised: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:16) N ji − 1 (cid:17)

Original: ζ
Revised: ζ

Original: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:16) N ji − 1 (cid:17)
Revised: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:16) N ji − 1 (cid:17)

Original: ζ
Revised: ζ

Original: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: N ji (cid:16) N ji − 1 (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: ζ
Revised: ζ

Original: N j i + ζ
Revised: N j i + ζ

Original: N j i = 2 ζ
Revised: N j i = 2 ζ

Original: N j i .
Revised: N j i .

Original: A. C ONVERGENCE A NALYSIS OF F ED HKD
Revised: A. C ONVERGENCE A NALYSIS OF F ED HKD

Original: It will be helpful to recall the notation before restating the theorems and providing their proofs.
Revised: It will be helpful to recall the notation before restating the theorems and providing their proofs.

Original: Let R ϕ
Revised: Let R ϕ

Original: i ( · ) : R d x → R d r denote the feature extractor function of client i , mapping the raw data of dimension d x into the representation space of dimension d r .
Revised: i ( · ) : R d x → R d r denote the feature extractor function of client i , mapping the raw data of dimension d x into the representation space of dimension d r .

Original: Let G ω
Revised: Let G ω

Original: i ( · ) : R d r → R n denote the classifier’s function of client i , projecting the data representation into the categorical space of dimension n .
Revised: i ( · ) : R d r → R n denote the classifier’s function of client i , projecting the data representation into the categorical space of dimension n .

Original: Let F
Revised: Let F

Original: θ i =( ϕ
Revised: θ i =( ϕ

Original: i , ω i ) ( · ) =
Revised: i , ω i ) ( · ) =

Original: G ω i ( · ) ◦
Revised: G ω i ( · ) ◦

Original: R ϕ
Revised: R ϕ

Original: i ( · ) denote the mapping of the entire model.
Revised: i ( · ) denote the mapping of the entire model.

Original: The local objective function of client i is formed as
Revised: The local objective function of client i is formed as

Original: [Equation]
Revised: [Equation]

Original: [Equation] where D i denotes the local dataset of client i ; input x k and label y k are drawn from D i ; B i is the number of samples in a batch of D i ; Q ( · , T ) is the soft target function with temperature T ;
Revised: where D i denotes the local dataset of client i ; input x k and label y k are drawn from D i ; B i is the number of samples in a batch of D i ; Q ( · , T ) is the soft target function with temperature T ;

Original: H j denotes the global mean data representation of class j ; Q y k is the corresponding global soft prediction of class y k ; and λ and γ are the hyper-parameters.
Revised: H j denotes the global mean data representation of class j ; Q y k is the corresponding global soft prediction of class y k ; and λ and γ are the hyper-parameters.

Original: Note that only ϕ i and ω i are variables in the loss function while the other terms are constant.
Revised: Note that only ϕ i and ω i are variables in the loss function while the other terms are constant.

Original: Let t denote the current global training round.
Revised: Let t denote the current global training round.

Original: During any global round, there are E local training epochs.
Revised: During any global round, there are E local training epochs.

Original: Assume the loss function is minimized by relying on stochastic gradient descent (SGD).
Revised: Assume the loss function is minimized by relying on stochastic gradient descent (SGD).

Original: To compare the loss before and after model/hyper-knowledge aggregation at the server, denote the local epoch by e ∈ { 12 , 1 , . .
Revised: To compare the loss before and after model/hyper-knowledge aggregation at the server, denote the local epoch by e ∈ { 12 , 1 , . .

Original: , E } ; e = 12 indicates the epoch between the end of the server’s aggregation in the previous communication round and the first epoch of the local training in the next round.
Revised: , E } ; e = 12 indicates the epoch between the end of the server’s aggregation in the previous communication round and the first epoch of the local training in the next round.

Original: After E epochs of local training in communication round t , the local model of client i is denoted as ( ϕ E,ti , ω E,ti ) .
Revised: After E epochs of local training in communication round t , the local model of client i is denoted as ( ϕ E,ti , ω E,ti ) .

Original: At the global communication round t + 1 , client i initializes the local model with the aggregated global model, ( ϕ 12 ,t +1 i , ω 12 ,t +1 i ) .
Revised: At the global communication round t + 1 , client i initializes the local model with the aggregated global model, ( ϕ 12 ,t +1 i , ω 12 ,t +1 i ) .

Original: Although client i does not begin the next training epoch, the local model is changed and so is the output of the loss function.
Revised: Although client i does not begin the next training epoch, the local model is changed and so is the output of the loss function.

Original: At the server, the global model is updated as
Revised: At the server, the global model is updated as

Original: [Equation]
Revised: [Equation]

Original: ) where θ E,ti is the local model of client i after E local training epoches at round t ; p i is the averaging weight of client i , where (cid:80) mi =1 p i = 1 .
Revised: ) where θ E,ti is the local model of client i after E local training epoches at round t ; p i is the averaging weight of client i , where (cid:80) mi =1 p i = 1 .

Original: ˜ h
Revised: ˜ h

Original: j,t and ¯ q j,t are aggregated as
Revised: j,t and ¯ q j,t are aggregated as

Original: [Equation]
Revised: [Equation]

Original: A.4. A SSUMPTIONS
Revised: A.6. A SSUMPTIONS

Original: Assumption 1.
Revised: Assumption 1.

Original: (Lipschitz Continuity).
Revised: (Lipschitz Continuity).

Original: The gradient of the local loss function L ( · ) is L 1 -Lipschitz continuous, the embedding functions of the local feature extractor R ϕ ( · ) is L 2 -Lipschitz continuous, and the embedding functions of the local classifier G ω ( · ) composition with soft prediction function
Revised: The gradient of the local loss function L ( · ) is L 1 -Lipschitz continuous, the embedding functions of the local feature extractor R ϕ ( · ) is L 2 -Lipschitz continuous, and the embedding functions of the local classifier G ω ( · ) composition with soft prediction function

Original: Q ( · , T ) is L 3 -Lipschitz continuous,
Revised: Q ( · , T ) is L 3 -Lipschitz continuous,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Inequality 30 also implies
Revised: Inequality 24 also implies

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: (Unbiased Gradient and Bounded Variance).
Revised: (Unbiased Gradient and Bounded Variance).

Original: The stochastic gradients on a batch of client i ’s data ξ i , denoted by g ti = ∇L ( θ ti , ξ ti ) , is an unbiased estimator of the local gradient for each client i ,
Revised: The stochastic gradients on a batch of client i ’s data ξ i , denoted by g ti = ∇L ( θ ti , ξ ti ) , is an unbiased estimator of the local gradient for each client i ,

Original: [Equation]
Revised: [Equation]

Original: ) with the variance bounded by σ 2 ,
Revised: ) with the variance bounded by σ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.
Revised: Assumption 3.

Original: (Bounded Expectation of Gradients).
Revised: (Bounded Expectation of Gradients).

Original: The expectation of the stochastic gradient is bounded by V ,
Revised: The expectation of the stochastic gradient is bounded by V ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Instate Assumptions 1-3.
Revised: Instate Assumptions 1-3.

Original: The loss function after E local training epoches at global round t + 1 can be bounded as
Revised: The loss function after E local training epoches at global round t + 1 can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where η e is the step-size (learning rate) at local epoch e .
Revised: where η e is the step-size (learning rate) at local epoch e .

Original: Proof:
Revised: Proof:

Original: [Equation]
Revised: [Equation]

Original: where inequality (1) follows from Assumption
Revised: where inequality (1) follows from Assumption

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of both sides (the sampling batch ξ t +1 )
Revised: Taking expectation of both sides (the sampling batch ξ t +1 )

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: [Equation] Inequality (2) follows from Assumption 2; (3) follows from V [ x ] =
Revised: Inequality (2) follows from Assumption 2; (3) follows from V [ x ] =

Original: E (cid:2) x 2 (cid:3) −
Revised: E (cid:2) x 2 (cid:3) −

Original: [Equation]
Revised: [Equation]

Original: [ x ] 2 , where x is a random variable; (4) holds due to Assumptions 2-3.
Revised: [ x ] 2 , where x is a random variable; (4) holds due to Assumptions 2-3.

Original: Let us set the learning step at the start of local training to η 12 = η 0 .
Revised: Let us set the learning step at the start of local training to η 12 = η 0 .

Original: By telescoping,
Revised: By telescoping,

Original: [Equation]
Revised: [Equation]

Original: The above inequality holds due to the fact that the learning rate η is non-increasing.
Revised: The above inequality holds due to the fact that the learning rate η is non-increasing.

Original: Lemma 2.
Revised: Lemma 2.

Original: Following the model and hyper-knowledge aggregation at the server, the loss function of any client i at global round t + 1 can be bounded as
Revised: Following the model and hyper-knowledge aggregation at the server, the loss function of any client i at global round t + 1 can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: [Equation]
Revised: [Equation]

Original: (42) where inequality (1) follows from Assumption
Revised: ) where inequality (1) follows from Assumption

Original: [Equation]
Revised: [Equation]

Original: and (2) is derived from Eq [Equation]
Revised: and (2) is derived from Eq. 21.

Original: Taking expectation of both side,
Revised: Taking expectation of both side,

Original: [Equation]
Revised: [Equation]

Original: [Table] − [Table]  + E L ( 
Revised: j = p j θ E,tj − θ E,ti (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + E L ( θ E,ti , K t +1 ) − E L

Original: [Table] E, [Table] i , K 
Revised: ( θ E,ti , K t ) =

Original: [Table]
Revised: L2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88)

Original:  ) − E L ( θ  E,ti  , K [Table] [Table] − 
Revised: j = p j θ E,tj − θ 12 ,t i − (cid:16) θ E,ti − θ 12 ,t i (cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + E L ( θ E,t , K t +1 ) − E L ( θ E,t , K t ) (2) ≤ L2 E (cid:13)(cid:13)(cid:13) θ E,ti − θ 12 ,t i (cid:13)(cid:13)(cid:13)2 + E L ( θ E,t , K t +1 ) − E L ( θ E,t , K t ) =

Original: [Table] [Table] − (cid:16) [Table]
Revised: L2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Table] − 
Revised: E − 1 (cid:88) e =η e g e,ti (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + E L ( θ E,t , K t +1 ) − E L

Original: [Table]
Revised: ( θ E,t , K t ) (3) ≤ L2 E

Original: [Table]
Revised: E − 1 (cid:88) e =

Original: (cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: Eη 2 e (cid:13)(cid:13)

Original: [Table] + E L ( [Table] E,t , K t +1 )− E L (
Revised: g e,t i (cid:13)(cid:13) 2 2 + E L ( θ E,t , K t +1 )

Original: [Table]
Revised: − E L

Original: E,t , K
Revised: θ E,

Original: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Due to Lemma 3 and the proof of Lemma 3 in Li et al. (2019), inequality (1) holds as E (cid:104)
Revised: Due to Lemma 3 and the proof of Lemma 3 in (Li et al., 2019), inequality (1) holds as E (cid:104)

Original: θ
Revised: θ

Original: E,tj (cid:105) =
Revised: E,t j (cid:105) =

Original: (cid:80) m
Revised: (cid:80) mj

Original: j =1 p j θ E,t j ; inequality (2) holds because E ∥
Revised: =1 p j θ E,tj ; inequality (2) holds because E ∥

Original: E X − X ∥ 

Revised: E X − X ∥

Original: ≤ E ∥
Revised: ≤ E ∥

Original: X ∥ 2 , where X = θ E,t i [Equation] inequality (3) is due to Jensen inequality; inequality (4) follows from that fact that the learning rate η e is non-increasing; inequality (5) holds due to Assumption 3.
Revised: X ∥ 2 , where X = θ E,ti − θ 12 ,t i ; inequality (3) is due to Jensen inequality; inequality (4) follows from that fact that the learning rate η e is non-increasing; inequality (5) holds due to Assumption 3.

Original: Let us consider the term
Revised: Let us consider the term

Original: L ( θ E,t , K t +1 ) − L ( θ E,t , K t ) ; note that the model parameters θ E,t are unchanged and thus the first term in the loss function 26 can be neglected.
Revised: L ( θ E,t , K t +1 ) − L ( θ E,t , K t ) ; note that the model parameters θ E,t are unchanged and thus the first term in the loss function 20 can be neglected.

Original: The difference between the two loss functions isdue to different global hyper-knowledge K t and K t +1 :
Revised: The difference between the two loss functions is

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 44) where (1) is due to the triangle inequality, ∥ a + b + c ∥ 2 ≤ ∥ a ∥ 2 + ∥ b ∥ 2 + ∥ c ∥ 2 with a =Q (cid:16) G ω E,tj ( H [Equation] (cid:16)
Revised: where (1) is due to the triangle inequality, ∥ a + b + c ∥ 2 ≤ ∥ a ∥ 2 + ∥ b ∥ 2 + ∥ c ∥ 2 with a =

Original: [Equation] tj ( H j [Equation] t ) (cid:17) and c = Q j,t − Q j,t +1 ; inequality (2) holds due to Assumption 1.
Revised: Q (cid:16) G ω E,tj ( H j,t ) (cid:17) − Q j,t , b = Q (cid:16) G ω E,tj ( H j,t +1 ) (cid:17) − Q (cid:16) G ω E,tj ( H j,t ) (cid:17) and c = Q j,t − Q j,t +1 ; inequality (2) holds due to Assumption 1.

Original: Then, let us consider the following difference:
Revised: Then, let us consider the following difference:

Original: [Equation]
Revised: [Equation]

Original: Inequality (1) holds due to Jensen’s inequality, while inequality (2) follows from Assumption 1.
Revised: Inequality (1) holds due to Jensen’s inequality, while inequality (2) follows from Assumption 1.

Original: For convenience (and perhaps clarity), we drop the superscript j denoting the class.
Revised: For convenience (and perhaps clarity), we drop the superscript j denoting the class.

Original: Taking expectation of both sides,
Revised: Taking expectation of both sides,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: [Equation]

Original:  (
Revised:  (

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: (cid:118)(cid:117) (cid:117)(cid:117)(cid:116) E (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) m (cid:88)
Revised: (cid:118)(cid:117) (cid:117)(cid:117)(cid:116) [Equation] (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) m (cid:88)

Original: j p
Revised: j p

Original: [Equation]
Revised: [Equation]

Original: E,t − 1 i
Revised: E,t − 1 i

Original: [Equation]
Revised: [Equation]

Original: 12 ,t − 1 i
Revised: 12 ,t − 1 i

Original: [Equation]
Revised: [Equation]

Original: 1 2 ,t − 1 i −
Revised: 1 2 ,t − 1 i −

Original: [Equation]
Revised: [Equation]

Original: E,t − 1 i (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)
Revised: E,t − 1 i (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (46) where (1) follows from the triangle inequality; inequality (2) holds due to Assumption 3 and the update rule of SGD; since f ( x ) = √ x is concave, (3) follows from Jensen’s inequality; inequality (4) holds due to the fact that E ∥
Revised: (40) where (1) follows from the triangle inequality; inequality (2) holds due to Assumption 3 and the update rule of SGD; since f ( x ) = √ x is concave, (3) follows from Jensen’s inequality; inequality (4) holds due to the fact that E ∥

Original: E X − X ∥ 

Revised: E X − X ∥

Original: ≤ E ∥
Revised: ≤ E ∥

Original: X ∥ 2 , where X = ϕ E,t − 1 i − ϕ 12 ,t − 1 i ; inequality (5) follows by using the fact that the learning rate η e is non-increasing.
Revised: X ∥ 2 , where X = ϕ E,t − 1 i − ϕ 12 ,t − 1 i ; inequality (5) follows by using the fact that the learning rate η e is non-increasing.

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Combining the above inequalities, we have
Revised: Combining the above inequalities, we have

Original: [Equation]
Revised: [Equation]

Original: A.4. T HEOREMS
Revised: A.6. T HEOREMS

Original: Theorem 2.
Revised: Theorem 2.

Original: Instate Assumptions 1-3.
Revised: Instate Assumptions 1-3.

Original: For an arbitrary client, after each communication round the loss function is bounded as
Revised: For an arbitrary client, after each communication round the loss function is bounded as

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning the learning rates η 0 , λ and γ ensures that
Revised: Fine-tuning the learning rates η 0 , λ and γ ensures that

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: (FedHKD convergence) Let η 0 > η e > αη 0 for e ∈ { 1 , . .
Revised: (FedHKD convergence) Let η 0 > η e > αη 0 for e ∈ { 1 , . .

Original: , E − 1 } , 0 < α < 1 .
Revised: , E − 1 } , 0 < α < 1 .

Original: The loss function of an arbitrary client monotonously decreases in each communication round if
Revised: The loss function of an arbitrary client monotonously decreases in each communication round if

Original: [Equation]
Revised: [Equation]

Original: where α denotes the hyper-parameter controlling learning rate decay.
Revised: where α denotes the hyper-parameter controlling learning rate decay.

Original: Proof:
Revised: Proof:

Original: Since η 0 < η e α , in each local epoch e we have η 2 e L
Revised: Since η 0 < η e α , in each local epoch e we have η 2 e L

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by η e ,
Revised: Dividing both sides by η e ,

Original: [Equation]
Revised: [Equation]

Original: Factoring out η e on the left hand side yields
Revised: Factoring out η e on the left hand side yields

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by (cid:16) L 1 2 α 2 (cid:0) EV 2 + σ 2 (cid:1) + L 1 2 ∥∇L e
Revised: Dividing both sides by (cid:16) L 1 2 α 2 (cid:0) EV 2 + σ 2 (cid:1) + L 1 2 ∥∇L e

Original: [Equation]
Revised: [Equation]

Original: t ∥ 22 (cid:17) results in
Revised: t ∥ 22 (cid:17) results in

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: (FedHKD convergence rate)
Revised: (FedHKD convergence rate)

Original: Instate Assumptions 1-3 and define regret ∆ =
Revised: Instate Assumptions 1-3 and define regret ∆ =

Original: L 12 , 1 −L ∗ .
Revised: L 12 , 1 −L ∗ .

Original: If the learning rate is set to η , for an arbitrary client after
Revised: If the learning rate is set to η , for an arbitrary client after

Original: T = 2∆
Revised: T = 2∆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: global rounds ( ϵ > 0 ), it holds that
Revised: global rounds ( ϵ > 0 ), it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: According to Theorem 1,
Revised: According to Theorem 1,

Original: TE
Revised: TE

Original: [Equation]
Revised: [Equation]

Original: Therefore, ∆
Revised: Therefore, ∆

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]