Original: T OWARDS M ODEL -A GNOSTIC F EDERATED L EARNING U SING K NOWLEDGE D ISTILLATION
Revised: T OWARDS M ODEL -A GNOSTIC F EDERATED L EARNING U SING K NOWLEDGE D ISTILLATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Is it possible to design an universal API for federated learning using which an ad-hoc group of data-holders (agents) collaborate with each other and perform federated learning?
Revised: Is it possible to design an universal API for federated learning using which an ad-hoc group of data-holders (agents) collaborate with each other and perform federated learning?

Original: Such an API would necessarily need to be model-agnostic i.e. make no assumption about the model architecture being used by the agents, and also cannot rely on having representative public data at hand.
Revised: Such an API would necessarily need to be model-agnostic i.e. make no assumption about the model architecture being used by the agents, and also cannot rely on having representative public data at hand.

Original: Knowledge distillation (KD) is the obvious tool of choice to design such protocols.
Revised: Knowledge distillation (KD) is the obvious tool of choice to design such protocols.

Original: However, surprisingly, we show that most natural KD-based federated learning protocols have poor performance.
Revised: However, surprisingly, we show that most natural KD-based federated learning protocols have poor performance.

Original: To investigate this, we propose a new theoretical framework, Federated Kernel ridge regression, which can capture both model heterogeneity as well as data heterogeneity.
Revised: To investigate this, we propose a new theoretical framework, Federated Kernel ridge regression, which can capture both model heterogeneity as well as data heterogeneity.

Original: Our analysis shows that the degradation is largely due to a fundamental limitation of knowledge distillation under data heterogeneity .
Revised: Our analysis shows that the degradation is largely due to a fundamental limitation of knowledge distillation under data heterogeneity .

Original: We further validate our framework by analyzing and designing new protocols based on KD.
Revised: We further validate our framework by analyzing and designing new protocols based on KD.

Original: Their performance on real world experiments using neural networks, though still unsatisfactory, closely matches our theoretical predictions.
Revised: Their performance on real world experiments using neural networks, though still unsatisfactory, closely matches our theoretical predictions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: “I speak and speak, but the listener retains only the words he is expecting...
Revised: “I speak and speak, but the listener retains only the words he is expecting...

Original: It is not the voice that commands the story: it is the ear.”
Revised: It is not the voice that commands the story: it is the ear.”

Original: - Invisible Cities, Italo Calvino.
Revised: - Invisible Cities, Italo Calvino.

Original: Federated learning (and more generally collaborative learning) involves multiple data holders (whom we call agents) collaborating with each other to train their machine learning model over their collective data.
Revised: Federated learning (and more generally collaborative learning) involves multiple data holders (whom we call agents) collaborating with each other to train their machine learning model over their collective data.

Original: Crucially, this is done without directly exchanging any of their raw data (McMahan et al., 2017;
Revised: Crucially, this is done without directly exchanging any of their raw data (McMahan et al., 2017;

Original: Kairouz et al., 2019).
Revised: Kairouz et al., 2019).

Original: Thus, communication is limited to only what is essential for the training process and the data holders (aka agents) retain full ownership over their datasets.
Revised: Thus, communication is limited to only what is essential for the training process and the data holders (aka agents) retain full ownership over their datasets.

Original: Algorithms for this setting such as FedAvg or its variants all proceed in rounds (Wang et al., 2021).
Revised: Algorithms for this setting such as FedAvg or its variants all proceed in rounds (Wang et al., 2021).

Original: In each such round, the agents ﬁrst train their models on their local data.
Revised: In each such round, the agents ﬁrst train their models on their local data.

Original: Then, the knowledge from these different models is aggregated by averaging the parameters .
Revised: Then, the knowledge from these different models is aggregated by averaging the parameters .

Original: However, exchanging knowledge via averaging the model parameters is only viable if all the agents use the same model architecture.
Revised: However, exchanging knowledge via averaging the model parameters is only viable if all the agents use the same model architecture.

Original: This fundamental assumption is highly restrictive.
Revised: This fundamental assumption is highly restrictive.

Original: Different agents may have different computational resources and hence may want to use different model architectures.
Revised: Different agents may have different computational resources and hence may want to use different model architectures.

Original: Further, directly averaging the model parameters can fail even when all clients have the same architecture (Wang et al., 2019b;
Revised: Further, directly averaging the model parameters can fail even when all clients have the same architecture (Wang et al., 2019b;

Original: Singh & Jaggi, 2020; Yu et al., 2021).
Revised: Singh & Jaggi, 2020; Yu et al., 2021).

Original: This is because the loss landscape of neural networks is highly non-convex and has numerous symmetries with different parameter values representing the same function.
Revised: This is because the loss landscape of neural networks is highly non-convex and has numerous symmetries with different parameter values representing the same function.

Original: Finally, these methods are also not applicable when using models which are not based on gradient descent such as random forests.
Revised: Finally, these methods are also not applicable when using models which are not based on gradient descent such as random forests.

Original: To overcome such limitations, we would need to take a functional view view of neural networks i.e. we need methods that are agnostic to the model architecture and parameters.
Revised: To overcome such limitations, we would need to take a functional view view of neural networks i.e. we need methods that are agnostic to the model architecture and parameters.

Original: This motivates the central question investigated in this work:
Revised: This motivates the central question investigated in this work:

Original: Can we design model agnostic
Revised: Can we design model agnostic

Original: federated
Revised: federated

Original: learning
Revised: learning

Original: algorithms which
Revised: algorithms which

Original: would allow each
Revised: would allow each

Original: agent to train their model of choice on the combined dataset?
Revised: agent to train their model of choice on the combined dataset?

Original: Speciﬁcally, we restrict the algorithms to access the models using only two primitives (a universal model API): train on some dataset i.e. ﬁt , and yield predictions on some inputs i.e. predict .
Revised: Speciﬁcally, we restrict the algorithms to access the models using only two primitives (a universal model API): train on some dataset i.e. ﬁt , and yield predictions on some inputs i.e. predict .

Original: Our goal is to be able to collaborate with and learn from any agent which provides these two functionalities.
Revised: Our goal is to be able to collaborate with and learn from any agent which provides these two functionalities.

Original: Simple algorithms.
Revised: Simple algorithms.

Original: A naive such model agnostic algorithm indeed exists—agents can simply transfer their entire training data to each other and then each agent can train any model of choice on the combined dataset.
Revised: A naive such model agnostic algorithm indeed exists—agents can simply transfer their entire training data to each other and then each agent can train any model of choice on the combined dataset.

Original: However, transferring of the dataset is disallowed in federated learning.
Revised: However, transferring of the dataset is disallowed in federated learning.

Original: Instead, we will replace the averaging primitive in federated learning with knowledge distillation (KD) (Bucilua et al., 2006; Hinton et al., 2015).
Revised: Instead, we will replace the averaging primitive in federated learning with knowledge distillation (KD) (Bucilua et al., 2006; Hinton et al., 2015).

Original: In knowledge distillation (KD), information is transferred from model A to model B by training model B on the predictions of model A on some data.
Revised: In knowledge distillation (KD), information is transferred from model A to model B by training model B on the predictions of model A on some data.

Original: Since we only access model A through its predictions, KD is a functional model-agnostic protocol.
Revised: Since we only access model A through its predictions, KD is a functional model-agnostic protocol.

Original: The key challenge of KD however is that it is poorly understood and cannot be formulated in the standard stochastic optimization framework like established techniques (Wang et al., 2021).
Revised: The key challenge of KD however is that it is poorly understood and cannot be formulated in the standard stochastic optimization framework like established techniques (Wang et al., 2021).

Original: Thus, designing and analyzing algorithms that utilize KD requires developing an entirely new framework and approach.
Revised: Thus, designing and analyzing algorithms that utilize KD requires developing an entirely new framework and approach.

Original: Our Contributions.
Revised: Our Contributions.

Original: The main results in this work are
Revised: The main results in this work are

Original: • We formulate the model agnostic learning problem as two agents with local datasets wanting to perform kernel regression on their combined datasets.
Revised: • We formulate the model agnostic learning problem as two agents with local datasets wanting to perform kernel regression on their combined datasets.

Original: Kernel regression is both simple enough to be theoretically tractable and rich enough to capture non-linear function ﬁtting thereby allowing each agent to have a different kernel (hence different models).
Revised: Kernel regression is both simple enough to be theoretically tractable and rich enough to capture non-linear function ﬁtting thereby allowing each agent to have a different kernel (hence different models).

Original: • We analyze alternating knowledge distillation (AKD) and show that it is closely linked to the alternating projection method for ﬁnding the intersection of convex sets.
Revised: • We analyze alternating knowledge distillation (AKD) and show that it is closely linked to the alternating projection method for ﬁnding the intersection of convex sets.

Original: Our analysis reveals that AKD sequentially loses information, leading to degradation of performance.
Revised: Our analysis reveals that AKD sequentially loses information, leading to degradation of performance.

Original: This degradation is especially severe when the two agents have heterogeneous data.
Revised: This degradation is especially severe when the two agents have heterogeneous data.

Original: • Using the connection to alternating projection, we analyze other possible variants such as averaged knowledge distillation (AvgKD) and attempt to construct an ‘optimal’ scheme.
Revised: • Using the connection to alternating projection, we analyze other possible variants such as averaged knowledge distillation (AvgKD) and attempt to construct an ‘optimal’ scheme.

Original: Finally, we evaluate all algorithms on real world deep learning models and datasets
Revised: Finally, we evaluate all algorithms on real world deep learning models and datasets

Original: , and show that the empirical behavior closely matches our insights from the theoretical analysis.
Revised: , and show that the empirical behavior closely matches our insights from the theoretical analysis.

Original: This demonstrates the utility of our framework for analyzing and designing new algorithms.
Revised: This demonstrates the utility of our framework for analyzing and designing new algorithms.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Federated learning (FL).
Revised: Federated learning (FL).

Original: In FL (Kairouz et al., 2019), training data is distributed over several agents or locations.
Revised: In FL (Kairouz et al., 2019), training data is distributed over several agents or locations.

Original: For instance, these could be several hospitals collaborating on a clinical trial, or billions of mobile phones involved in training a voice recognition application.
Revised: For instance, these could be several hospitals collaborating on a clinical trial, or billions of mobile phones involved in training a voice recognition application.

Original: The purpose of FL is to enable training on the union of all agents’ individual data without needing to transmit any of the raw sensitive data.
Revised: The purpose of FL is to enable training on the union of all agents’ individual data without needing to transmit any of the raw sensitive data.

Original: Typically, the training is coordinated by some trusted server.
Revised: Typically, the training is coordinated by some trusted server.

Original: One can also instead use direct peer-to-peer communications (Nedic, 2020).
Revised: One can also instead use direct peer-to-peer communications (Nedic, 2020).

Original: A large body of work has designed algorithms for FL under the identical model setting where we either learn a single global model (McMahan et al., 2017;
Revised: A large body of work has designed algorithms for FL under the identical model setting where we either learn a single global model (McMahan et al., 2017;

Original: Reddi et al., 2020; Karimireddy et al., 2020b;a; Wang et al., 2021), or multiple personalized models (Wang et al., 2019a;
Revised: Reddi et al., 2020; Karimireddy et al., 2020b;a; Wang et al., 2021), or multiple personalized models (Wang et al., 2019a;

Original: Deng et al., 2020; Mansour et al., 2020; Grimberg et al., 2021).
Revised: Deng et al., 2020; Mansour et al., 2020; Grimberg et al., 2021).

Original: Knowledge distillation (KD).
Revised: Knowledge distillation (KD).

Original: Initially, KD was introduced as a way to compress models i.e. as a way to transfer the knowledge of a large model to a smaller model (Bucilua et al., 2006; Hinton et al., 2015).
Revised: Initially, KD was introduced as a way to compress models i.e. as a way to transfer the knowledge of a large model to a smaller model (Bucilua et al., 2006; Hinton et al., 2015).

Original: Since then, it has found much broader applications such as improving generalization performance via self-distillation, learning with noisy data, and transfer learning (Yim et al., 2017).
Revised: Since then, it has found much broader applications such as improving generalization performance via self-distillation, learning with noisy data, and transfer learning (Yim et al., 2017).

Original: We refer to a recent survey (Gou et al., 2021) for progress in this vast area.
Revised: We refer to a recent survey (Gou et al., 2021) for progress in this vast area.

Original: KD in FL.
Revised: KD in FL.

Original: Numerous works propose to use KD to transfer knowledge from the agent models to a centralized server model (Seo et al., 2020; Sattler et al., 2020;
Revised: Numerous works propose to use KD to transfer knowledge from the agent models to a centralized server model (Seo et al., 2020; Sattler et al., 2020;

Original: Lin et al., 2020; Li et al., 2020; Wu et al., 2021).
Revised: Lin et al., 2020; Li et al., 2020; Wu et al., 2021).

Original: However, all of these methods rely on access to some common public dataset which may be impractical.
Revised: However, all of these methods rely on access to some common public dataset which may be impractical.

Revised: KD has also been proposed to combine personalization with model compression (Ozkara et al., 2021), but our focus is for the agents to learn on the combined data.

Original: In the closely related codistillation setting (Zhang et al., 2018; Anil et al., 2018; Sodhani et al., 2020), an ensemble of students learns collaboratively without a central server model.
Revised: In the closely related codistillation setting (Zhang et al., 2018; Anil et al., 2018; Sodhani et al., 2020), an ensemble of students learns collaboratively without a central server model.

Original: While codistillation does not need additional unlabelled data, it is only suitable for distributed training within a datacenter since it assumes all agents have access to the same dataset.
Revised: While codistillation does not need additional unlabelled data, it is only suitable for distributed training within a datacenter since it assumes all agents have access to the same dataset.

Original: In FL however, there is both model and data heterogeneity.
Revised: In FL however, there is both model and data heterogeneity.

Original: Further, none of these methods have a theoretical analysis.
Revised: Further, none of these methods have a theoretical analysis.

Original: KD analysis.
Revised: KD analysis.

Original: Despite the empirical success of KD, it is poorly understood with very little theoretical analysis.
Revised: Despite the empirical success of KD, it is poorly understood with very little theoretical analysis.

Original: Phuong & Lampert (2021) explore a generalization bound for a distillation-trained linear model, and Tang et al.
Revised: Phuong & Lampert (2021) explore a generalization bound for a distillation-trained linear model, and Tang et al.

Original: (2021) conclude that by using KD one re-weights the training examples for the student, and Menon et al.
Revised: (2021) conclude that by using KD one re-weights the training examples for the student, and Menon et al.

Original: (2020) consider a Bayesian view showing that the student learns better if the teacher provides the true Bayes probability distribution.
Revised: (2020) consider a Bayesian view showing that the student learns better if the teacher provides the true Bayes probability distribution.

Original: Allen-Zhu & Li (2020) show how ensemble distillation can preserve their diversity in the student model.
Revised: Allen-Zhu & Li (2020) show how ensemble distillation can preserve their diversity in the student model.

Original: Finally, Mobahi et al.
Revised: Finally, Mobahi et al.

Original: (2020) consider self-distillation in a kernel regression setting i.e. the model is retrained using its own predictions on the training data.
Revised: (2020) consider self-distillation in a kernel regression setting i.e. the model is retrained using its own predictions on the training data.

Original: They show that iterative self-distillation induces a strong regularization effect.
Revised: They show that iterative self-distillation induces a strong regularization effect.

Original: We signiﬁcantly extend their theoretical framework in our work to analyze KD in federated learning where agents have different models and different datasets.
Revised: We signiﬁcantly extend their theoretical framework in our work to analyze KD in federated learning where agents have different models and different datasets.

Original:  F RAMEWORK AND SETUP
Revised:  F RAMEWORK AND SETUP

Original: Notation.
Revised: Notation.

Original: We denote a set as A , a matrix as A , and a vector as a .
Revised: We denote a set as A , a matrix as A , and a vector as a .

Original: A [ i, j ] is the (i, j)-th element of matrix A , a [ i ] denotes the i’th element of vector a .
Revised: A [ i, j ] is the (i, j)-th element of matrix A , a [ i ] denotes the i’th element of vector a .

Original: ||
Revised: ||

Original: a || denotes the (cid:96) 2 norm of vector a .
Revised: a || denotes the (cid:96) 2 norm of vector a .

Original: Centralized kernel regression (warmup).
Revised: Centralized kernel regression (warmup).

Original: Consider, as a warm-up, the centralized setting with a training dataset D ⊆
Revised: Consider, as a warm-up, the centralized setting with a training dataset D ⊆

Original: R
Revised: R

Original: d × R .
Revised: d × R .

Original: That is, D = ∪ Ni { ( x i , y i ) } , where x n ∈ X ⊆ R d and y n ∈ Y ⊆ R .
Revised: That is, D = ∪ N i { ( x i , y i ) } , where x n ∈ X ⊆ R d and y n ∈ Y ⊆ R .

Original: Given training set D , our aim is to ﬁnd best function f (cid:63) ∈ F : X → Y .
Revised: Given training set D , our aim is to ﬁnd best function f (cid:63) ∈ F : X → Y .

Original: To ﬁnd f (cid:63) we solve the following regularized optimization problem:
Revised: To ﬁnd f (cid:63) we solve the following regularized optimization problem:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, F is deﬁned to be the space of all functions such that (2) is ﬁnite, c is the regularization parameter, and u ( x , x (cid:48) ) is a kernel function.
Revised: Here, F is deﬁned to be the space of all functions such that (2) is ﬁnite, c is the regularization parameter, and u ( x , x (cid:48) ) is a kernel function.

Original: That is, u is symmetric u ( x , x (cid:48) )
Revised: That is, u is symmetric u ( x , x (cid:48) )

Original: =
Revised: =

Original: u ( x (cid:48) , x ) and positive with R u ( f ) = 0 only when f = 0 and R u ( f ) > 0 o.w.
Revised: u ( x (cid:48) , x ) and positive with R u ( f ) = 0 only when f = 0 and R u ( f ) > 0 o.w.

Original: Further, let k ( x , t ) be the function s.t.
Revised: Further, let k ( x , t ) be the function s.t.

Original: [Equation]
Revised: [Equation]

Original: where δ ( · ) is the Dirac delta function.
Revised: where δ ( · ) is the Dirac delta function.

Original: [Equation]
Revised: [Equation]

Original: Now, we can deﬁne the positive deﬁnite matrix K ∈ R N × N and vector k x ∈ R N as: 
Revised: Now, we can deﬁne the positive deﬁnite matrix K ∈ R N × N and vector k x ∈ R N as: K [ i [Equation] [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original:  Note that k x is actually a vector valued function which takes any x ∈ X as input, and both k x and
Revised: [Equation] , for x i ∈ D , ∀ i ∈ [ N ] . Note that k x is actually a vector valued function which takes any x ∈ X as input, and both k x and

Original: K depend on the training data D .
Revised: K depend on the training data D .

Original: We can then derive a closed form solution for f (cid:63) .
Revised: We can then derive a closed form solution for f (cid:63) .

Original: Proposition I (Schölkopf et al.
Revised: Proposition I (Schölkopf et al.

Original: The f (cid:63) which minimizes (1) is given by
Revised: The f (cid:63) which minimizes (1) is given by

Original: [Equation]
Revised: [Equation]

Original:  Note that on the training data X ∈ R N × d with X [ i, :]
Revised: for [Equation] N ] . Note that on the training data X ∈ R N × d with X [ i, :]

Original: = x i , we have f (cid:63) ( X ) =
Revised: = x i , we have f (cid:63) ( X ) =

Original: K ( c I + K ) − 1 y .
Revised: K ( c I + K ) − 1 y .

Original: Kernel regression for an input x outputs a weighted average of the training { y n } .
Revised: Kernel regression for an input x outputs a weighted average of the training { y n } .

Original: These weights are computed using a learned measure of distance between the input x and the training { x n } .
Revised: These weights are computed using a learned measure of distance between the input x and the training { x n } .

Original: Intuitively, the choice of the kernel u ( x , x (cid:48) ) creates an inductive bias and corresponds to the choice of a model in deep learning, and the regularization parameter c acts similarly to tricks like early stopping, large learning rate, etc.
Revised: Intuitively, the choice of the kernel u ( x , x (cid:48) ) creates an inductive bias and corresponds to the choice of a model in deep learning, and the regularization parameter c acts similarly to tricks like early stopping, large learning rate, etc.

Original: which help in generalization.
Revised: which help in generalization.

Original: When c = 0 , we completely ﬁt the training data and the predictions exactly recover the labels with f (cid:63) ( X )
Revised: When c = 0 , we completely ﬁt the training data and the predictions exactly recover the labels with f (cid:63) ( X )

Original: =
Revised: =

Original: K ( K ) − 1 y = y .
Revised: K ( K ) − 1 y = y .

Original: When c > 0 the predictions f (cid:63) ( X )
Revised: When c > 0 the predictions f (cid:63) ( X )

Original: =
Revised: =

Original: K ( c I + K ) − 1 y
Revised: K ( c I + K ) − 1 y

Original: (cid:54) =
Revised: (cid:54) =

Original: y
Revised: y

Original: and they incorporate the inductive bias of the model.
Revised: and they incorporate the inductive bias of the model.

Original: In knowledge distillation, this extra information carried by the predictions about the inductive bias of the model is popularly referred to as “dark knowledge” (Hinton et al., 2015).
Revised: In knowledge distillation, this extra information carried by the predictions about the inductive bias of the model is popularly referred to as “dark knowledge” (Hinton et al., 2015).

Original: Federated kernel regression (our setting).
Revised: Federated kernel regression (our setting).

Original: We have two agents, with agent 1 having dataset
Revised: We have two agents, with agent 1 having dataset

Original: D 1 = ∪ N 

Revised: D 1 = ∪ N

Original: i { ( x 1 i , y 1 i ) } and agent 2 with dataset D 2 = ∪ N 2 i { ( x 2 i , y 2 i ) } .
Revised: i { ( x 1 i , y 1 i ) } and agent 2 with dataset D 2 = ∪ N 2 i { ( x 2 i , y 2 i ) } .

Original: Agent 1 aims to ﬁnd the best approximation mapping g 1 (cid:63)∈ F 1 : X → Y using a kernel u 1 ( x , x (cid:48) ) and objective:
Revised: Agent 1 aims to ﬁnd the best approximation mapping g 1 (cid:63)

Original: [Equation]
Revised: [Equation]

Original:  , with
Revised: F 1 : X → Y using a kernel u 1 ( x , x (cid:48) ) and objective:

Original: [Equation]
Revised: [Equation]

Original: Note that the objective of agent 1 is deﬁned using its individual kernel u 1 ( x , x (cid:48) ) , but over the joint dataset ( D 1 , D 2 ) .
Revised: Note that the objective of agent 1 is deﬁned using its individual kernel u 1 ( x , x (cid:48) ) , but over the joint dataset ( D 1 , D 2 ) .

Original: Correspondingly, agent 2 also uses its own kernel function u 2 ( x , x (cid:48) ) to deﬁne the regularizer R u 2 ( g 2 ) over the space of functions g 2 ∈ F 2 and optimum g 2 (cid:63) .
Revised: Correspondingly, agent 2 also uses its own kernel function u 2 ( x , x (cid:48) ) to deﬁne the regularizer R u 2 ( g 2 ) over the space of functions g 2 ∈ F 2 and optimum g 2 (cid:63) .

Original: Thus, our setting has model heterogeneity (different kernel functions u 1 and u 2 ) and data heterogeneity ( D 1 and D 2 are not i.i.d.).
Revised: Thus, our setting has model heterogeneity (different kernel functions u 1 and u 2 ) and data heterogeneity ( D 1 and D 2 are not i.i.d.).

Original: Given that the setting is symmetric between agents 1 and 2, we can focus solely on error in terms of agent 1’s objective (5) without loss of generality.
Revised: Given that the setting is symmetric between agents 1 and 2, we can focus solely on error in terms of agent 1’s objective (5) without loss of generality.

Original: Proposition I can be used to derive a closed form form for function g 1 (cid:63) minimizing objective (5).
Revised: Proposition I can be used to derive a closed form form for function g 1 (cid:63) minimizing objective (5).

Original: However, computing this requires access to the datasets of both agents.
Revised: However, computing this requires access to the datasets of both agents.

Original: Instead, we ask “can we design an iterative federated learning algorithm which can approximate g 1 (cid:63) ”?
Revised: Instead, we ask “can we design an iterative federated learning algorithm which can approximate g 1 (cid:63) ”?

Original:  A LTERNATING KNOWLEDGE DISTILLATION
Revised:  A LTERNATING KNOWLEDGE DISTILLATION

Original: In this section, we describe a popular iterative knowledge distillation algorithm and analyze its updates in our framework.
Revised: In this section, we describe a popular iterative knowledge distillation algorithm and analyze its updates in our framework.

Original: Our analysis leads to some surprising connections between KD and projection onto convex sets, and shows some limitations of the current algorithm.
Revised: Our analysis leads to some surprising connections between KD and projection onto convex sets, and shows some limitations of the current algorithm.

Original: Algorithm.
Revised: Algorithm.

Original: Denote the data on agent 1 as D 1 =
Revised: Denote the data on agent 1 as D 1 =

Original: ( X 1 , y 1 ) where X 

Revised: ( X 1 , y 1 ) where X

Original: [ i, :] = x 1 n and y 1 [ i ] =
Revised: [ i, :] = x 1 n and y 1 [ i ] =

Original: y 1 i .
Revised: y 1 i .

Original: Correspondingly, we have D 2 =
Revised: Correspondingly, we have D 2 =

Original: ( X 2 , y 2 ) .
Revised: ( X 2 , y 2 ) .

Original: Now starting from ˆ y 10 =
Revised: Now starting from ˆ y 10 =

Original: y 1 , in each rounds t and t + 1 : 
Revised: y 1 , in each rounds t and t + 1 : a.

Original: a. Agent1 trains their model on dataset( X 1 , ˆ y 1 t) to obtain g 1 t .
Revised: Agent trains their model on dataset X 1 , ˆ y 1 t to obtain g 1 t .

Revised: b.

Original: b. Agent 2 receives g 1 t and uses it to predict labels ˆ y 2 t
Revised: Agent 2 receives g 1 t and uses it to predict labels ˆ y 2 t

Original: +1 =
Revised: +1 =

Original: g 1 t ( X 2 ) .
Revised: g 1 t ( X 2 ) .

Original: c. Agent 2 trains their model on dataset ( X 2 , ˆ y 2 t +1 ) to obtain g 1 t +1 .
Revised: c. Agent 2 trains their model on dataset ( X 2 , ˆ y 2 t +1 ) to obtain g 1 t +1 .

Original: d. Agent 1 receives a model g 1 t +1 from agent 2 and predicts ˆ y 1 t
Revised: d. Agent 1 receives a model g 1 t +1 from agent 2 and predicts ˆ y 1 t

Original: +2 =
Revised: +2 =

Original: g 1 t
Revised: g 1 t

Original: +

Revised: +

Original: X 1 ) .
Revised: X 1 ) .

Original: Thus the algorithm alternates between training and knowledge distillation on each of the two agents.
Revised: Thus the algorithm alternates between training and knowledge distillation on each of the two agents.

Original: We also summarize the algorithm in Figure 1a.
Revised: We also summarize the algorithm in Figure 1a.

Original: Importantly, note that there is no exchange of raw data but only of the trained models.
Revised: Importantly, note that there is no exchange of raw data but only of the trained models.

Original: Further, each agent trains their choice of a model on their data with agent 1 training { g 1 t } and agent 2 training { g 1 t +1 } .
Revised: Further, each agent trains their choice of a model on their data with agent 1 training { g 1 t } and agent 2 training { g 1 t +1 } .

Original: Superscript 1 means we start AKD from agent 1.
Revised: Superscript 1 means we start AKD from agent 1.

Original: 4. T HEORETICAL ANALYSIS
Revised: 4. T HEORETICAL ANALYSIS

Original: Similar to (3), let us deﬁne functions k 1 ( x , x (cid:48) ) and k 2 ( x , x (cid:48) )
Revised: Similar to (3), let us deﬁne functions k 1 ( x , x (cid:48) ) and k 2 ( x , x (cid:48) )

Original: such that they satisfy
Revised: such that they satisfy

Original: [Equation]
Revised: [Equation]

Original: For such functions, we can then deﬁne the following positive deﬁnite matrix L ∈R ( N 1 + N 2 ) × ( N 1 + N 2 ) : L [Equation]
Revised: a ∈

Original: [Equation]
Revised: [Equation]

Original:  N 
Revised: For such functions, we can then deﬁne the following positive deﬁnite matrix L ∈ R ( N 1 + N 2 ) ×

Original: [Equation] N 
Revised: ( N 1 + N 2 ) :

Original: [Equation]
Revised: [Equation]

Original: k 1 ( x ai , x bj ) for a, b ∈ { 1 , 2 } and i ∈ [ N 1 ] , j ∈ [ N 2 ] .
Revised: for a, b ∈ { 1 , 2 } and i ∈ [ N 1 ] , j ∈ [ N 2 ] .

Original: Note L is symmetric (with L (cid:62) 

Revised: Note L is symmetric (with L (cid:62)

Original: = L 21 ) and is also positive deﬁnite.
Revised: = L 21 ) and is also positive deﬁnite.

Original: Further, each component
Revised: Further, each component

Original: L a,b measures pairwise similarities between inputs of agent a and agent b using the kernel k 1 .
Revised: L a,b measures pairwise similarities between inputs of agent a and agent b using the kernel k 1 .

Original: Correspondingly, we deﬁne M ∈ R ( N 1 + N 2 ) ×
Revised: Correspondingly, we deﬁne M ∈ R ( N 1 + N 2 ) ×

Original: ( N 1 + N 2 ) which uses kernel k 2 :
Revised: ( N 1 + N 2 ) which uses kernel k 2 :

Original: [Equation]
Revised: [Equation]

Original: We can now derive the closed form of the AKD algorithm repeatedly using Proposition I.
Revised: We can now derive the closed form of the AKD algorithm repeatedly using Proposition I.

Original: Proposition II.
Revised: Proposition II.

Original: The model in round 2 t learned by the alternating knowledge distillation algorithm is
Revised: The model in round 2 t learned by the alternating knowledge distillation algorithm is

Original: [Equation]
Revised: [Equation]

Original: where l x ∈ R N 1 is deﬁned as l x [ i ] = 1 N 1 + N 2 k 1 ( x , x 1 i ) .
Revised: where l x ∈ R N 1 is deﬁned as l x [ i ] = [Equation] N [Equation]

Revised: 1 ( x , x 1 i ) .

Original: Further, for any ﬁxed x we have
Revised: Further, for any ﬁxed x we have

Original: [Equation]
Revised: [Equation]

Original: First, note that if agents 1 and 2 are identical with the same data and same model, we have M 12 =
Revised: First, note that if agents 1 and 2 are identical with the same data and same model, we have M 12 =

Original: M 

Revised: M

Original: =
Revised: =

Original: L 11 =
Revised: L 11 =

Original: L 12 .
Revised: L 12 .

Original: This setting corresponds to self-distillation where the model is repeatedly retrained on its own predictions.
Revised: This setting corresponds to self-distillation where the model is repeatedly retrained on its own predictions.

Original: Proposition II shows that after 2 t rounds of self-distillation, we obtain a model is of the form g 12 t ( x )
Revised: Proposition II shows that after 2 t rounds of self-distillation, we obtain a model is of the form g 12 t ( x )

Original: = l (cid:62) x
Revised: = l (cid:62) x

Original: ( c I + L 11 ) − 1 (cid:16) L 11 ( c I + L 11 ) − 1 (cid:17) 2 t y .
Revised: ( c I + L 11 ) − 1 (cid:16) L 11 ( c I + L 11 ) − 1 (cid:17) 2 t y .

Original: Here, the effect of c is ampliﬁed as t increases.
Revised: Here, the effect of c is ampliﬁed as t increases.

Original: Thus, this shows that repeated self-distillation induces a strong regularization effect, recovering the results of (Mobahi et al., 2020).
Revised: Thus, this shows that repeated self-distillation induces a strong regularization effect, recovering the results of (Mobahi et al., 2020).

Original: Perhaps more strikingly, Proposition II shows that not only does AKD fail to converge to the actual optimal solution g 1 (cid:63) as deﬁned in (5), it will also slowly degrade and eventually converges to 0.
Revised: Perhaps more strikingly, Proposition II shows that not only does AKD fail to converge to the actual optimal solution g 1 (cid:63) as deﬁned in (5), it will also slowly degrade and eventually converges to 0.

Original: We next expand upon this and explain this phenomenon.
Revised: We next expand upon this and explain this phenomenon.

Original: 4. D EGRADATION AND CONNECTION TO PROJECTIONS
Revised: 4. D EGRADATION AND CONNECTION TO PROJECTIONS

Original: While mathematically, Proposition II completely describes the AKD algorithm, it does not provide much intuition.
Revised: While mathematically, Proposition II completely describes the AKD algorithm, it does not provide much intuition.

Original: In this section, we rephrase the result in terms of projections and contractions which provides a more visual understanding of the method.
Revised: In this section, we rephrase the result in terms of projections and contractions which provides a more visual understanding of the method.

Original: Oblique projections.
Revised: Oblique projections.

Original: A projection operator P linearly maps (projects) all inputs onto some linear subspace A =
Revised: A projection operator P linearly maps (projects) all inputs onto some linear subspace A =

Original: Range ( A ) .
Revised: Range ( A ) .

Original: In general, we can always rewrite such a projection operation as
Revised: In general, we can always rewrite such a projection operation as

Original: [Equation]
Revised: [Equation]

Original: Here, A deﬁnes an orthonormal basis of the space A and W is a positive deﬁnite weight matrix which deﬁnes the geometry (cid:104) x , y (cid:105) W :=
Revised: Here, A deﬁnes an orthonormal basis of the space A and W is a positive deﬁnite weight matrix which deﬁnes the geometry (cid:104) x , y (cid:105) W :=

Original: x (cid:62) W y .
Revised: x (cid:62) W y .

Original: When W =
Revised: When W =

Original: I , we recover the familiar orthogonal projection.
Revised: I , we recover the familiar orthogonal projection.

Original: Otherwise, projections can happen ‘obliquely’ following the geometry deﬁned by W .
Revised: Otherwise, projections can happen ‘obliquely’ following the geometry deﬁned by W .

Original: Contractions.
Revised: Contractions.

Original: A contraction is a linear operator C which contracts all inputs towards the origin:
Revised: A contraction is a linear operator C which contracts all inputs towards the origin:

Original: [Equation]
Revised: [Equation]

Original: any x .
Revised: x .

Original: Given these notions, we can rewrite Proposition II as follows.
Revised: Given these notions, we can rewrite Proposition II as follows.

Original: Proposition III.
Revised: Proposition III.

Original: There exist oblique projection operators P(cid:62) 1 and P(cid:62) 2 , contraction matrices C 1 and
Revised: There exist oblique projection operators P 1 and P 2 , contraction matrices C 1 and

Original: C 2 , and orthonormal matrices V 1 and V 2 such that the model in round 2 t learned by the alternating knowledge distillation algorithm is
Revised: C 2 , and orthonormal matrices V 1 and V 2 such that the model in round 2 t learned by the alternating knowledge distillation algorithm is

Original: [Equation]
Revised: [Equation]

Original: In particular, the predictions on agent 2’s inputs are
Revised: In particular, the predictions on agent 2’s inputs are

Original: [Equation]
Revised: [Equation]

Original: Further, the projection matrices satisfy P (cid:62) 1 P (cid:62) 2 x = x only if x = 0 .
Revised: Further, the projection matrices satisfy P (cid:62) 1 P (cid:62) 2 x = x only if x = 0 .

Original: The term (cid:0) C 1 P (cid:62) 2 C 2 P (cid:62) 1 (cid:1) t is the only one which depends on t and so captures the dynamics of the algorithm.
Revised: The term (cid:0) C 1 P (cid:62) 2 C 2 P (cid:62) 1 (cid:1) t is the only one which depends on t and so captures the dynamics of the algorithm.

Original: Two rounds of AKD (ﬁrst to agent 1 and then back to 2) correspond to a multiplication with C 1 P (cid:62) 2 C 2 P (cid:62) 1 i.e. alternating projections interspersed by contractions.
Revised: Two rounds of AKD (ﬁrst to agent 1 and then back to 2) correspond to a multiplication with C 1 P (cid:62) 2 C 2 P (cid:62) 1 i.e. alternating projections interspersed by contractions.

Original: Thus, the dynamics of AKD is exactly the same as that of alternating projections interspersed by contractions.
Revised: Thus, the dynamics of AKD is exactly the same as that of alternating projections interspersed by contractions.

Original: The projections P (cid:62) 1 and P (cid:62) 2 have orthogonal ranges whose intersection only contains the origin 0.
Revised: The projections P (cid:62) 1 and P (cid:62) 2 have orthogonal ranges whose intersection only contains the origin 0.

Original: As
Revised: As

Original: Fig.
Revised: Fig.

Original: 1b shows, non-orthogonal alternating projections between orthogonal spaces converge to the origin.
Revised: 1b shows, non-orthogonal alternating projections between orthogonal spaces converge to the origin.

Original: Contractions further pull the inputs closer to the origin, speeding up the convergence.
Revised: Contractions further pull the inputs closer to the origin, speeding up the convergence.

Original: Remark 1.
Revised: Remark 1.

Original: To understand the connection to projection more intuitively, suppose that we had a 4-way classiﬁcation task with agent 1 having data only of the ﬁrst two classes, and agent 2 with the last two classes.
Revised: To understand the connection to projection more intuitively, suppose that we had a 4-way classiﬁcation task with agent 1 having data only of the ﬁrst two classes, and agent 2 with the last two classes.

Original: Any model trained by agent 1 will only learn about the ﬁrst two classes and its predictions on the last two classes will be meaningless.
Revised: Any model trained by agent 1 will only learn about the ﬁrst two classes and its predictions on the last two classes will be meaningless.

Original: Thus, no information can be transferred between the agents in this setting.
Revised: Thus, no information can be transferred between the agents in this setting.

Original: More generally, the transfer of knowledge from agent 1 to agent 2 is mediated by its data X 2 .
Revised: More generally, the transfer of knowledge from agent 1 to agent 2 is mediated by its data X 2 .

Original: This corresponds to a projection of the knowledge of agent 1 onto the data of agent 2.
Revised: This corresponds to a projection of the knowledge of agent 1 onto the data of agent 2.

Original: If there is a mismatch between the two, knowledge is bound to be lost.
Revised: If there is a mismatch between the two, knowledge is bound to be lost.

Revised: [Figure]

Revised: rounds

Original: Speed of degradation.
Revised: [Table] [Figure] Train [Table]

Revised: round

Revised: [Table]

Revised: [Table]

Revised: Speed of degradation.

Original: The alternating projection algorithm converges to the intersection of the subspaces corresponding to the projection operators (their range)
Revised: The alternating projection algorithm converges to the intersection of the subspaces corresponding to the projection operators (their range)

Original: (Boyd & Dattorro, 2003).
Revised: (Boyd & Dattorro, 2003).

Original: In our particular case, the ﬁxed point of P (cid:62) 1 and P (cid:62) 2 is 0, and hence this is the point the algorithm will converge to.
Revised: In our particular case, the ﬁxed point of P (cid:62) 1 and P (cid:62) 2 is 0, and hence this is the point the algorithm will converge to.

Original: The contraction operations C 1 and C 2 only speed up the convergence to the origin(also see Fig. 1b).
Revised: The contraction operations C 1 and C 2 only speed up the convergence to the origin(also see Fig. 1b).

Original: This explains the degradation process notes in Proposition II.
Revised: This explains the degradation process notes in Proposition II.

Original: We can go further and examine the rate of degradation using known analyses of alternating projections (Aronszajn, 1950).[Figure] Train rounds [Table]
Revised: We can go further and examine the rate of degradation using known analyses of alternating projections (Aronszajn, 1950).

Original: Train

Original: Train

Original: [Table]

Original: Proposition IV (Informal) .
Revised: Proposition IV (Informal) .

Original: The rate of convergence to g 1 t ( x ) to 0 gets faster if:
Revised: The rate of convergence to g 1 t ( x ) to 0 gets faster if:

Original: • a stronger inductive bias is induced via a larger regularization constant c , • the kernels k 1 ( x , y ) and k 2 ( x , y ) are very different, or • the difference between the datasets D 1 and D 2 as measured by k 1 ( x , y ) increases.
Revised: • a stronger inductive bias is induced via a larger regularization constant c , • the kernels k 1 ( x , y ) and k 2 ( x , y ) are very different, or • the difference between the datasets D 1 and D 2 as measured by k 1 ( x , y ) increases.

Original: In summary, both data and model heterogeneity may speed up the degradation defeating the purpose of model agnostic FL.
Revised: In summary, both data and model heterogeneity may speed up the degradation defeating the purpose of model agnostic FL.

Original: All formal proofs and theorem statements are moved to the Appendix.
Revised: All formal proofs and theorem statements are moved to the Appendix.

Original:  A DDITIONAL V ARIANTS
Revised:  A DDITIONAL V ARIANTS

Original: In the previous section, we saw that the alternating knowledge distillation (AKD) algorithm over multiple iterations suffered slow degradation, eventually losing all information about the training data.
Revised: In the previous section, we saw that the alternating knowledge distillation (AKD) algorithm over multiple iterations suffered slow degradation, eventually losing all information about the training data.

Original: In this section, we explore some alternative approaches which attempt to correct this.
Revised: In this section, we explore some alternative approaches which attempt to correct this.

Original: We ﬁrst analyze a simple way to re-inject the training data after every KD iteration which we call averaged distillation.
Revised: We ﬁrst analyze a simple way to re-inject the training data after every KD iteration which we call averaged distillation.

Original: Then, we show an ensemble algorithm that can recover the optimal model g 1 (cid:63) .
Revised: Then, we show an ensemble algorithm that can recover the optimal model g 1 (cid:63) .

Original: 5. A VERAGED KNOWLEDGE DISTILLATION
Revised: 5. A VERAGED KNOWLEDGE DISTILLATION

Original: As we saw earlier, each step of knowledge distillation seems to lose some information about the training data, replacing it with the inductive bias of the model.
Revised: As we saw earlier, each step of knowledge distillation seems to lose some information about the training data, replacing it with the inductive bias of the model.

Original: One approach to counter this slow loss of information is to recombine it with the original training data labels such as is commonly done in co-distillation (Sodhani et al., 2020).
Revised: One approach to counter this slow loss of information is to recombine it with the original training data labels such as is commonly done in co-distillation (Sodhani et al., 2020).

Original: Algorithm.
Revised: Algorithm.

Original: Recall that agent 1 has data D 1 =
Revised: Recall that agent 1 has data D 1 =

Original: ( X 1 , y 1 ) and correspondingly agent 2 has data
Revised: ( X 1 , y 1 ) and correspondingly agent 2 has data

Original: D 2 =
Revised: D 2 =

Original:  X 2 , y 2 ) .
Revised: ( X 2 , y 2 ) .

Original: Now starting from ˆ y 10 =
Revised: Now starting from ˆ y 10 =

Original: y 1 , ˆ y 20 =
Revised: y 1 , ˆ y 20 =

Original: y 2 , in each round t ≥ 
Revised: y 2 , in each round t ≥ 0 : a

Original: : a. Agents 1 and 2 train their model on datasets( X 1 , ˆ y 1 t) and ( X 2 , ˆ y 2 t ) toobtain g 1 t and g 2 t .
Revised: Agents 1 and 2 train their model on datasets X 1 , ˆ y 1 t and ( X 2 , ˆ y 2 t ) to

Original: b. Agents exchange their models g 1 t and g 2 tbetween each other.
Revised: obtain g 1 t and g 2 t

Original:  c. Agents use exchanged models to predict labelsˆ y
Revised: . c. Agents use exchanged models to predict labels

Original: [Equation]

Original: [Equation]

Original: 2 t

Original: [Equation]
Revised: [Equation]

Original: The summary the algorithm is depicted in Figure 2a.
Revised: The summary the algorithm is depicted in Figure 2a.

Original: Again, notice that there is no exchange of raw data but only of the trained models.
Revised: Again, notice that there is no exchange of raw data but only of the trained models.

Original: The main difference between AKD and AvgKD (averaged knowledge distillation) is that we average the predictions with the original labels.
Revised: The main difference between AKD and AvgKD (averaged knowledge distillation) is that we average the predictions with the original labels.

Original: This re-injects information y 1 and y 2 at every iteration, preventing degradation.
Revised: This re-injects information y 1 and y 2 at every iteration, preventing degradation.

Original: We theoretically characterize its dynamics next in terms of the afore-mentioned contraction and projection operators.
Revised: We theoretically characterize its dynamics next in terms of the afore-mentioned contraction and projection operators.

Original: Proposition V. There exist oblique projection operators P 1 and P 2 , contraction matrices C 1 and
Revised: Proposition V. There exist oblique projection operators P 1 and P 2 , contraction matrices C 1 and

Original: C 2 , and orthonormal matrices V 1 and V 2 such that the model of agent 1 in round t learned by the averaged knowledge distillation (AvgKD) algorithm is
Revised: C 2 , and orthonormal matrices V 1 and V 2 such that the model of agent 1 in round t learned by the averaged knowledge distillation (AvgKD) algorithm is

Original: [Equation]
Revised: [Equation]

Original: where F = l (cid:62) x ( c I + L 11 ) − 1 V (cid:62) 1 .
Revised: where F = l (cid:62) x ( c I + L 11 ) − 1 V (cid:62) 1 .

Original: Further, in the limit of rounds for any ﬁxed x we have
Revised: Further [Equation] in the limit of rounds for any ﬁxed x we have

Original: [Equation]
Revised: [Equation]

Original: This shows that the model learned through AvgKD does not degrade to 0, unlike AKD.
Revised: This shows that the model learned through AvgKD does not degrade to 0, unlike AKD.

Original: Instead, it converges to a limit for which we can derive closed-form expressions.
Revised: Instead, it converges to a limit for which we can derive closed-form expressions.

Original: Unfortunately, this limit model is still not the same as our desired optimal model g 1 (cid:63) .
Revised: Unfortunately, this limit model is still not the same as our desired optimal model g 1 (cid:63) .

Original: We next try to overcome this using ensembling.
Revised: We next try to overcome this using ensembling.

Original:  E NSEMBLED K NOWLEDGE D ISTILLATION
Revised: 5. E NSEMBLED K NOWLEDGE D ISTILLATION

Original: We ﬁrst analyze how the limit solution of AvgKD differs from the actual optimum g 1 (cid:63) .
Revised: We ﬁrst analyze how the limit solution of AvgKD differs from the actual optimum g 1 (cid:63) .

Original: We will build upon this understanding to construct an ensemble that approximates g 1 (cid:63) .
Revised: We will build upon this understanding to construct an ensemble that approximates g 1 (cid:63) .

Revised: For simplicity, we assume that the regularization coefﬁcient c = 0 .

Original: Understanding AvgKD.
Revised: Understanding AvgKD.

Original: For simplicity, we assume that the regularization coefﬁcient c = 0 . Then,
Revised: Consider AvgKD algorithm, then

Original: Proposition VI.
Revised: Proposition VI.

Original: There exist matrices A 1 and A 2 such that the minimizer g 1 (cid:63) of objective (5) predicts g 1 (cid:63) ( X i ) =
Revised: There exist matrices A 1 and A 2 such that the minimizer g 1 (cid:63) of objective (5) predicts g 1 (cid:63) ( [Equation]

Original:  A 1 β 1 + A 2 β 2 for i ∈ { 1 , 2 } , where β 1 and β 2 satisfy
Revised: i ) = A 1 β 1 + A 2 β 2 for i ∈ { 1 , 2 } , where β 1 and β 2 satisfy

Original: [Equation]
Revised: [Equation]

Original: In contrast, for the same matrices A 1 and A 2 , the limit models of AvgKD predict g 1 ∞ ( X i ) =
Revised: In contrast, for the same matrices A 1 and A 2 , the limit models of AvgKD predict g 1 ∞ ( X i ) =

Original: 12 A 1 βand g 2 ∞ ( X i ) = − 

Revised: 12 A 1 βand g 2 ∞ ( X i ) = −

Original: A 2 β 2 , for i ∈ { 1 , 2 } for β 1 and β 2 satisfying (cid:18)
Revised: A 2 β 2 , for i ∈ { 1 , 2 } for β 1 and β 2 satisfying (cid:18)

Original: [Equation]
Revised: [Equation]

Original: By comparing the equations (7) and (8), the output 2( g 1 ∞ ( x ) − g 2 ∞ ( x )) is close to the output of g 1 (cid:63) ( x ) , except that the off-diagonal matrices are scaled by 12 and we have − y 2 on the right hand side.
Revised: By comparing the equations (7) and (8), the output 2( g 1 ∞ ( x ) − g 2 ∞ ( x )) is close to the output of g 1 (cid:63) ( x ) , except that the off-diagonal matrices are scaled by 12 and we have − y 2 on the right hand side.

Original: We need two tricks if we want to approximate g 1 (cid:63) :
Revised: We need two tricks if we want to approximate g 1 (cid:63) :

Original: ﬁrst we need an ensemble using differences of models, and second we need to additionally correct for the bias in the algorithm.
Revised: ﬁrst we need an ensemble using differences of models, and second we need to additionally correct for the bias in the algorithm.

Original: Correcting bias using inﬁnite ensembles.
Revised: Correcting bias using inﬁnite ensembles.

Original: Consider the initial AKD (alternating knowledge distillation) algorithm illustrated in the Fig.
Revised: Consider the initial AKD (alternating knowledge distillation) algorithm illustrated in the Fig.

Original: 1a.
Revised: 1a.

Original: Let us run two simultaneous runs, ones starting from agent 1 and another starting from agent 2, outputting models { g 1 t } , and { g 2 t } respectively.
Revised: Let us run two simultaneous runs, ones starting from agent 1 and another starting from agent 2, outputting models { g 1 t } , and { g 2 t } respectively.

Original: Then, instead of just using the ﬁnal models, we construct the following inﬁnite ensemble.
Revised: Then, instead of just using the ﬁnal models, we construct the following inﬁnite ensemble.

Original: For an input x , we output:
Revised: For an input x , we output:

Original: [Equation]
Revised: [Equation]

Original: That is, we take the models from odd steps t with positive signs and from even ones with negative signs and sum their predictions.
Revised: That is, we take the models from odd steps t with positive signs and from even ones with negative signs and sum their predictions.

Original: We call this scheme Ensembled Knowledge Distillation (EKD).
Revised: We call this scheme Ensembled Knowledge Distillation (EKD).

Original: The intuition behind our ensemble method is schematically visualized in 1-dimensional case in the Fig.
Revised: The intuition behind our ensemble method is schematically visualized in 1-dimensional case in the Fig.

Original: 2b where numbers denote the t variable in the equation (9).
Revised: 2b where numbers denote the t variable in the equation (9).

Original: We start from the sum of both agents models obtained after learning from ground truth labels (0-th round).
Revised: We start from the sum of both agents models obtained after learning from ground truth labels (0-th round).

Original: Then we subtract the sum of  both agents models obtained after the ﬁrst round of KD.
Revised: Then we subtract the sum of [Figure] both agents models obtained after the ﬁrst round of KD.

Original: Then we add the sum of both agents models obtained after the second round of KD and so on.
Revised: Then we add the sum of both agents models obtained after the second round of KD and so on.

Original: From the section 4.2 we know that in the AKD process  model gradually degrades towards 0 , in other words,  each next round obtained model adds less value to the whole sum.
Revised: From the section 4.2 we know that in the AKD process with regularization model gradually degrades towards 0 , in other words, intuitively each next round obtained model adds less value to the whole sum in the EKD scheme.

Revised: Although, in the lack of regularization models degradation towards 0 is not always the case (see App. C).

Original: Hence we gradually converge to the limit model, which is the point ∞ in the Fig.
Revised: But under such an assumption we we gradually converge to the limit model, which is the point ∞ in the Fig.

Original: 2b.
Revised: 2b.

Original: We formalize this and prove the following.
Revised: We formalize this and prove the following.

Original: Proposition VII.
Revised: Proposition VII.

Original: The predictions of f ∞ using (9) satisﬁes f ∞ ( X i ) =
Revised: The predictions of f ∞ using (9) satisﬁes f ∞ ( X i ) =

Original: g 1 (cid:63) ( X i ) for i ∈ { 1 , 2 } .
Revised: g 1 (cid:63) ( X i ) for i ∈ { 1 , 2 } .

Original: Thus, not only did we succeed in preventing degeneracy to 0, but we also managed to recover the predictions of the optimal model.
Revised: Thus, not only did we succeed in preventing degeneracy to 0, but we also managed to recover the predictions of the optimal model.

Original: However, note that this comes at a cost of an inﬁnite ensemble.
Revised: However, note that this comes at a cost of an inﬁnite ensemble.

Original: While we can approximate this using just a ﬁnite set of models (as we explore experimentally next), this still does not recover a single model which matches g 1 (cid:63) .[Figure] [Figure]
Revised: While we can approximate this using just a ﬁnite set of models (as we explore experimentally next), this still does not recover a single model which matches g 1 (cid:63) .

Original:  E XPERIMENTS 6. S ETUP
Revised:  E XPERIMENTS 6. S ETUP

Original: We consider three settings corresponding to the cases Proposition IV with the agents having
Revised: We consider three settings corresponding to the cases Proposition IV with the agents having

Original: • the same model architecture and close data distributions (Same model, Same data) • different model architectures and close data distributions (Different model, Same data) • the same model architecture and different data distributions (Same model, Different data).
Revised: • the same model architecture and close data distributions (Same model, Same data) • different model architectures and close data distributions (Different model, Same data) • the same model architecture and different data distributions (Same model, Different data).

Original: The toy experiments solve a linear regression problem of the form Ax (cid:63)
Revised: The toy experiments solve a linear regression problem of the form Ax (cid:63)

Original: = b .
Revised: = b .

Original: The data A and b is split between the two agents randomly in the ‘same data’ case, whereas in the ‘different data’ case the data is sorted according to b before splitting to maximize heterogeneity.
Revised: The data A and b is split between the two agents randomly in the ‘same data’ case, whereas in the ‘different data’ case the data is sorted according to b before splitting to maximize heterogeneity.

Original: The real world experiments are conducted using Convolutional Neural Network (CNN), Multi-Layer
Revised: The real world experiments are conducted using Convolutional Neural Network (CNN), Multi-Layer

Original: Perceptron network (MLP), and Random Forest (RF).
Revised: Perceptron network (MLP), and Random Forest (RF).

Original: We use squared loss since it is closer to the theoretical setting.
Revised: We use squared loss since it is closer to the theoretical setting.

Original: In the ‘same model’ setting both agents use the CNN model, whereas agentinstead uses an MLP in the ‘different model’ setting.
Revised: In the ‘same model’ setting both agents use the CNN model, whereas agentinstead uses an MLP in the ‘different model’ setting.

Original: Further, we split the training data randomly in the ’same data’ setting.
Revised: Further, we split the training data randomly in the ’same data’ setting.

Original: For the ’different data’ setting, we split the data by labels and take some portion Alpha of data from each agent and randomly shufﬂe taken points between agents.
Revised: For the ’different data’ setting, we split the data by labels and take some portion Alpha of data from each agent and randomly shufﬂe taken points between agents.

Original: By varying hyperparameter Alpha we control the level of data heterogeneity between agents.
Revised: By varying hyperparameter Alpha we control the level of data heterogeneity between agents.

Original: Notice that if
Revised: Notice that if

Original: Alpha = 

Revised: Alpha =

Original: then datasets are completely different between agents, if Alpha = 1 then we have the ’same data’ setting with i.i.d.
Revised: then datasets are completely different between agents, if Alpha = 1 then we have the ’same data’ setting with i.i.d.

Original: split of data between two agents.
Revised: split of data between two agents.

Original: All other details are presented in the
Revised: All other details are presented in the

Original: Appendix A. We next summarize and discuss our results.
Revised: Appendix A. We next summarize and discuss our results.

Original: 6. R ESULTS AND D ISCUSSION
Revised: 6. R ESULTS AND D ISCUSSION

Original: AvgKD > AKD.
Revised: AvgKD > AKD.

Original: In all settings (both synthetic in Fig.
Revised: In all settings (both synthetic in Fig.

Original: 3 and real world in Fig. 4), we see that with the increasing number of rounds the performance of AKD signiﬁcantly degrades whereas that of
Revised: 3 and real world in Fig. 4), we see that with the increasing number of rounds the performance of AKD signiﬁcantly degrades whereas that of

Original: AvgKD stabilizes (Fig. 5) regardless of regularization, model and data heterogeneity.
Revised: AvgKD stabilizes (Fig. 5) regardless of regularization, model and data heterogeneity.

Original: Moreover, from the experiments on MNIST in Fig. 4, we see the following: AKD degrades faster if there is regularization used, model heterogeneity or data heterogeneity between agents, and the last plays the most signiﬁcant role.
Revised: Moreover, from the experiments on MNIST in Fig. 4, we see the following: AKD degrades faster if there is regularization used, model heterogeneity or data heterogeneity between agents, and the last plays the most signiﬁcant role.

Original: AvgKD, on the other hand, quickly converges in a few rounds and does not degrade.
Revised: AvgKD, on the other hand, quickly converges in a few rounds and does not degrade.

Original: However, it fails to match the centralized accuracy as well.
Revised: However, it fails to match the centralized accuracy as well.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: on MNIST with varying data heterogeneity in the setting of ’same model’.

Revised: In case of high data heterogeneity (small Alphas ), both agents beneﬁt from the AvgKD, PKD and EKD schemes.

Revised: Moreover, AvgKD and EKD consistently outperform FedAvg scheme.

Original: EKD works but needs large ensembles.
Revised: EKD works but needs large ensembles.

Original: In the synthetic setting (Fig. 3), in 250 rounds (ensemble of 500 models) it even matches the centralized model.
Revised: In the synthetic setting (Fig. 3), in 250 rounds (ensemble of 500 models) it even matches the centralized model.

Original: However, in the real world setting (Fig. 10) the improvement is slower and it does not match centralized performance.
Revised: However, in the real world setting (Fig. 6) the improvement is slower and it does not match centralized performance.

Original: This might be due to the small number of rounds run (only 10).
Revised: This might be due to the small number of rounds run (only 20).

Original: EKD is also the only method that improves with subsequent rounds.
Revised: EKD is also the only method that improves with subsequent rounds.

Original: Finally, we observed that increasing regularization actually speeds up the convergence of EKD, while it worsened in the cases of AKD and AvgKD.
Revised: Finally, we observed that increasing regularization actually speeds up the convergence of EKD in the synthetic setting (Fig. 3).

Original: Data heterogeneity is the main bottleneck.
Revised: Data heterogeneity is the main bottleneck.

Original: In all our experiments, both data and model heterogeneity degraded the performance of AKD and AvgKD.
Revised: In all our experiments, both data and model heterogeneity degraded the performance of AKD, PKD (Parallel KD introduced in App. E) and AvgKD.

Original: However, data heterogeneity has a much stronger effect.
Revised: However, data heterogeneity has a much stronger effect.

Original: This conﬁrms our theory that mismatch between agents data leads to loss of information when using knowledge distillation.
Revised: This conﬁrms our theory that mismatch between agents data leads to loss of information when using knowledge distillation.

Original: Overcoming this data heterogeneity is the crucial challenge for practical model agnostic FL.
Revised: Overcoming this data heterogeneity is the crucial challenge for practical model agnostic FL.

Original: Fig.
Revised: Fig.

Original: 9 and 6 show how AKD and AvgKD schemes behave in dependence on data heterogeneity.
Revised: 6 shows how all schemes behave in dependence on data heterogeneity.

Original: Indeed, the higher data heterogeneity the faster speed of degradation for the AKD scheme.
Revised: Indeed, the higher data heterogeneity the faster speed of degradation for the AKD and PKD schemes.

Original: In the case of the AvgKD scheme, we see that agents do improve over their local models and this improvement is larger with more data heterogeneity.However
Revised: In the case of the AvgKD scheme, we see that agents do improve over their local models and this improvement is larger with more data heterogeneity.

Original: 6. A DDITIONAL EXTENSIONS AND EXPERIMENTS
Revised: 6. A DDITIONAL EXTENSIONS AND EXPERIMENTS

Original: In App.
Revised: In App.

Original: H, we extend our algorithms to M agents and show experimentally in App.
Revised: G, we extend our algorithms to M agents and show experimentally in App.

Original: I.6 that the same trends hold there as well.
Revised: H.5 for AvgKD algorithm that the same trends hold there as well.

Original: Our conclusions also hold for the cross-entropy loss (Figs. 11, 12), for highly heterogeneous model case with MLP and Random Forests (Figs. 15, 16), as well on other datasets and models (VGG on CIFAR10 in Figs.
Revised: Our conclusions also hold for the cross-entropy loss (Figs. 11, 12), for highly heterogeneous model case with MLP and Random Forests (Figs. 13, 14), as well on other datasets and models (VGG on CIFAR10 in Figs.

Original: In the latter, we see higher speeds of degradation which is probably due to CIFAR10 being much more complex (meaning potentially larger data heterogeneity), and also more complex models such as VGG (larger model heterogeneity).
Revised: In the latter, we see the same trends as on MNIST for all the schemes except EKD which is probably due to the use of cross-entropy loss function and, as a result, models being further from the kernel regime. Moreover, speed of degradation is higher if there is the model heterogeneity (Figs. 8, 10) and EKD does not help even on MNIST dataset (Fig.

Original: Finally, in Fig. 18, we compare AvgKD to the more standard FedAvg  and yet another variant (parallel
Revised: Finally, all the schemes are compared to the more standard FedAvg that is not applicable in the ’different model’ setting and is outperformed by the AvgKD scheme at highly data heterogeneous regimes.

Original: KD or PKD). AvgKD consistently outperforms these methods  indicating it is  most promising variant.
Revised: That is, AvgKD consistently outperforms all the methods at highly data heterogeneous regimes indicating it is the most promising variant.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: While the stochastic optimization framework has been very useful in analyzing and developing new algorithms for federated learning so far, it fundamentally cannot capture learning with different models.
Revised: While the stochastic optimization framework has been very useful in analyzing and developing new algorithms for federated learning so far, it fundamentally cannot capture learning with different models.

Original: We instead introduced the federated kernel regression framework where we formalized notions of both model and data heterogeneity.
Revised: We instead introduced the federated kernel regression framework where we formalized notions of both model and data heterogeneity.

Original: Using this, we analyzed different knowledge distillation schemes and came to the conclusion that data heterogeneity poses a fundamental challenge limiting the knowledge that can be transmitted.
Revised: Using this, we analyzed different knowledge distillation schemes and came to the conclusion that data heterogeneity poses a fundamental challenge limiting the knowledge that can be transmitted.

Original: Further, these theoretical predictions were exactly reﬂected in our deep learning experiments as well.
Revised: Further, these theoretical predictions were exactly reﬂected in our deep learning experiments as well.

Original: Overcoming this data heterogeneity will be crucial to making
Revised: Overcoming this data heterogeneity will be crucial to making

Original: KD based model agnostic federated algorithms practical.
Revised: KD based model agnostic federated algorithms practical.

Original: We also utilized our framework to design a novel ensembling method motivated by our theory.
Revised: We also utilized our framework to design a novel ensembling method motivated by our theory.

Original: However, this method could require very large ensembles (up to 500 models) in order to match the centralized performance.
Revised: However, this method could require very large ensembles (up to 500 models) in order to match the centralized performance.

Original: Thus, we view our method not as a practical algorithm, but more of a demo on how our framework can be leveraged.
Revised: Thus, we view our method not as a practical algorithm, but more of a demo on how our framework can be leveraged.

Original: Similarly, our experiments are preliminary and are not on real world  datasets.
Revised: Similarly, our experiments are preliminary and are not on real world complex datasets.

Original: We believe there is great potential in further exploring our results and framework, especially in investigating how to mitigate the effect of data heterogeneity in knowledge distillation
Revised: We believe there is great potential in further exploring our results and framework, especially in investigating how to mitigate the effect of data heterogeneity in knowledge distillation.

Revised: A CKNOWLEDGEMENTS

Revised: We are really grateful to Martin Jaggi for his insightful comments and support throughout this work.

Revised: SPK is partly funded by an SNSF Fellowship and AA is funded by a research scholarship from MLO lab, EPFL headed by Martin Jaggi.

Revised: SPK also thanks Celestine Dünner for conversations inspiring this project.

Original: A E XPERIMENTS DETAILS
Revised: A E XPERIMENTS DETAILS

Original: In all real world experiments we use the Adam optimizer with a default regularization (weight decay) of 3 × 10 − 4 , unless in the ‘no regularization’ case when it is set to 0.
Revised: In all real world experiments we use the Adam optimizer with a default regularization (weight decay) of 3 × 10 − 4 , unless in the ‘no regularization’ case when it is set to 0.

Original: We split the data betweenagents by giving a bigger part of data to agent 1 at all ’same data’ experiments.
Revised: We split the data betweenagents by giving a bigger part of data to agent 1 at all ’same data’ experiments.

Original: That is, for ’different data’ experiments and heterogeneous data experiments where we vary Alpha hyperparameter we split data equally between 2 agents.
Revised: That is, for ’different data’ experiments and heterogeneous data experiments where we vary Alpha hyperparameter we split data equally between 2 agents.

Original: The non-equal split can help us to see another effect in the experiments: if agent 2 (with less data) beneﬁts from the communication with agent 1 (with more data).
Revised: The non-equal split can help us to see another effect in the experiments: if agent 2 (with less data) beneﬁts from the communication with agent 1 (with more data).

Original: In heterogeneous data experiments, we explore the signiﬁcance of data heterogeneity effect on the behavior of KD scheme.
Revised: In heterogeneous data experiments, we explore the signiﬁcance of data heterogeneity effect on the behavior of KD scheme.

Original: That is why we design an almost ideal setting at all such experiments: the ’same model’ setting (except experiments with RF and MLP), the equal split of data in terms of its amount between agents.
Revised: That is why we design an almost ideal setting at all such experiments: the ’same model’ setting (except experiments with RF and MLP), the equal split of data in terms of its amount between agents.

Original: Toy experiments.
Revised: Toy experiments.

Original: The toy experiments solve a linear regression problem of the form Ax (cid:63)
Revised: The toy experiments solve a linear regression problem of the form Ax (cid:63)

Original: =
Revised: =

Original: b where A ∈ R n × d and x (cid:63) ∈ R
Revised: b where A ∈ R n × d and x (cid:63) ∈ R

Original: d is randomly generated for n = 1 .
Revised: d is randomly generated for n = 1 .

Original: 5 d and d = 100 .
Revised: 5 d and d = 100 .

Original: Then, the data A and b is split between the two agents at proportion 0 .
Revised: Then, the data A and b is split between the two agents at proportion 0 .

Original: This is done randomly in the ‘same data’ case, whereas in the ‘different data’ case the data is sorted according to b before splitting to maximize heterogeneity.
Revised: This is done randomly in the ‘same data’ case, whereas in the ‘different data’ case the data is sorted according to b before splitting to maximize heterogeneity.

Original: This experiment with the linear kernel is supposed to show if our theory is correct, especially if the EKD scheme really works.
Revised: This experiment with the linear kernel is supposed to show if our theory is correct, especially if the EKD scheme really works.

Original: Real world experiments.
Revised: Real world experiments.

Original: The real world experiments are conducted using CNN and MLP networks on MNIST, MLP network and RF model on MNIST, and VGG16 1 (Simonyan & Zisserman, 2015) and CNN models on CIFAR10 datasets.
Revised: The real world experiments are conducted using CNN and MLP networks on MNIST, MLP network and RF model on MNIST, and VGG16 1 (Simonyan & Zisserman, 2015) and CNN models on CIFAR10 datasets.

Original: Further, we split the training data randomly with a 70% − 30% split in the ’same data’ setting.
Revised: Further, we split the training data randomly at proportion 0 . 3 in the ’same data’ setting.

Original: For the ’different data’ setting, we split the data by labels: agent 1 has ’0’ to ’4’ labeled data points, agent 2 has ’5’ to ’9’.
Revised: For the ’different data’ setting, we split the data by labels: agent 1 has ’0’ to ’4’ labeled data points, agent 2 has ’5’ to ’9’.

Original: Then we take randomly from each agent some Alpha = 0 .
Revised: Then we take randomly from each agent some Alpha = 0 .

Original: 1 portion of data, combine it and randomly return data points to both agents from this combined dataset.
Revised: 1 portion of data, combine it and randomly return data points to both agents from this combined dataset.

Original: B A LTERNATING KD WITH REGULARIZATION
Revised: B A LTERNATING KD WITH REGULARIZATION

Original: Different models.
Revised: Different models.

Original: Keeping the notation of section 4 we can construct the following matrix:
Revised: Keeping the notation of section 4 we can construct the following matrix:

Original: [Equation]
Revised: [Equation]

Original: Notice that each of the sub-blocks L and M are symmetric positive semi-deﬁnite matrices.
Revised: Notice that each of the sub-blocks L and M are symmetric positive semi-deﬁnite matrices.

Original: This makes matrix K symmetric positive semi-deﬁnite matrix
Revised: This makes matrix K symmetric positive semi-deﬁnite matrix

Original: and it has eigendecomposition form:
Revised: and it has eigendecomposition form:

Original: [Equation]
Revised: [Equation]

Original:  where D and V are R 2( N 1 + N 2 ) × 2( N 1 + N 2 ) diagonal and orthogonal matrices correspondingly.
Revised: and [Equation] where D and V are R 2( N 1 + N 2 ) × 2( N 1 + N 2 ) diagonal and orthogonal matrices correspondingly.

Original: This means that the R 2( N 1 + N 2 ) × 2( N 1 + N 2 ) matrices V 1 , V 2 , ˜ V 1 , ˜ V 2 are also orthogonal to each other.
Revised: This means that the R 2( N 1 + N 2 ) × 2( N 1 + N 2 ) matrices V 1 , V 2 , ˜ V 1 , ˜ V 2 are also orthogonal to each other.

Original: Then one can deduce:
Revised: Then one can deduce:

Original: [Equation]
Revised: [Equation]

Original: In AKD setting only agent 1 has labeled data.
Revised: In AKD setting only agent 1 has labeled data.

Original: The solution of learning from the dataset D 1 evaluated at set X 2 is the following:
Revised: The solution of learning from the dataset D 1 evaluated at set X 2 is the following:

Original: [Equation]
Revised: [Equation]

Original: Let us introduce notation:
Revised: Let us introduce notation:

Original: [Equation]
Revised: [Equation]

Original: These are well known oblique (weighted) projection matrices on the subspaces spanned by the columns of matrices V 1 and ˜ V 2 correspondingly, where the scalar product is deﬁned with Gram matrix G = c
Revised: These are well known oblique (weighted) projection matrices on the subspaces spanned by the columns of matrices V 1 and ˜ V 2 correspondingly, where the scalar product is deﬁned with Gram matrix G = c

Original: I + D .
Revised: I + D .

Original: Similarly one can deﬁne P 2 and ˜ P 1 .
Revised: Similarly one can deﬁne P 2 and ˜ P 1 .

Original: Given the introduced notation and using the fact that V (cid:62) 2 V 1 = 0 , we can rewrite equation 10 as:
Revised: Given the introduced notation and using the fact that V (cid:62) 2 V 1 = 0 , we can rewrite equation 10 as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, given X 1 ⊆ D 1 and agent 1 learned from ˆ Y 2 =
Revised: Similarly, given X 1 ⊆ D 1 and agent 1 learned from ˆ Y 2 =

Original: g 10 ( X 2 ) , inferred prediction ˆ Y 1 =
Revised: g 10 ( X 2 ) , inferred prediction ˆ Y 1 =

Original: g 11 ( X 1 ) by agent 2 can be written as:
Revised: g 11 ( X 1 ) by agent 2 can be written as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: At this point we need to introduce additional notation:
Revised: At this point we need to introduce additional notation:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: These are matrices of contraction linear mappings.
Revised: These are matrices of contraction linear mappings.

Original: One can now repeat the whole process again with replacement Y 1 by ˆ Y 1 .
Revised: One can now repeat the whole process again with replacement Y 1 by ˆ Y 1 .

Original: The predictions by agentand agent 2 after such 2 t rounds of AKD are:
Revised: The predictions by agentand agent 2 after such 2 t rounds of AKD are:

Original: [Equation]
Revised: [Equation]

Original: If one considers the case where agents have the same kernel u 1 = u 2 = u then one should ’remove all tildas’ in the above expressions and the obtained expressions are applied.
Revised: If one considers the case where agents have the same kernel u 1 = u 2 = u then one should ’remove all tildas’ in the above expressions and the obtained expressions are applied.

Original: This means M = L ,
Revised: This means M = L ,

Original: V 1 =
Revised: V 1 =

Original: ˜ V 1 and V 2 =
Revised: ˜ V 1 and V 2 =

Original: ˜ V 2 .
Revised: ˜ V 2 .

Original: Crucial changes in this case are C 1 = ˜ C 

Revised: Crucial changes in this case are C 1 = ˜ C

Original: → I and (cid:16) C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 (cid:17) t → (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1) t .
Revised: → I and (cid:16) C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 (cid:17) t → (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1) t .

Original: The matrix (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1) t is an alternating projection (Boyd & Dattorro, 2003) operator after t steps.
Revised: The matrix (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1) t is an alternating projection (Boyd & Dattorro, 2003) operator after t steps.

Original: Given 2 closed convex sets, alternating projection algorithm in the limit ﬁnds a point in the intersection of these sets, provided they intersect (Boyd & Dattorro, 2003; Lewis et al., 2007).
Revised: Given 2 closed convex sets, alternating projection algorithm in the limit ﬁnds a point in the intersection of these sets, provided they intersect (Boyd & Dattorro, 2003; Lewis et al., 2007).

Original: In our case matrices operators P 1 and P 2 project onto linear spaces spanned by columns of matrices
Revised: In our case matrices operators P 1 and P 2 project onto linear spaces spanned by columns of matrices

Original: V 1 and V 2 correspondingly.
Revised: V 1 and V 2 correspondingly.

Original: These are orthogonal linear subspaces, hence the unique point of their intersection is the origin point 0 .
Revised: These are orthogonal linear subspaces, hence the unique point of their intersection is the origin point 0 .

Original: This means that (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1)
Revised: This means that (cid:0) P (cid:62) 2 P (cid:62) 1 (cid:1)

Original: t −−−→ t →∞ 0 and in the limit of such AKD procedure both agents predict 0 for any data point.
Revised: t −−−→ t →∞ 0 and in the limit of such AKD procedure both agents predict 0 for any data point.

Original: Speed of degradation.
Revised: Speed of degradation.

Original: The speed of convergence for the alternating projections algorithm is known and deﬁned by the minimal angle φ between corresponding sets (Aronszajn, 1950) (only non-zero elements from sets one has to consider):
Revised: The speed of convergence for the alternating projections algorithm is known and deﬁned by the minimal angle φ between corresponding sets (Aronszajn, 1950) (only non-zero elements from sets one has to consider):

Original: [Equation]
Revised: [Equation]

Original: where v is one of the columns of matrix V 1 .
Revised: where v is one of the columns of matrix V 1 .

Original: In our case we can write the expression for the cosine: cos
Revised: In our case we can write the expression for the cosine: cos

Original: [Equation]
Revised: [Equation]

Original: where v 1 and v 2 are the vectors from subspaces spanned by columns of matrices V 1 and V 2 correspondingly.
Revised: where v 1 and v 2 are the vectors from subspaces spanned by columns of matrices V 1 and V 2 correspondingly.

Original: Hence the speed of convergence depends on the elements of the matrix L 12 .
Revised: Hence the speed of convergence depends on the elements of the matrix L 12 .

Original: Intuitively these elements play the role of the measure of ’closeness’ between data points.
Revised: Intuitively these elements play the role of the measure of ’closeness’ between data points.

Original: This means that the ’closer’ points of sets X 1 and X 2 the higher absolute values of elements in matrix L 12 .
Revised: This means that the ’closer’ points of sets X 1 and X 2 the higher absolute values of elements in matrix L 12 .

Original: Moreover, we see inversely proportional dependence on the regularization constant c .
Revised: Moreover, we see inversely proportional dependence on the regularization constant c .

Original: All these sums up in the following proposition which is the formal version of the proposition IV:
Revised: All these sums up in the following proposition which is the formal version of the proposition IV:

Original: Proposition VIII (Formal) .
Revised: Proposition VIII (Formal) .

Original: The rate of convergence of g 1 t ( x ) to 0 gets faster if:
Revised: The rate of convergence of g 1 t ( x ) to 0 gets faster if:

Original: • larger regularization constant c is used during the training, • smaller the eigenvalues of the matrix V 1 ˜ V (cid:62) 1 , or • smaller absolute value of non-diagonal block
Revised: • larger regularization constant c is used during the training, • smaller the eigenvalues of the matrix V 1 ˜ V (cid:62) 1 , or • smaller absolute value of non-diagonal block

Original: L
Revised: L

Original: C A LTERNATING KD WITHOUT REGULARIZATION
Revised: C A LTERNATING KD WITHOUT REGULARIZATION

Original: Before we saw that models of both agents degrade if one uses regularization.
Revised: Before we saw that models of both agents degrade if one uses regularization.

Original: A natural question to ask if models will degrade in the lack of regularization.
Revised: A natural question to ask if models will degrade in the lack of regularization.

Original: Consider the problem (1) with c → 0 , so the regularization term cancels out.
Revised: Consider the problem (1) with c → 0 , so the regularization term cancels out.

Original: In the general case, there are many possible solutions as kernel matrix K may have 0 eigenvalues.
Revised: In the general case, there are many possible solutions as kernel matrix K may have 0 eigenvalues.

Original: Motivated by the fact that Stochastic Gradient Descent (SGD) tends to ﬁnd the solution with minimal norm (Wilson et al., 2017), we propose to analyze the minimal norm solution in the problem of alternating knowledge distillation with 2 agents each with private dataset.
Revised: Motivated by the fact that Stochastic Gradient Descent (SGD) tends to ﬁnd the solution with minimal norm (Wilson et al., 2017), we propose to analyze the minimal norm solution in the problem of alternating knowledge distillation with 2 agents each with private dataset.

Original: For simplicity let us assume that agents have the same model architecture.
Revised: For simplicity let us assume that agents have the same model architecture.

Original: Then the agent I solution evaluated at set X 2 with all the above notation can be written as:
Revised: Then the agent I solution evaluated at set X 2 with all the above notation can be written as:

Original: [Equation]
Revised: [Equation]

Original: ) where † stands for pseudoinverse.
Revised: ) where † stands for pseudoinverse.

Original: In this section let us consider 3 possible settings one can have:
Revised: In this section let us consider 3 possible settings one can have:

Original: Self-distillation: The datasets and models of both agents are the same.
Revised: Self-distillation: The datasets and models of both agents are the same.

Original: Distillation with K > 0 :
Revised: Distillation with K > 0 :

Original: The datasets of both agents are different and private.
Revised: The datasets of both agents are different and private.

Original: Kernel matrix K is positive deﬁnite.
Revised: Kernel matrix K is positive deﬁnite.

Original: Distillation with K ≥ 0 : The datasets of both agents are different and private.
Revised: Distillation with K ≥ 0 : The datasets of both agents are different and private.

Original: Kernel matrix K is positive semi-deﬁnite.
Revised: Kernel matrix K is positive semi-deﬁnite.

Original: Self-distillation.
Revised: Self-distillation.

Original: Given the dataset D = X
Revised: Given the dataset D = X

Original: × Y , the solution of the supervised learning with evaluation at any x ∈ R d is:
Revised: × Y , the solution of the supervised learning with evaluation at any x ∈ R d is:

Original: [Equation]
Revised: [Equation]

Original: Then the expression for the self-distillation step with evaluation at any x ∈ R d is:
Revised: Then the expression for the self-distillation step with evaluation at any x ∈ R d is:

Original: [Equation]
Revised: [Equation]

Original: where property of pseudoinverse matrix was used: A † AA † =
Revised: where property of pseudoinverse matrix was used: A † AA † =

Original: A † .
Revised: A † .

Original: That is, self-distillation round does not change obtained model.
Revised: That is, self-distillation round does not change obtained model.

Original: One can repeat self-distillation step t times and there is no change in the model:
Revised: One can repeat self-distillation step t times and there is no change in the model:

Original: [Equation]
Revised: [Equation]

Original: Hence in the no regularization setting self-distillation step does not give any change in the obtained model.
Revised: Hence in the no regularization setting self-distillation step does not give any change in the obtained model.

Original: Distillation with K > 0 .
Revised: Distillation with K > 0 .

Original: In this setting we have the following identity K † =
Revised: In this setting we have the following identity K † =

Original: K − 1 and results are quite similar to the setting with regularization.
Revised: K − 1 and results are quite similar to the setting with regularization.

Original: But now the Gram matrix of scalar product in linear space is D instead of c
Revised: But now the Gram matrix of scalar product in linear space is D instead of c

Original: I + D in the regularized setting.
Revised: I + D in the regularized setting.

Original: By assumption on matrix K it follows that
Revised: By assumption on matrix K it follows that

Original: D is of full rank and the alternating projection algorithm converges to the origin point 0 .
Revised: D is of full rank and the alternating projection algorithm converges to the origin point 0 .

Original: Therefore in the limit of AKD steps predictions by models of two agents will degrade towards 0 .
Revised: Therefore in the limit of AKD steps predictions by models of two agents will degrade towards 0 .

Original: Distillation with K ≥ 0 .
Revised: Distillation with K ≥ 0 .

Original: In this setting kernel matrix K has at least one 0 eigenvalue and we take for the analysis the minimal norm solution (13).
Revised: In this setting kernel matrix K has at least one 0 eigenvalue and we take for the analysis the minimal norm solution (13).

Original: We can rewrite this solution:
Revised: We can rewrite this solution:

Original: [Equation]
Revised: [Equation]

Original: where ˆ P 1 = V 1 ( V (cid:62) 1 DV 1 ) † V (cid:62) 1 D and z 1 = V 1 y 1 .
Revised: where ˆ P 1 = V 1 ( V (cid:62) 1 DV 1 ) † V (cid:62) 1 D and z 1 = V 1 y 1 .

Original: One can notice the following projection properties of matrix ˆ P 1 : ˆ
Revised: One can notice the following projection properties of matrix ˆ P 1 : ˆ

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: That is, ˆ P 1 is a projection matrix with eigenspace spanned by columns of the matrix V 1 ( V (cid:62) 1 DV 1 ) † .
Revised: That is, ˆ P 1 is a projection matrix with eigenspace spanned by columns of the matrix V 1 ( V (cid:62) 1 DV 1 ) † .

Original: Similarly, one can deﬁne ˆ P 2 .
Revised: Similarly, one can deﬁne ˆ P 2 .

Original: Then the solution for the ﬁrst AKD step evaluated at set X 1 is:
Revised: Then the solution for the ﬁrst AKD step evaluated at set X 1 is:

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: 1 z 1 , and after t rounds we obtain for agent I and agent II correspondingly:
Revised: 1 z 1 , and after t rounds we obtain for agent I and agent II correspondingly:

Original: [Equation]
Revised: [Equation]

Original: and g (cid:63) 2 t
Revised: and g (cid:63) 2 t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: ( ˆ P (cid:62)
Revised: ( ˆ P (cid:62)

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: In the limit of the distillation rounds operator ( ˆ P 2 ˆ P 1 ) t tends to the projection on the intersection of 2 subspaces spanned by the columns of matrices V 1 ( V (cid:62) 1 DV 1 ) † and V 2 ( V (cid:62) 2 DV 2 ) † .
Revised: In the limit of the distillation rounds operator ( ˆ P 2 ˆ P 1 ) t tends to the projection on the intersection of 2 subspaces spanned by the columns of matrices V 1 ( V (cid:62) 1 DV 1 ) † and V 2 ( V (cid:62) 2 DV 2 ) † .

Original: We
Revised: We

Original: should
Revised: should

Original: [Figure]
Revised: [Figure]

Original: Train Train
Revised: Train Train

Original: [Figure]
Revised: [Figure]

Original: highlight that in general, the intersection set in this case consists not only from the origin point 

Revised: highlight that in general, the intersection set in this case consists not only from the origin point

Original: but some rays that lie simultaneously in the eigenspaces of both projectors ˆ P 1 and ˆ P 2 .
Revised: but some rays that lie simultaneously in the eigenspaces of both projectors ˆ P 1 and ˆ P 2 .

Original: The illustrative example is shown in the Fig.
Revised: The illustrative example is shown in the Fig.

Original: 7a.
Revised: 7a.

Original: We perform the alternating projection algorithm between orthogonal linear spaces that have a ray in the intersection and we converge to some non-zero point that belongs to this ray.
Revised: We perform the alternating projection algorithm between orthogonal linear spaces that have a ray in the intersection and we converge to some non-zero point that belongs to this ray.

Original: The intersection of eigenspaces of both projectors ˆ P 1 and ˆ P 2 is the set of points
Revised: The intersection of eigenspaces of both projectors ˆ P 1 and ˆ P 2 is the set of points

Original: [Equation]
Revised: [Equation]

Original: D A VERAGED KD
Revised: D A VERAGED KD

Original: In this section we present the detailed analysis of the algorithm presented in the section 4.2 keeping the notation of the section B. As a reminder, models of both agents after round t of AvgKD algorithm are as follows:
Revised: In this section we present the detailed analysis of the algorithm presented in the section 4.2 keeping the notation of the section B. As a reminder, models of both agents after round t of AvgKD algorithm are as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t = 1 :
Revised: t = 1 :

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The illustration of this process is presented in the Fig.
Revised: The illustration of this process is presented in the Fig.

Original: 2a.
Revised: 2a.

Original: Consider the sequence of solutions for the agent 1 with evaluation at X 2 :
Revised: Consider the sequence of solutions for the agent 1 with evaluation at X 2 :

Original: • Supervised Learning:
Revised: • Supervised Learning:

Original: [Equation]
Revised: [Equation]

Original: • 1st round of KD:
Revised: • 1st round of KD:

Original: [Equation]
Revised: [Equation]

Original: • 2nd round of KD:
Revised: • 2nd round of KD:

Original: [Equation]
Revised: [Equation]

Original: • 3rd round of KD:
Revised: • 3rd round of KD:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: • t -th round of KD:
Revised: • t -th round of KD:

Original: [Equation]
Revised: [Equation]

Original: • The limit of KD steps:
Revised: • The limit of KD steps:

Original: [Equation]
Revised: [Equation]

Original: where † stands for pseudoinverse.
Revised: where † stands for pseudoinverse.

Original: Let us analyze the limit solution and consider the ﬁrst term of its expression.
Revised: Let us analyze the limit solution and consider the ﬁrst term of its expression.

Original: One can deduce the following identity:
Revised: One can deduce the following identity:

Original: [Equation]
Revised: [Equation]

Original: In a similar manner
Revised: In a similar manner

Original: [Equation]
Revised: [Equation]

Original: we can deal with the second term:
Revised: we can deal with the second term:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we notice Schur complement expression in the equation (18):
Revised: we notice Schur complement expression in the equation (18):

Original: [Equation]
Revised: [Equation]

Original: This means that we can consider the following problem:
Revised: This means that we can consider the following problem:

Original: [Equation]
Revised: [Equation]

Original: and derive that g 1 ∞ ( X 2 )
Revised: and derive that g 1 ∞ ( X 2 )

Original: =
Revised: =

Original: V (cid:62) 2 DV 1β 1 and g
Revised: V (cid:62) 2 DV 1β 1 and g

Original: [Equation]
Revised: [Equation]

Original: ( X
Revised: ( X

Original: [Equation]
Revised: [Equation]

Original: β 2 , where β 1 , β 2 are deﬁned as the solution to the problem (19).
Revised: β 2 , where β 1 , β 2 are deﬁned as the solution to the problem (19).

Original: Given this
Revised: Given this

Original: [Equation]
Revised: [Equation]

Original: we can derive the following:
Revised: we can derive the following:

Original: [Equation]
Revised: [Equation]

Original: As one can see, there is a strong relation between the limit KD solutions and the solution of a linear system of equations with modiﬁed matrix K and right-hand side.
Revised: As one can see, there is a strong relation between the limit KD solutions and the solution of a linear system of equations with modiﬁed matrix K and right-hand side.

Original: Mainly, we take the kernel matrix and divide its non-diagonal blocks by 2, which intuitively shows that our ﬁnal model accounts for the reduction of the ’closeness’ between datasets D 1 and D 2 .
Revised: Mainly, we take the kernel matrix and divide its non-diagonal blocks by 2, which intuitively shows that our ﬁnal model accounts for the reduction of the ’closeness’ between datasets D 1 and D 2 .

Original: And on the right-hand side, we see − y 2 instead of y 2 which is quite a ’artiﬁcial’ effect.
Revised: And on the right-hand side, we see − y 2 instead of y 2 which is quite a ’artiﬁcial’ effect.

Original: Overall these results in the fact that both limit solutions (for each agent) do not give a ground truth prediction for X 1 and X 2 individually, which one can see from equation (20) with c = 0 .
Revised: Overall these results in the fact that both limit solutions (for each agent) do not give a ground truth prediction for X 1 and X 2 individually, which one can see from equation (20) with c = 0 .

Original: That is, we need combine the predictions of both agents in a speciﬁc way to get ground truth labels for datasets D 1 and D 2 .
Revised: That is, we need combine the predictions of both agents in a speciﬁc way to get ground truth labels for datasets D 1 and D 2 .

Original: Moreover, the way we combine the solutions differs between datasets D 1 and D 2 that one can see from comparison of right-hand sides of expressions (20) and (21).
Revised: Moreover, the way we combine the solutions differs between datasets D 1 and D 2 that one can see from comparison of right-hand sides of expressions (20) and (21).

Original: Given all the above, to predict optimal labels we need to change we way we combine models of agents in dependence on a dataset, but an usual desire is to have one model that predicts ground truth labels for at least both training datasets.
Revised: Given all the above, to predict optimal labels we need to change we way we combine models of agents in dependence on a dataset, but an usual desire is to have one model that predicts ground truth labels for at least both training datasets.

Original: To obtain the expressions for the case of identical models one should ’remove all tildas’ in the above expressions and by setting V 1 =
Revised: To obtain the expressions for the case of identical models one should ’remove all tildas’ in the above expressions and by setting V 1 =

Original: ˜ V 1 , V 2 =
Revised: ˜ V 1 , V 2 =

Original: ˜ V 2 , C 1 = ˜ C 

Revised: ˜ V 2 , C 1 = ˜ C

Original: → I and ( C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 ) t → ( P (cid:62) 2 P (cid:62) 1 ) t 2 In case of c = 0 , the whole analysis can be repeated by replacing inverse sign with † sign and using the following fact for positivesemideﬁnite matrices (Zhang,
Revised: → I and ( C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 ) t → ( P (cid:62) 2 P (cid:62) 1 ) t 2 In case of c = 0 , the whole analysis can be repeated by replacing inverse sign with † sign and using the following fact for positive

Original: [Equation]
Revised: [Equation]

Original: E P ARALLEL KD
Revised: E P ARALLEL KD

Original: In this section, we theoretically analyze a slight modiﬁcation of the AvgKD algorithm which we call Parallel KD (PKD).
Revised: In this section, we theoretically analyze a slight modiﬁcation of the AvgKD algorithm which we call Parallel KD (PKD).

Original: Keeping the notation of sections B and C, denote the data on agentas D 1 =
Revised: Keeping the notation of sections B and C, denote the data on agentas D 1 =

Original: ( X 1 , y 1 ) where X 

Revised: ( X 1 , y 1 ) where X

Original: [ i, :] = x 1 n and y 1 [ i ] =
Revised: [ i, :] = x 1 n and y 1 [ i ] =

Original: y 1 i .
Revised: y 1 i .

Original: Correspondingly, for agent 2 we have
Revised: Correspondingly, for agent 2 we have

Original: D 2 =
Revised: D 2 =

Original: ( X 2 , y 2 ) .
Revised: ( X 2 , y 2 ) .

Original: Now starting from ˆ y 10 =
Revised: Now starting from ˆ y 10 =

Original: y 1 , ˆ y 20 =
Revised: y 1 , ˆ y 20 =

Original: y 2 , in each round t ≥ 0 : a
Revised: y 2 , in each round t ≥ 0 : a

Original: Agents
Revised: Agents

Original: and 2 train their
Revised: and 2 train their

Original: model on datasets ( X
Revised: model on datasets ( X

Original: , ˆ y 1 t )
Revised: , ˆ y 1 t )

Original: and ( X
Revised: and ( X

Original: ˆ y
Revised: ˆ y

Original: t
Revised: t

Original: ) to obtain g 1 t
Revised: ) to obtain g 1 t

Original: and g
Revised: and g

Original: 2 t .
Revised: 2 t .

Original: b. Agents exchange g 1 t and g 2 t between each other.
Revised: b. Agents exchange g 1 t and g 2 t between each other.

Original: c. Agents use exchanged models to predict labels
Revised: c. Agents use exchanged models to predict labels

Original: [Equation]
Revised: [Equation]

Original: The summary of the algorithm is depicted in Figure 7b.
Revised: The summary of the algorithm is depicted in Figure 7b.

Original: That is, we learn from the average ofagents’ predictions.
Revised: That is, we learn from the average ofagents’ predictions.

Original: For simplicity, we are going to analyze the scheme without regularization and we take always a minimum norm solution.
Revised: For simplicity, we are going to analyze the scheme without regularization and we take always a minimum norm solution.

Original: Notice that there is no exchange of raw data but only of the trained models.
Revised: Notice that there is no exchange of raw data but only of the trained models.

Original: Let us analyse KD algorithm where solutions for agent
Revised: Let us analyse KD algorithm where solutions for agent

Original: I ( g 1 ) and agent II ( g 2 ) obtained as:
Revised: I ( g 1 ) and agent II ( g 2 ) obtained as:

Original: [Equation]
Revised: [Equation]

Original: where g
Revised: where g

Original: [Equation]
Revised: [Equation]

Original: : g
Revised: : g

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: zand g 20 =
Revised: zand g 20 =

Original: ˆ P (cid:62) 2 z 2 , where z i
Revised: ˆ P (cid:62) 2 z 2 , where z i

Original: = V i
Revised: = V i

Original: y i , i = 1 , 2 .
Revised: y i , i = 1 , 2 .

Original: Consider g 1 t for t ≥ 1 :
Revised: Consider g 1 t for t ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: The form of the solutions reminds the method of averaged projection (Lewis et al., 2007) with operator ˆ P 1 + ˆ P 2 , which is similar to alternating projection converges to the intersection point ofsubspaces 3 .
Revised: The form of the solutions reminds the method of averaged projection (Lewis et al., 2007) with operator ˆ P 1 + ˆ P 2 , which is similar to alternating projection converges to the intersection point ofsubspaces 3 .

Original: That is, similarly to AKD in the case with the regularization we expect the solution to converge to the origin point 0 in the limit of the distillation steps.
Revised: That is, similarly to AKD in the case with the regularization we expect the solution to converge to the origin point 0 in the limit of the distillation steps.

Original: As a result, after some point, one expects steady degradation of the predictions of both agents in the PKD scheme.
Revised: As a result, after some point, one expects steady degradation of the predictions of both agents in the PKD scheme.

Original: F E NSEMBLED KD
Revised: F E NSEMBLED KD

Original: One can consider the following problem:
Revised: One can consider the following problem:

Original: [Equation]
Revised: [Equation]

Original: with the following identities:
Revised: with the following identities:

Original: [Equation]
Revised: [Equation]

Original: One can ﬁnd β 1 , β 2 and deduce the following prediction by the model associated with the system (27) for i = 1 , 2 :
Revised: One can ﬁnd β 1 , β 2 and deduce the following prediction by the model associated with the system (27) for i = 1 , 2 :

Original: [Equation]
Revised: [Equation]

Original: From this one can easily deduce (28) which means that for datasets of both agents this model predicts ground truth labels.
Revised: From this one can easily deduce (28) which means that for datasets of both agents this model predicts ground truth labels.

Original: To obtain the expressions for the case of identical models one should ’remove all tildas’ in all the above expressions and by setting V 1 =
Revised: To obtain the expressions for the case of identical models one should ’remove all tildas’ in all the above expressions and by setting V 1 =

Original: ˜ V 1 , V 2 =
Revised: ˜ V 1 , V 2 =

Original: ˜ V 2 , C 1 = ˜ C 

Revised: ˜ V 2 , C 1 = ˜ C

Original: → I and ( C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 ) [Equation] → ( P (cid:62) 
Revised: → I and ( C 1 ˜ P (cid:62) 2 ˜ C 2 P (cid:62) 1 ) t → ( P (cid:62) 2 P (cid:62) 1 )

Original: [Equation] P (cid:62) 1 ) t
Revised: t

Original: The last question is how one can construct the scheme of iterative KD rounds to obtain the above expression for the limit model.
Revised: The last question is how one can construct the scheme of iterative KD rounds to obtain the above expression for the limit model.

Original: From the form of the prediction, we conclude that one should use models obtained in the process of AKD.
Revised: From the form of the prediction, we conclude that one should use models obtained in the process of AKD.

Original: There are many possible schemes how one can combine these models to obtain the desired result.
Revised: There are many possible schemes how one can combine these models to obtain the desired result.

Original: One of the simplest possibilities is presented in the section 5.2.
Revised: One of the simplest possibilities is presented in the section 5.2.

Original: G L ABEL CORRECTION

Original: In the section F, it was shown that theoretically there is a way to obtain the ’ideal’ prediction for an unseen dataset on each agent’s side.

Original: But this requires an inﬁnite number of communications between agents tightened with the knowledge distillation on each side.

Original: Moreover, this ’ideal’ prediction requires the presence of subtraction terms in the expression which seems to be not tractable in highly non-linear cases such as the use of neural networks as the agents models.

Original: Given this, in this section we present another way of interpreting the expression (29).

Original: Let us take a closer look at the ’ideal’ prediction from the side of agent 1 with the simplifying assumption of the identical models case:

Original: [Equation]

Original: where the last rough approximation was obtained by omitting the matrix terms and taking some scalar β instead.

Original: The last expression highlights the following: from the perspective of agent 1, one of the closest predictions to the ’ideal’ prediction can be obtained by taking the average with some constant β of the ground truth label and the prediction of the agent 2 model obtained after supervised learning from its data.

Original: In a similar way this prediction can be expressed as follows:

Original: [Equation]

Original: where we notice the average of models of both agents obtained after supervised learning from corresponding agent’s data.

Original: Above approximations explain the picks which we observe in the experiments for AvgKD and PKD schemes in the ﬁrst round.

Original: At this round we explicitly take the above predictions with β = 12 to learn from.

Original: Meaning that the knowledge distillation is done from the logits that are close to the ’ideal’ ones according to (30), (31).

Original: Moreover, we observe that β is ’proportional’ to ( I − P (cid:62) 2 ) and 1 − β to ( I − P (cid:62) 1 ) .

Original: Let us notice that in case with the regularization used the number of non-zero eigenvalues for P 1 equals to number of agent’s 1 training data points N 1 .

Original: Hence we conclude that hyperparameter

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Alpha =

Original: CentralizedAgents mean

Original: AvgKDPKDFedAvgIdealKD

Original: [Figure]

Original: [Table]

Original: N 1 N

Original: [Equation]

Original: N 2 .

Original: Finally, we have the following approximation of the ’ideal’ prediction:

Original: [Equation]

Original: In the sense of this rough approximation, the models obtained by AvgKD and PKD schemes are optimal for the experiments with an equal split of data in terms of a number of points.

Original: In Fig.

Original: 8 we present the comparison results of the described scheme with others in CIFAR10 dataset with 70/ split of data between 2 agents.

Original: H M- AGENT SCHEMES
Revised: G M- AGENT SCHEMES

Original: The natural question to ask is how one could extend the discussed schemes to the setting of M agents.
Revised: The natural question to ask is how one could extend the discussed schemes to the setting of M agents.

Original: In this section, we address this question and explicitly provide the description of each algorithm for the setting of M agents.
Revised: In this section, we address this question and explicitly provide the description of each algorithm for the setting of M agents.

Original: Scalability and privacy.
Revised: Scalability and privacy.

Original: Before diving into particular algorithms we investigate some important concerns about our KD based framework.
Revised: Before diving into particular algorithms we investigate some important concerns about our KD based framework.

Original: A naive implementation of our methods will require each agent sharing their model with all other agents.
Revised: A naive implementation of our methods will require each agent sharing their model with all other agents.

Original: This will incur signiﬁcant communication costs ( M 2 ), storage costs (each agent has to store M models), and is not compatible with secure aggregation (Bonawitz et al., 2016) potentially leaking information.
Revised: This will incur signiﬁcant communication costs ( M 2 ), storage costs (each agent has to store M models), and is not compatible with secure aggregation (Bonawitz et al., 2016) potentially leaking information.

Original: One potential approach to alleviating these concerns is to use an server and homomorphic encryption (Graepel et al., 2012).
Revised: One potential approach to alleviating these concerns is to use an server and homomorphic encryption (Graepel et al., 2012).

Original: Homomorphic encryption allows agent 1 to compute predictions on agent 2’s data f 1 ( X 2 ) without learning anything about X 2 i.e. there exists a procedure Hom such that given encrypted data Enc ( X 2 ) , we can compute
Revised: Homomorphic encryption allows agent 1 to compute predictions on agent 2’s data f 1 ( X 2 ) without learning anything about X 2 i.e. there exists a procedure Hom such that given encrypted data Enc ( X 2 ) , we can compute

Original: Hom ( f 1 , Enc ( X 2 ))
Revised: Hom ( f 1 , Enc ( X 2 ))

Original: =
Revised: =

Original: Enc ( f 1 ( X 2 )) .
Revised: Enc ( f 1 ( X 2 )) .

Original: Given access to such a primitive, we can use a dedicated server (agent 0) to whom all models f 1 , . . .
Revised: Given access to such a primitive, we can use a dedicated server (agent 0) to whom all models f 1 , . . .

Original: , f M are sent.
Revised: , f M are sent.

Original: Let us deﬁne some weighted sum of the predictions as f α ( X ) :
Revised: Let us deﬁne some weighted sum of the predictions as f α ( X ) :

Original: = (cid:80)
Revised: = (cid:80)

Original: Mi =

Revised: Mi =

Original: α
Revised: α

Original: i f i ( X ) .
Revised: i f i ( X ) .

Original: Then, using homomorphic encryption, each agent i can compute Enc ( f α ( X i ))) in a private manner without leaking any information to agent 0.
Revised: Then, using homomorphic encryption, each agent i can compute Enc ( f α ( X i ))) in a private manner without leaking any information to agent 0.

Original: This makes the communication cost linear in M instead of quadratic, and also makes it more private and secure.
Revised: This makes the communication cost linear in M instead of quadratic, and also makes it more private and secure.

Original: A full ﬂedged investigation of the scalability, privacy, and security of such an approach is left for future work.
Revised: A full ﬂedged investigation of the scalability, privacy, and security of such an approach is left for future work.

Original: With this caveat out of the way, we next discuss some concrete algorithms.
Revised: With this caveat out of the way, we next discuss some concrete algorithms.

Original: AKD with M agents.
Revised: AKD with M agents.

Original: To extend AKD scheme to a multi-agent setting and corresponding theory we need to start by understanding what the alternating projection algorithm is in the case of M convex sets.
Revised: To extend AKD scheme to a multi-agent setting and corresponding theory we need to start by understanding what the alternating projection algorithm is in the case of M convex sets.

Original: Suppose we want to ﬁnd the intersection point of M afﬁne sets C i , for i = 1 , ..., M .
Revised: Suppose we want to ﬁnd the intersection point of M afﬁne sets C i , for i = 1 , ..., M .

Original: In terms of alternating projection algorithm we can write the following extension of it (Halperin, 1962):
Revised: In terms of alternating projection algorithm we can write the following extension of it (Halperin, 1962):

Original: [Equation]
Revised: [Equation]

Original: For our algorithm, it means that agent 1 passes its model to agent 2, then agent 2 passes its model to agent 3 and so until agent M that passes its model to agent 1, and then the cycle repeats.
Revised: For our algorithm, it means that agent 1 passes its model to agent 2, then agent 2 passes its model to agent 3 and so until agent M that passes its model to agent 1, and then the cycle repeats.

Original: That is, as before, we denote the data on agent 1 as D 1 =
Revised: That is, as before, we denote the data on agent 1 as D 1 =

Original: ( X 1 , y 1 ) where X 

Revised: ( X 1 , y 1 ) where X

Original: [ i, :] = x 1 n and y 1 [ i ] =
Revised: [ i, :] = x 1 n and y 1 [ i ] =

Original: y 1 i ., for all other agents we have D i =
Revised: y 1 i ., for all other agents we have D i =

Original: ( X i , y i ) , for i = 2 , ..., M .
Revised: ( X i , y i ) , for i = 2 , ..., M .

Original: Now starting from ˆ y 10 =
Revised: Now starting from ˆ y 10 =

Original: y 1 , in each rounds t, ..., t + M − 1 , t ≥ 0 :
Revised: y 1 , in each rounds t, ..., t + M − 1 , t ≥ 0 :

Original: a. Agent 1 trains their model on dataset ( X 1 , ˆ y 1 t ) to obtain g 1 t .
Revised: a. Agent 1 trains their model on dataset ( X 1 , ˆ y 1 t ) to obtain g 1 t .

Original: b. for i = 2 , ..., M : b.

Revised: b. for i = 2 , ..., M : b.

Original: Agent i receives g 1 t
Revised: Agent i receives g 1 t

Original: + i − 2 and uses it to predict labels
Revised: + i − 2 and uses it to predict labels

Original: ˆ y it
Revised: ˆ y it

Original: + i − 

Revised: + i −

Original: = g 1 t
Revised: = g 1 t

Original: + i − 2 ( X i ) .
Revised: + i − 2 ( X i ) .

Original: b.

Revised: b.

Original: Agent i trains their model on dataset ( X i , ˆ y it
Revised: Agent i trains their model on dataset ( X i , ˆ y it

Original: + i − 1 ) to obtain g 1 t
Revised: + i − 1 ) to obtain g 1 t

Original: + i − 1 .
Revised: + i − 1 .

Original: c. Agent 1 receives a model g 1 t + M − 1 from agent M and predicts ˆ y 1 t + M = g 1 t + M − 1 ( X 1 ) .
Revised: c. Agent 1 receives a model g 1 t + M − 1 from agent M and predicts ˆ y 1 t + M = g 1 t + M − 1 ( X 1 ) .

Original: As before, there is no exchange of raw data but only of the trained models.
Revised: As before, there is no exchange of raw data but only of the trained models.

Original: And given all the results deduced before, all the models from some point will start to degenerate.
Revised: And given all the results deduced before, all the models from some point will start to degenerate.

Original: The rate of convergence of such an algorithm is deﬁned similarly to alternating projection in case 2 sets and can be found in
Revised: The rate of convergence of such an algorithm is deﬁned similarly to alternating projection in case 2 sets and can be found in

Original: Smith et al.
Revised: Smith et al.

Original: PKD with M agents.
Revised: PKD with M agents.

Original: PKD scheme can be easily extended to the multi-agent setting analogously to how averaged projection algorithm can be extended to the multi-set setting (Lewis et al., 2007).
Revised: PKD scheme can be easily extended to the multi-agent setting analogously to how averaged projection algorithm can be extended to the multi-set setting (Lewis et al., 2007).

Original: Suppose we want to ﬁnd the intersection point of M afﬁne sets C i , for i
Revised: Suppose we want to ﬁnd the intersection point of M afﬁne sets C i , for i

Original: = 1 , ..., M then the in terms of averaged projection we have the following:[Equation]
Revised: = 1 , ..., M then the in terms of averaged projection we have the following:

Original: [Equation]
Revised: [Equation]

Original: This expression easily translates into the PKD algorithm for M agents.
Revised: This expression easily translates into the PKD algorithm for M agents.

Original: Denote the data on all agents as D i =
Revised: Denote the data on all agents as D i =

Original: ( X i , y i ) , for i = 2 , ..., M .
Revised: ( X i , y i ) , for i = 2 , ..., M .

Original: Now starting from ˆ y i 

Revised: Now starting from ˆ y i

Original: = y
Revised: = y

Original: i , for i = 2 , ..., M , in each round t ≥ 0 :
Revised: i , for i = 2 , ..., M , in each round t ≥ 0 :

Original: a. for i = 1 , ..., M: a.1 Agent i trains their model on dataset ( X i , ˆ y it ) to obtain g it .
Revised: a. for i = 1 , ..., M a.1 Agent i trains their model on dataset ( X i , ˆ y it ) to obtain g it .

Original: b. for i = 1 , ..., M : b.

Revised: b. for i = 1 , ..., M : b.

Original: Agent i receives models g jt , for j = 1 , ..., M, j (cid:54)
Revised: Agent i receives models g jt , for j = 1 , ..., M, j (cid:54)

Original: =
Revised: =

Original: i from all other agents.
Revised: i from all other agents.

Original: b.

Revised: b.

Original: Agent i use received models to predict ˆ
Revised: Agent i use received models to predict ˆ

Original: y
Revised: y

Original: it +

Revised: it +

Original: =
Revised: =

Original: ˆ y it + (cid:80)
Revised: ˆ y it + (cid:80)

Original: Mj
Revised: Mj

Original: =1 ,j (cid:54) =
Revised: =1 ,j (cid:54) =

Original: i g jt
Revised: i g jt

Original: ( X i ) M .
Revised: ( X i ) M .

Original: As in the case of 2 agents, there is no exchange of data between agents, but only models.
Revised: As in the case of 2 agents, there is no exchange of data between agents, but only models.

Original: This scheme requires all to all communication.
Revised: This scheme requires all to all communication.

Original: AvgKD with M agents.
Revised: AvgKD with M agents.

Original: Similarly to PKD algorithm we can extend AvgKD algorithm as follows: starting from ˆ y i 

Revised: Similarly to PKD algorithm we can extend AvgKD algorithm as follows: starting from ˆ y i

Original: = y
Revised: = y

Original: i , for i = 2 , ..., M , in each round t ≥ 0 :
Revised: i , for i = 2 , ..., M , in each round t ≥ 0 :

Original: a. for i = 1 , ..., M : a.1 Agent i trains their model on dataset ( X i , ˆ y it ) to obtain g it .
Revised: a. for i = 1 , ..., M : a.1 Agent i trains their model on dataset ( X i , ˆ y it ) to obtain g it .

Original: b. for i = 1 , ..., M : b.

Revised: b. for i = 1 , ..., M : b.

Original: Agent i receives models g jt , for j = 1 , ..., M, j (cid:54)
Revised: Agent i receives models g jt , for j = 1 , ..., M, j (cid:54)

Original: =
Revised: =

Original: i from all other agents.
Revised: i from all other agents.

Original: b.

Revised: b.

Original: Agent i use received models to predict ˆ
Revised: Agent i use received models to predict ˆ

Original: y
Revised: y

Original: it +

Revised: it +

Original: =
Revised: =

Original: y i + (cid:80)
Revised: y i + (cid:80)

Original: Mj
Revised: Mj

Original: =1 ,j (cid:54) =
Revised: =1 ,j (cid:54) =

Original: i g jt
Revised: i g jt

Original: ( X i ) M .
Revised: ( X i ) M .

Original: That is, there is no exchange of data between agents, but only models.
Revised: That is, there is no exchange of data between agents, but only models.

Original: This scheme as well as PKD requires all to all communication.
Revised: This scheme as well as PKD requires all to all communication.

Original: That means that the scheme can not be scaled, but it is still useful for small numbers of agents (e.g collaboration of companies).
Revised: That means that the scheme can not be scaled, but it is still useful for small numbers of agents (e.g collaboration of companies).

Original: The last is motivated by the simplicity of the scheme without any need for hyperparameter tuning, the non-degrading behavior as well as its superiority over the FedAvg scheme at highly heterogeneous data regimes.
Revised: The last is motivated by the simplicity of the scheme without any need for hyperparameter tuning, the non-degrading behavior as well as its superiority over the FedAvg scheme at highly heterogeneous data regimes.

Original: EKD with M agents.
Revised: EKD with M agents.

Original: EKD scheme in the case of 2 agents is based on models obtained in the process of 2 simultaneous runs of the AKD algorithm.
Revised: EKD scheme in the case of 2 agents is based on models obtained in the process of 2 simultaneous runs of the AKD algorithm.

Original: This means that the extension of EKD to the multi-agent setting, in this case, is straightforward by using the M simultaneous runs of AKD algorithm starting from each agent in the multi-agent setting and summing the obtained models in the process as follows:
Revised: This means that the extension of EKD to the multi-agent setting, in this case, is straightforward by using the M simultaneous runs of AKD algorithm starting from each agent in the multi-agent setting and summing the obtained models in the process as follows:

Original: [Equation]
Revised: [Equation]

Original: I A DDITIONAL EXPERIMENTS I. AKD WITH VARYING DATA HETEROGENEITY
Revised: H A DDITIONAL EXPERIMENTS H. MNIST WITH VARYING DATA HETEROGENEITY

Original: In this section [Figure] we present the results for AKD scheme with varying data heterogeneityon MNIST dataset. We vary Alpha hyperparameter in the experiments with AKD scheme on MNIST.
Revised: In this section, we present the results for MNIST dataset with varying data heterogeneity in the setting of ’different model’.

Original: The results one can see in Fig. 
Revised: The results one can see in Fig. 8.

Original: 9 which shows that the increase in data heterogeneity imposes faster degradation through rounds of AKD scheme which is aligned with the proposition IV. [Figure]
Revised: There is a faster degradation trend for both AKD and PKD schemes if different models for agents are used (Fig. 8) comparing to rhe ’same model’ setting (Fig. 6) at all data heterogeneity regimes.

Original: I. EKD ON MNIST
Revised: The PKD scheme is a slight modiﬁcation of the AvgKD scheme which is proven to degrade through rounds of distillation. We see the degradation trend for PKD scheme which is suggested by our theory presented in App. E. EKD does not improve with subsequent rounds in the setting of ’different models’. AvgKD scheme outperforms both PKD and AKD in all settings.

Original: In this section
Revised: However, its convergence is not stable in extremely high heterogeneous settings, showing large oscillations. Investigating and mitigating this could be interesting future work.

Original: [Figure]
Revised: [Figure]

Original:  we present the results of Ensembled KD (EKD) scheme on MNIST dataset.
Revised: In this section, we present the results for CIFAR10 dataset with varying data heterogeneity. Note that we use cross-entropy loss function here.

Original: The results are presented in Fig.
Revised: The results one can see in Fig.

Original: 10 for MSE loss and accuracy in comparison with centralized model and separate agents models.
Revised: 9 and 10 that again show data heterogeneity plays key role in the behavior of all the schemes.

Original: From theresults [Figure]
Revised: All the trends we saw on the

Original: we see that EKD consistently improves with the increase in the number of models used
Revised: MNIST dataset are repeated here except one: EKD does not improve in subsequent rounds in the ’same model’ setting.

Original: [Figure]
Revised: [Figure]

Original: I. C ROSS -E NTROPY OBJECTIVE
Revised: H. C ROSS -E NTROPY OBJECTIVE

Original: In this section, we present the results of experiments on MNIST with Cross-Entropy (CE) loss formain schemes under investigation: AKD and AvgKD.
Revised: In this section, we present the results of experiments on MNIST with Cross-Entropy (CE) loss formain schemes under investigation: AKD and AvgKD.

Original: In Fig.
Revised: In Fig.

Original: 11 and 12 one can see the results of
Revised: 11 and 12 one can see the results of

Original: AKD and AvgKD schemes correspondingly for CE loss.
Revised: AKD and AvgKD schemes correspondingly for CE loss.

Original: The results are aligned with our theory: in
Revised: The results are aligned with our theory: in

Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 11 we see the degradation trend for AKD which is dependent on the amount of the regularization, model and data heterogeneity.
Revised: 11 we see the degradation trend for AKD which is dependent on the amount of the regularization, model and data heterogeneity.

Original: In Fig.
Revised: In Fig.

Original: 12 we see steady behavior of AvgKD scheme for both agents models: there is no degradation even if model and data are different.
Revised: 12 we see steady behavior of AvgKD scheme for both agents models: there is no degradation even if model and data are different

Original: [Figure]
Revised: [Figure]

Original: In all cases, performance degrades with increasing rounds with degradation speeding up with  increase regularization model 
Revised: In all cases, performance degrades with increasing rounds with degradation speeding up with the increase in regularization, model heterogeneity

Original: [Figure]
Revised: [Figure]

Revised: or data heterogeneity

Original: [Figure]
Revised: [Figure]

Original: I. P ERFORMANCE OF A VG KD AND AKD ON CIFAR

Original: In this section, we present the results for CIFAR10 dataset for AKD scheme in Fig.

Original: 13 and 14 for

Original: MSE and CE losses correspondingly.

Original: One can see the same results for CIFAR dataset: degradation trend in AKD scheme for both MSE and CE losses and dependence on model and data heterogeneity of this trend.

Original: I. C OLLABORATION BETWEEN MLP S AND R ANDOM F ORESTS
Revised: H. C OLLABORATION BETWEEN MLP S AND R ANDOM F ORESTS

Original: In this section, we present the results for MNIST dataset for Random Forests (RF) and MLP models with MSE loss.
Revised: In this section, we present the results for MNIST dataset for Random Forests (RF) and MLP models with MSE loss.

Original: That is, the experiments are in the setting ’different models’, where agent 1 has MLP model and agent 2 has RF model.
Revised: That is, the experiments are in the setting ’different model’, where agent 1 has MLP model and agent 2 has RF model.

Original: These experiments can show how AKD and AvgKD schemes behave in the setting of fundamentally different models.
Revised: These experiments can show how AKD and AvgKD schemes behave in the setting of fundamentally different models.

Original: In the Figs. 15 and 16 the results for accuracy are presented.
Revised: In the Figs. 13 and 14 the results for accuracy are presented.

Original: We see the alignment of these results with theory and other experiments with deep learning models.
Revised: We see the alignment of these results with theory and other experiments with deep learning models.

Original: Mainly, there is a degradation trend for AKD scheme which is speeding up with the increase in data heterogeneity, there is no degradation for AvgKD scheme, and the performance of both agents in AvgKD scheme is highly dependent on data heterogeneity. I. A VG KD WITH M AGENTS
Revised: Mainly, there is a degradation trend for AKD scheme which is speeding up with the increase in data heterogeneity, there is no degradation for AvgKD scheme, and the performance of both agents in AvgKD scheme is highly dependent on data heterogeneity

Original: The AvgKD scheme does not degrade in comparison with AKD and PKD schemes that degrade already in the case of 2 agents.

Original: Let us experiment if AvgKD keeps its steady behavior in the case of M agents.

Original: In this section, we present the results of AvgKD scheme with M agents where each round for each agent we average its ground truth labels with the predictions of all other actors.

Original: Here

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: varying data heterogeneity.

Original: For AKD performance degrades with increasing rounds. Degradation is speeding up with the increase in data heterogeneity. For AvgKD there is no degradation of performance.
Revised: The AvgKD scheme does not degrade in comparison with AKD and PKD schemes that degrade already in the case of 2 agents.

Original:  we  use 5 agents on  MNIST dataset in the setting of the ’same model’ and vary data heterogeneity.Such experiments show how AvgKD model can perform in a more practical setting with a bigger number of agents.
Revised: In this section, we present the results of the AvgKD scheme with M agents and use 5 agents on the MNIST dataset in the setting of the ’same model’ with varying data heterogeneity.

Original: Moreover, we can understand if there is a dependence on data heterogeneity.

Original: In case of full data heterogeneity ( Alpha = 0 ) we assign the labels (2( i − 1) , 2 i − 1) to the actor number i, for i = 1 ... 5 .
Revised: In case of full data heterogeneity ( Alpha = 0 ) we assign the labels (2( i − 1) , 2 i − 1) to the actor number i, for i = 1 ... 5 .

Original: The results are presented in the Fig.
Revised: The results are presented in the Fig.

Original: We see that all the agents repeat the behavior [Figure] of each other and in all cases of data heterogeneity can beneﬁt from the AvgKD scheme:
Revised: We see that all the agents repeat the behavior pattern in all cases of data heterogeneity.

Original: in cases of Alpha < 0 .
Revised: In cases of Alpha < 0 .

Original: 05 (high data heterogeneity) early stopping is beneﬁcial [Figure] I.
Revised: 05 (high data heterogeneity) early stopping is beneﬁcial.

Original: C OMPARISOIN WITH PKD AND F ED A VG

Original: In this section, we present the results of PKD scheme deﬁned and analyzed in section E with 2 agents.

Original: The PKD scheme is a slight modiﬁcation of the AvgKD scheme which is proven to degrade through rounds of distillation.

Original: The comparison results of AvgKD and PKD are presented in the Fig.

Original: We see the degradation trend for PKD scheme which is suggested by our theory presented in section

Original: E. Moreover, AvgKD scheme outperforms both PKD and FedAvg in nearly all settings.

Original: However, its convergence is less stable in extremely high heterogeneous settings, showing large oscillations.

Original: Investigating and mitigating this could be interesting future work.

Original: [Figure]
Revised: [Figure]