Original: F EDERATED L EARNING FROM S MALL D ATASETS
Revised: F EDERATED L EARNING FROM S MALL D ATASETS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning allows multiple parties to collaboratively train a joint model without having to share any local data.
Revised: Federated learning allows multiple parties to collaboratively train a joint model without having to share any local data.

Original: It enables applications of machine learning in settings where data is inherently distributed and undisclosable, such as in the medical domain.
Revised: It enables applications of machine learning in settings where data is inherently distributed and undisclosable, such as in the medical domain.

Original: Joint training is usually achieved by aggregating local models.
Revised: Joint training is usually achieved by aggregating local models.

Original: When local datasets are small, locally trained models can vary greatly from a globally good model.
Revised: When local datasets are small, locally trained models can vary greatly from a globally good model.

Original: Bad local models can arbitrarily deteriorate the aggregate model quality, causing federating learning to fail in these settings.
Revised: Bad local models can arbitrarily deteriorate the aggregate model quality, causing federating learning to fail in these settings.

Original: We propose a novel approach that avoids this problem by interleaving model aggregation and permutation steps.
Revised: We propose a novel approach that avoids this problem by interleaving model aggregation and permutation steps.

Original: During a permutation step we redistribute local models across clients through the server, while preserving data privacy, to allow each local model to train on a daisy chain of local datasets.
Revised: During a permutation step we redistribute local models across clients through the server, while preserving data privacy, to allow each local model to train on a daisy chain of local datasets.

Original: This enables successful training in data-sparse domains.
Revised: This enables successful training in data-sparse domains.

Original: Combined with model aggregation, we so achieve effective learning even if the local datasets are extremely small, while retaining the privacy beneﬁts of federated learning.
Revised: Combined with model aggregation, this approach enables effective learning even if the local datasets are extremely small, while retaining the privacy beneﬁts of federated learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: How can we learn high quality models when data is inherently distributed across sites and cannot be shared or pooled?
Revised: How can we learn high quality models when data is inherently distributed across sites and cannot be shared or pooled?

Original: In federated learning, the solution is to iteratively train models locally at each site and share these models with the server to be aggregated to a global model.
Revised: In federated learning, the solution is to iteratively train models locally at each site and share these models with the server to be aggregated to a global model.

Original: As only models are shared, data usually remains undisclosed.
Revised: As only models are shared, data usually remains undisclosed.

Original: This process, however, requires sufﬁcient data to be available at each site in order for the locally trained models to achieve a minimum quality—even a single bad model can render aggregation arbitrarily bad (Shamir and Srebro, 2014).
Revised: This process, however, requires sufﬁcient data to be available at each site in order for the locally trained models to achieve a minimum quality—even a single bad model can render aggregation arbitrarily bad (Shamir and Srebro, 2014).

Original: In many relevant applications this requirement is not met:
Revised: In many relevant applications this requirement is not met:

Original: In healthcare settings we often have as little as a few dozens of samples (Granlund et al., 2020; Su et al., 2021;
Revised: In healthcare settings we often have as little as a few dozens of samples (Granlund et al., 2020; Su et al., 2021;

Original: Painter et al., 2020).
Revised: Painter et al., 2020).

Original: Also in domains where deep learning is generally regarded as highly successful, such as natural language processing and object detection, applications often suffer from a lack of data (Liu et al., 2020; Kang et al., 2019).
Revised: Also in domains where deep learning is generally regarded as highly successful, such as natural language processing and object detection, applications often suffer from a lack of data (Liu et al., 2020; Kang et al., 2019).

Original: To tackle this problem, we propose a new building block called daisy-chaining for federated learning in which models are trained on one local dataset after another, much like a daisy chain.
Revised: To tackle this problem, we propose a new building block called daisy-chaining for federated learning in which models are trained on one local dataset after another, much like a daisy chain.

Original: In a nutshell, at each client a model is trained locally, sent to the server, and then—instead of aggregating local models—sent to a random other client as is (see Fig. 1).
Revised: In a nutshell, at each client a model is trained locally, sent to the server, and then—instead of aggregating local models—sent to a random other client as is (see Fig. 1).

Original: This way, each local model is exposed to a daisy chain of clients and their local datasets.
Revised: This way, each local model is exposed to a daisy chain of clients and their local datasets.

Original: This allows us to learn from small, distributed datasets simply by consecutively training the model with the data available at each site.
Revised: This allows us to learn from small, distributed datasets simply by consecutively training the model with the data available at each site.

Original: Daisy-chaining alone, however, violates privacy, since a client can infer from a model upon the data of the client it received it from (Shokri et al., 2017).
Revised: Daisy-chaining alone, however, violates privacy, since a client can infer from a model upon the data of the client it received it from (Shokri et al., 2017).

Original: Moreover, performing daisy-chaining naively would lead to overﬁtting which can cause learning to diverge (Haddadpour and Mahdavi, 2019).
Revised: Moreover, performing daisy-chaining naively would lead to overﬁtting which can cause learning to diverge (Haddadpour and Mahdavi, 2019).

Original: In this paper, we propose to combine daisy-chaining of local datasets with aggregation of models, both orchestrated by the server, and term this method federated daisy-chaining (F ED DC).
Revised: In this paper, we propose to combine daisy-chaining of local datasets with aggregation of models, both orchestrated by the server, and term this method federated daisy-chaining (F ED DC).

Original: We show that our simple, yet effective approach maintains privacy of local datasets, while it provably converges and guarantees improvement of model quality in convex problems with a suitable aggregation method.
Revised: We show that our simple, yet effective approach maintains privacy of local datasets, while it provably converges and guarantees improvement of model quality in convex problems with a suitable aggregation method.

Original: Formally, we show convergence for F ED DC on non-convex problems.
Revised: Formally, we show convergence for F ED DC on non-convex problems.

Original: We then show for convex problems that F ED DC succeeds on small datasets where standard federated learning fails.
Revised: We then show for convex problems that F ED DC succeeds on small datasets where standard federated learning fails.

Original: For that, we analyze F ED DC combined with aggregation via the Radon point from a
Revised: For that, we analyze F ED DC combined with aggregation via the Radon point from a

Original: PAC-learning perspective.
Revised: PAC-learning perspective.

Original: We substantiate this theoretical analysis for convex problems by showing that F ED DC in practice matches the accuracy of a model trained on the full data of the SUSY binary classiﬁcation dataset with only 2 samples per client, outperforming standard federated learning by a wide margin.
Revised: We substantiate this theoretical analysis for convex problems by showing that F ED DC in practice matches the accuracy of a model trained on the full data of the SUSY binary classiﬁcation dataset with only 2 samples per client, outperforming standard federated learning by a wide margin.

Original: For non-convex settings, we provide an extensive empirical evaluation, showing thattwo real-world medical imaging datasets.
Revised: For non-convex settings, we provide an extensive empirical evaluation, showing that

Revised: F ED DC outperforms naive daisy-chaining, vanilla federated learning F ED A VG (McMahan et al., 2017), F ED P ROX (Li et al., 2020a), F ED A DAGRAD , F ED A DAM , and

Revised: F ED Y OGI (Reddi et al., 2020) on low-sample CIFAR10 (Krizhevsky, 2009), including non-iid settings, and, more importantly, on two real-world medical imaging datasets.

Original: Not only does F ED DC provide a wide margin of improvement over existing federated methods, but it comes close to the performance of a gold-standard (centralized) neural network of the same architecture trained on the pooled data.
Revised: Not only does F ED DC provide a wide margin of improvement over existing federated methods, but it comes close to the performance of a gold-standard (centralized) neural network of the same architecture trained on the pooled data.

Original: To achieve that, it requires a small communication overhead compared to standard federated learning for the additional daisy-chaining rounds.
Revised: To achieve that, it requires a small communication overhead compared to standard federated learning for the additional daisy-chaining rounds.

Original: As often found in healthcare, we consider a cross-SILO scenario where  small communication overhead is negligible.
Revised: As often found in healthcare, we consider a cross-SILO scenario where such small communication overhead is negligible.

Original: Moreover we show that with equal communication, standard federated averaging still underperforms in our considered settings.
Revised: Moreover we show that with equal communication, standard federated averaging still underperforms in our considered settings.

Original: In summary, our contributions are (i) F ED DC, a novel approach to federated learning from small datasets via a combination of model permutations across clients and aggregation, (ii) a formal proof of convergence for F ED DC, (iii) a theoretical guarantee that F ED DC improves models in terms of (cid:15), δ -guarantees which standard federated learning can not, (iv) a discussion of the privacy aspects and mitigations suitable for F ED DC, including an empirical evaluation of differentially private F ED DC, and (v) an extensive set of experiments showing that F ED DC substantially improves model quality on small datasets compared to standard federated learning approaches.
Revised: In summary, our contributions are (i) F ED DC, a novel approach to federated learning from small datasets via a combination of model permutations across clients and aggregation, (ii) a formal proof of convergence for F ED DC, (iii) a theoretical guarantee that F ED DC improves models in terms of (cid:15), δ -guarantees which standard federated learning can not, (iv) a discussion of the privacy aspects and mitigations suitable for F ED DC, including an empirical evaluation of differentially private F ED DC, and (v) an extensive set of experiments showing that F ED DC substantially improves model quality on small datasets compared to standard federated learning approaches.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning from small datasets is a well studied problem in machine learning.
Revised: Learning from small datasets is a well studied problem in machine learning.

Original: In the literature, we ﬁnd both general solutions, such as using simpler models and transfer learning (Torrey and Shavlik, 2010), and more specialized ones, such as data augmentation (Ibrahim et al., 2021) and few-shot learning (Vinyals et al., 2016; Prabhu et al., 2019).
Revised: In the literature, we ﬁnd both general solutions, such as using simpler models and transfer learning (Torrey and Shavlik, 2010), and more specialized ones, such as data augmentation (Ibrahim et al., 2021) and few-shot learning (Vinyals et al., 2016; Prabhu et al., 2019).

Original: In our scenario overall data is abundant, but the problem is that data is distributed into small local datasets at each site, which we are not allowed to pool.
Revised: In our scenario overall data is abundant, but the problem is that data is distributed into small local datasets at each site, which we are not allowed to pool.

Original: Yurochkin et al.
Revised: Hao et al.

Original: (2019) propose local data augmentation for federated learning, but their method requires a sufﬁcient quality of the local model for augmentation which is the opposite of the scenario we are considering.
Revised: (2021) propose local data augmentation for federated learning, but their method requires a sufﬁcient quality of the local model for augmentation which is the opposite of the scenario we are considering.

Original: Huang et al.
Revised: Huang et al.

Original: (2021) provide generalization bounds for federated averaging via the NTK-framework, but requires one-layer inﬁnite-width NNs and inﬁnitesimal learning rates.
Revised: (2021) provide generalization bounds for federated averaging via the NTK-framework, but requires one-layer inﬁnite-width NNs and inﬁnitesimal learning rates.

Original: Federated learning and its variants have been shown to learn from incomplete local data sources, e.g., non-iid label distributions (Li et al., 2020a;
Revised: Federated learning and its variants have been shown to learn from incomplete local data sources, e.g., non-iid label distributions (Li et al., 2020a;

Original: Wang et al., 2019) and differing feature distributions (Li et al., 2020b; Reisizadeh et al., 2020a), but fail in case of large gradient diversity (Haddadpour and Mahdavi, 2019) and strongly dissimilar label distribution (Marfoq et al., 2021).
Revised: Wang et al., 2019) and differing feature distributions (Li et al., 2020b; Reisizadeh et al., 2020a), but fail in case of large gradient diversity (Haddadpour and Mahdavi, 2019) and strongly dissimilar label distribution (Marfoq et al., 2021).

Original: For small datasets, local empirical distributions may vary greatly from the global distribution: the difference of empirical to true distribution decreases exponentially with the sample size (e.g., according to the
Revised: For small datasets, local empirical distributions may vary greatly from the global distribution: the difference of empirical to true distribution decreases exponentially with the sample size (e.g., according to the

Original: Dvoretzky–Kiefer–Wolfowitz inequality), but for small samples the difference can be substantial, in particular if the distribution differs from a Normal distribution (Kwak and Kim, 2017).
Revised: Dvoretzky–Kiefer–Wolfowitz inequality), but for small samples the difference can be substantial, in particular if the distribution differs from a Normal distribution (Kwak and Kim, 2017).

Original: Shamir and
Revised: Shamir and

Original: Srebro (2014) have shown the adverse effect of bad local models on averaging, proving that even due to a single bad model averaging can be arbitrarily bad.
Revised: Srebro (2014) have shown the adverse effect of bad local models on averaging, proving that even due to a single bad model averaging can be arbitrarily bad.

Original: A different approach to dealing with biased local data is by learning personalized models at each client.
Revised: A different approach to dealing with biased local data is by learning personalized models at each client.

Original: Such personalized FL (Achituve et al., 2021; Li et al., 2021) can reduce sample complexity, e.g., by using shared representations (Collins et al., 2021) for client-speciﬁc models, e.g., in the medical domain (Yang et al.,  2021).
Revised: Such personalized FL (Li et al., 2021) can reduce sample complexity, e.g., by using shared representations (Collins et al., 2021) for client-speciﬁc models, e.g., in the medical domain (Yang et al., 2021), or by training sample-efﬁcient personalized Bayesian methods (Achituve et al., 2021).

Original: It is not applicable, however, to settings where you are not allowed to learn the biases or batch effects of local clients, e.g., in many medical applications where this would expose sensitive client information.
Revised: It is not applicable, however, to settings where you are not allowed to learn the biases or batch effects of local clients, e.g., in many medical applications where this would expose sensitive client information.

Original: Kiss and Horvath (2021) propose a decentralized and communication-efﬁcient variant of federated learning that migrates models over a decentralized network, storing incoming models locally at each client until sufﬁciently many models are collected on each client for an averaging step, similar to Gossip federated learing (Jelasity et al., 2005).
Revised: Kiss and Horvath (2021) propose a decentralized and communication-efﬁcient variant of federated learning that migrates models over a decentralized network, storing incoming models locally at each client until sufﬁciently many models are collected on each client for an averaging step, similar to Gossip federated learing (Jelasity et al., 2005).

Original: The variant without averaging is similar to simple daisy-chaining which we compare to in Section 7.
Revised: The variant without averaging is similar to simple daisy-chaining which we compare to in Section 7.

Original: F ED DC is compatible with any aggregation operator, including the Radon machine (Kamp et al., 2017) and the geometric median (Pillutla et al., 2019) and can be straightforwardly combined with approaches to improve communication-efﬁciency, such as dynamic averaging (Kamp et al., 2018), and model quantization (Reisizadeh et al., 2020b).
Revised: F ED DC is compatible with any aggregation operator, including the Radon machine (Kamp et al., 2017), the geometric median (Pillutla et al., 2019), or neuron-clustering (Yurochkin et al., 2019), and can be straightforwardly combined with approaches to improve communication-efﬁciency, such as dynamic averaging (Kamp et al., 2018), and model quantization (Reisizadeh et al., 2020b).

Original: We combine F ED DC with averaging, the Radon machine, and FedProx (Li et al., 2020a) in Sec. 7.
Revised: We combine F ED DC with averaging, the Radon machine, and FedProx (Li et al., 2020a) in Sec. 7.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We assume iterative learning algorithms (cf.
Revised: We assume iterative learning algorithms (cf.

Original: Chp.
Revised: Chp.

Original: 2.1.4 Kamp, 2019)
Revised: 2.1.4 Kamp, 2019)

Original: A : X × Y × H → H that update a model h ∈ H using a dataset D ⊂ X × Y from an input space X and output space Y , i.e., h t +

Revised: A : X × Y × H → H that update a model h ∈ H using a dataset D ⊂ X × Y from an input space X and output space Y , i.e., h t +

Original: =
Revised: =

Original: A ( D, h t ) .
Revised: A ( D, h t ) .

Original: Given a set of m ∈ N clients with local datasets D 1 , . . .
Revised: Given a set of m ∈ N clients with local datasets D 1 , . . .

Original: , D m ⊂ X × Y drawn iid from a data distribution D and a loss function (cid:96) :
Revised: , D m ⊂ X × Y drawn iid from a data distribution D and a loss function (cid:96) :

Original: Y × Y → R , the goal is to ﬁnd a single model h ∗ ∈ H that minimizes the risk ε ( h ) =
Revised: Y × Y → R , the goal is to ﬁnd a single model h ∗ ∈ H that minimizes the risk ε ( h ) =

Original: E ( x,y ) ∼D
Revised: E ( x,y ) ∼D

Original: [ (cid:96) ( h ( x ) , y )] .
Revised: [ (cid:96) ( h ( x ) , y )] .

Original: In centralized learning , datasets are pooled as D =
Revised: In centralized learning , datasets are pooled as D =

Original: (cid:83)
Revised: (cid:83)

Original: i ∈ [ m ] D i and A is applied to D until convergence.
Revised: i ∈ [ m ] D i and A is applied to D until convergence.

Original: Note that applying A on D can be the application to any random subset, e.g., as in mini-batch training, and convergence is measured in terms of low training loss, small gradient, or small deviation from previous iterate.
Revised: Note that applying A on D can be the application to any random subset, e.g., as in mini-batch training, and convergence is measured in terms of low training loss, small gradient, or small deviation from previous iterate.

Original: In standard federated learning (McMahan et al., 2017), A is applied in parallel for b ∈ N rounds on each client locally to produce local models h 1 , . . .
Revised: In standard federated learning (McMahan et al., 2017), A is applied in parallel for b ∈ N rounds on each client locally to produce local models h 1 , . . .

Original: , h m .
Revised: , h m .

Original: These models are then centralized and aggregated using an aggregation operator agg :
Revised: These models are then centralized and aggregated using an aggregation operator agg :

Original: H m → H , i.e., h = agg( h 1 , . .
Revised: H m → H , i.e., h = agg( h 1 , . .

Original: h m ) .
Revised: h m ) .

Original: The aggregated model h is then redistributed to local clients which perform another b rounds of training using h as a starting point.
Revised: The aggregated model h is then redistributed to local clients which perform another b rounds of training using h as a starting point.

Original: This is iterated until convergence of h .
Revised: This is iterated until convergence of h .

Original: When aggregating by averaging, this method is known as federated averaging (F ED A VG ).
Revised: When aggregating by averaging, this method is known as federated averaging (F ED A VG ).

Original: Next, we describe F ED DC.
Revised: Next, we describe F ED DC.

Original:  F EDERATED D AISY -C HAINING
Revised:  F EDERATED D AISY -C HAINING

Original: We propose federated daisy chaining as an extension to federated learning in a setup with m clients and one designated sever.
Revised: We propose federated daisy chaining as an extension to federated learning in a setup with m clients and one designated sever.

Original: 1 We provide the pseudocode of our approach as Algorithm 1.
Revised: 1 We provide the pseudocode of our approach as Algorithm 1.

Original: The client:
Revised: The client:

Original: Each client trains its local model in each round on local data (line 4), and sends its model to the server every b rounds for aggregation, where b is the aggregation period, and every d rounds for daisy chaining, where d is the daisy-chaining period (line 6).
Revised: Each client trains its local model in each round on local data (line 4), and sends its model to the server every b rounds for aggregation, where b is the aggregation period, and every d rounds for daisy chaining, where d is the daisy-chaining period (line 6).

Original: This re-distribution of models results in each individual model conceptually following a daisy chain of clients, training on each local dataset.
Revised: This re-distribution of models results in each individual model conceptually following a daisy chain of clients, training on each local dataset.

Original: Such a daisy chain is interrupted by each aggregation round.
Revised: Such a daisy chain is interrupted by each aggregation round.

Original: The server: Upon receiving models, in a daisy-chaining round (line 10) the server draws a random permutation π of clients (line 11) and re-distributes the model of client i to client π ( i ) (line 12), while in an aggregation round (line 14), the server instead aggregates all local models and re-distributes the aggregate to all clients (line 15).
Revised: The server: Upon receiving models, in a daisy-chaining round (line 9) the server draws a random permutation π of clients (line 10) and re-distributes the model of client i to client π ( i ) (line 11), while in an aggregation round (line 12), the server instead aggregates all local models and re-distributes the aggregate to all clients (line 13-14).

Original: Communication complexity: Note that we consider cross-SILO settings, such as healthcare, were communication is not a bottleneck and, hence, restrict ourselves to a brief discussion in the interest of space.
Revised: Communication complexity: Note that we consider cross-SILO settings, such as healthcare, were communication is not a bottleneck and, hence, restrict ourselves to a brief discussion in the interest of space.

Original: Communication between clients and server happens in O ( Td + Tb ) many rounds, where T is the overall number of rounds.
Revised: Communication between clients and server happens in O ( Td + Tb ) many rounds, where T is the overall number of rounds.

Original: Since F ED DC communicates every d th and b th round, the amount of communication rounds is similar to F ED A VG with averaging period b FedAvg = min { d, b } .
Revised: Since F ED DC communicates every d th and b th round, the amount of communication rounds is similar to F ED A VG with averaging period b FedAvg = min { d, b } .

Original: That is,
Revised: That is,

Original: F ED DC increases communication over F ED A VG by a constant factor depending on the setting of b and d .
Revised: F ED DC increases communication over F ED A VG by a constant factor depending on the setting of b and d .

Original: The amount of communication per communication round is linear in the number of clients and model size, similar to federated averaging.
Revised: The amount of communication per communication round is linear in the number of clients and model size, similar to federated averaging.

Original: We investigate the performance of F ED A VG provided with the same communication capacity as F ED DC in our experiments and in App.
Revised: We investigate the performance of F ED A VG provided with the same communication capacity as F ED DC in our experiments and in App.

Original: A.1.6.
Revised: A.3.6.

Original: m2 Locally time t do sample i h

Original: it ← A

Original: S

Original: ) if mod

Original: send

Original: // receives h t jt 8 At time t do

Original: // aggregation

Original:  T HEORETICAL GUARANTEES
Revised:  T HEORETICAL GUARANTEES

Original: In this section, we formally show that F ED DC converges for averaging.
Revised: In this section, we formally show that F ED DC converges for averaging.

Original: We, further, provide theoretical bounds on the model quality in convex settings, showing that F ED DC has favorable generalization error in low sample settings compared to standard federated learning.
Revised: We, further, provide theoretical bounds on the model quality in convex settings, showing that F ED DC has favorable generalization error in low sample settings compared to standard federated learning.

Original: More formally, we ﬁrst show that under standard assumptions on the empirical risk, it follows from a result of Yu et al.
Revised: More formally, we ﬁrst show that under standard assumptions on the empirical risk, it follows from a result of Yu et al.

Original: (2019) that F ED DC converges when using averaging as aggregation and SGD for learning—a standard setting in, e.g., federated learning of neural networks.
Revised: (2019) that F ED DC converges when using averaging as aggregation and SGD for learning—a standard setting in, e.g., federated learning of neural networks.

Original: We provide all proofs in the appendix.
Revised: We provide all proofs in the appendix.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let the empirical risks E iemp ( h )
Revised: Let the empirical risks E iemp ( h )

Original: = (cid:80) ( x,y ) ∈
Revised: = (cid:80) ( x,y ) ∈

Original: D i (cid:96) ( h i ( x ) , y ) at each client i ∈ [ m ] be
Revised: D i (cid:96) ( h i ( x ) , y ) at each client i ∈ [ m ] be

Original: L -smooth with σ 2 -bounded gradient variance and G 2 -bounded second moments, then F ED DC with averaging and SGD has a convergence rate of O (1 / √ mT ) , where T is the number of local updates.
Revised: L -smooth with σ 2 -bounded gradient variance and G 2 -bounded second moments, then F ED DC with averaging and SGD has a convergence rate of O (1 / √ mT ) , where T is the number of local updates.

Original: Since model quality in terms of generalization error does not necessarily depend on convergence of training (Haddadpour and Mahdavi, 2019; Kamp et al., 2018), we additionally analyze model quality in terms of probabilistic worst-case guarantees on the generalization error (Shalev-Shwartz and Ben-David, 2014).
Revised: Since model quality in terms of generalization error does not necessarily depend on convergence of training (Haddadpour and Mahdavi, 2019; Kamp et al., 2018), we additionally analyze model quality in terms of probabilistic worst-case guarantees on the generalization error (Shalev-Shwartz and Ben-David, 2014).

Original: The average of local models can yield as bad a generalization error as the worst local model, hence, using averaging as aggregation scheme in standard federated learning can yield arbitrarily bad results (cf.
Revised: The average of local models can yield as bad a generalization error as the worst local model, hence, using averaging as aggregation scheme in standard federated learning can yield arbitrarily bad results (cf.

Original: Shamir and Srebro, 2014).
Revised: Shamir and Srebro, 2014).

Original: As the probability of bad local models starkly increases with smaller sample sizes, this trivial bound often carries over to our considered practical settings.
Revised: As the probability of bad local models starkly increases with smaller sample sizes, this trivial bound often carries over to our considered practical settings.

Original: The Radon machine (Kamp et al., 2017) is a federated learning approach that overcomes these issues for a wide range of learning algorithms and allows us to analyze (non-trivial) quality bounds of aggregated models under the assumption of convexity.
Revised: The Radon machine (Kamp et al., 2017) is a federated learning approach that overcomes these issues for a wide range of learning algorithms and allows us to analyze (non-trivial) quality bounds of aggregated models under the assumption of convexity.

Original: Next, we show that F ED DC can improve model quality for small local datasets where standard federated learning fails to do so.
Revised: Next, we show that F ED DC can improve model quality for small local datasets where standard federated learning fails to do so.

Original: A Radon point (Radon, 1921) of a set of points S from a space X is—similar to the geometric median—a point in the convex hull of S with a high centrality (more precisely, a Tukey depth (Tukey, 1975; Gilad-Bachrach et al., 2004) of at least 2 ).
Revised: A Radon point (Radon, 1921) of a set of points S from a space X is—similar to the geometric median—a point in the convex hull of S with a high centrality (i.e [Figure] Gilad-Bachrach et al., 2004) of at least 2 ).

Original: For a Radon point to exist,the size of S  has to be sufﬁciently large; the minimum size of S ⊂ X is denoted the Radon number ofthe space X and for
Revised: For a Radon point to exist, S ⊂ X has to have a minimum size r ∈ N called the Radon number of X .

Original:  X ⊆ R d the radon number is d + 2 .
Revised: For X ⊆ R d the radon number is d + 2 .

Original: Here, the set of points S are the local models, or more precisely their parameter vectors.
Revised: Here, the set of points S are the local models, or more precisely their parameter vectors.

Original: We make the following standard assumption (Von Luxburg and Sch¨olkopf, 2011) on the local learning algorithm A .
Revised: We make the following standard assumption (Von Luxburg and Sch¨olkopf, 2011) on the local learning algorithm A .

Original: Assumption 2 ( ( (cid:15), δ ) -guarantees) .
Revised: Assumption 2 ( ( (cid:15), δ ) -guarantees) .

Original: The learning algorithm
Revised: The learning algorithm

Original: A applied on a dataset drawn iid from D of size n ≥ n 0 ∈ N produces a model h ∈ H s.t.
Revised: A applied on a dataset drawn iid from D of size n ≥ n 0 ∈ N produces a model h ∈ H s.t.

Original: with probability δ ∈ (0 , 1] it holds for (cid:15) > 

Revised: with probability δ ∈ (0 , 1] it holds for (cid:15) >

Original: that P ( ε ( h ) > (cid:15) ) <
Revised: that P ( ε ( h ) > (cid:15) ) <

Original: δ .
Revised: δ .

Original: The sample size n 0 is monotonically decreasing in δ and (cid:15) (note that typically nis a polynomial in (cid:15) − 1 and log( δ − 1 ) ).
Revised: The sample size n 0 is monotonically decreasing in δ and (cid:15) (note that typically nis a polynomial in (cid:15) − 1 and log( δ − 1 ) ).

Original: Here ε ( h ) is the risk deﬁned in Sec. 3.
Revised: Here ε ( h ) is the risk deﬁned in Sec. 3.

Original: Now let r ∈ N be the Radon number of H ,
Revised: Now let r ∈ N be the Radon number of H ,

Original: A be a learning algorithm as in assumption 2, and risk ε be convex.
Revised: A be a learning algorithm as in assumption 2, and risk ε be convex.

Original: Assume m ≥ r h many clients with h ∈ N .
Revised: Assume m ≥ r h many clients with h ∈ N .

Original: For[Figure] (cid:15) > 0 , δ ∈ (0 , 1] assume local datasets D 1 , . . 
Revised: For (cid:15) > 0 , δ ∈ (0 , 1] assume local datasets D 1 , . . .

Original: , D m of size larger than n 0 ( (cid:15), δ ) drawn iid from D , and h 1 , . .
Revised: , D m of size larger than n 0 ( (cid:15), δ ) drawn iid from D , and h 1 , . .

Original: , h m be local models trained on them using A .
Revised: , h m be local models trained on them using A .

Original: Let r h be the iterated Radon point (Clarkson et al., 1996) with h iterations computed on the local models (for details, see App. A.8).
Revised: Let r h be the iterated Radon point (Clarkson et al., 1996) with h iterations computed on the local models (for details, see App. A.2).

Original: Then it follows from Theorem 3 in Kamp et al.
Revised: Then it follows from Theorem 3 in Kamp et al.

Original: (2017) that for all i ∈ [ m ] it holds that
Revised: (2017) that for all i ∈ [ m ] it holds that

Original: [Equation]
Revised: [Equation]

Original: (1) where the probability is over the random draws of local datasets.
Revised: where the probability is over the random draws of local datasets.

Original: That is, the probability that the aggregate r h is bad is doubly-exponentially smaller than the probability that a local model is bad.
Revised: That is, the probability that the aggregate r h is bad is doubly-exponentially smaller than the probability that a local model is bad.

Original: Note that in PAC-learning, the error bound and the probability of the bound to hold are typically linked, so that improving one can be translated to improving the other (Von Luxburg and Sch¨olkopf, 2011).
Revised: Note that in PAC-learning, the error bound and the probability of the bound to hold are typically linked, so that improving one can be translated to improving the other (Von Luxburg and Sch¨olkopf, 2011).

Original: Eq.
Revised: Eq.

Original: 1 implies that the iterated Radon point only improves the guarantee on the conﬁdence compared to that for local models if δ < r − 1 , i.e. P ( ε ( r h ) > (cid:15) ) ≤ ( r P ( ε ( h i ) > (cid:15) ))
Revised: 1 implies that the iterated Radon point only improves the guarantee on the conﬁdence compared to that for local models if δ < r − 1 , i.e. P ( ε ( r h ) > (cid:15) ) ≤ ( r P ( ε ( h i ) > (cid:15) ))

Original: 2 h < ( rδ ) 2 h <only holds for rδ < 1 .
Revised: 2 h < ( rδ ) 2 h <only holds for rδ < 1 .

Original: Consequently, local models need to achieve a minimum quality for the federated learning system to improve model quality.
Revised: Consequently, local models need to achieve a minimum quality for the federated learning system to improve model quality.

Original: Corollary 3.
Revised: Corollary 3.

Original: Let H be a model space with Radon number r ∈ N , ε a convex risk, and A a learning algorithm with sample size n 0 ( (cid:15), δ ) .
Revised: Let H be a model space with Radon number r ∈ N , ε a convex risk, and A a learning algorithm with sample size n 0 ( (cid:15), δ ) .

Original: Given (cid:15) > 0 and any h ∈ N , if local datasets D 1 , . . .
Revised: Given (cid:15) > 0 and any h ∈ N , if local datasets D 1 , . . .

Original: , D m with m ≥ r h are smaller than n 0 ( (cid:15), r − 1 ) , then federated learning using the Radon point does not improve model quality in terms of ( (cid:15), δ ) -guarantees.
Revised: , D m with m ≥ r h are smaller than n 0 ( (cid:15), r − 1 ) , then federated learning using the Radon point does not improve model quality in terms of ( (cid:15), δ ) -guarantees.

Original: In other words, when using aggregation by Radon points alone, an improvement in terms of ( (cid:15), δ ) - guarantees is strongly dependent on large enough local datasets.
Revised: In other words, when using aggregation by Radon points alone, an improvement in terms of ( (cid:15), δ ) - guarantees is strongly dependent on large enough local datasets.

Original: Furthermore, given δ > r − 1 , the guarantee can become arbitrarily bad by increasing the number of aggregation rounds.
Revised: Furthermore, given δ > r − 1 , the guarantee can become arbitrarily bad by increasing the number of aggregation rounds.

Original: Federated Daisy-Chaining as given in Alg.
Revised: Federated Daisy-Chaining as given in Alg.

Original: 1 permutes local models at random, which is in theory equivalent to permuting local datasets.
Revised: 1 permutes local models at random, which is in theory equivalent to permuting local datasets.

Original: Since the permutation is drawn at random, the amount of permutation rounds T necessary for each model to observe a minimum number of distinct datasets k  can be given with high probability via a variation of the coupon collector problem. Lemma 4. Given δ ∈ (0 , 1] , m ∈ N clients, and k ∈ [ m ] , if Algorithm 1 with daisy chaining period d ∈ N is run for T ∈ N rounds with
Revised: Since the permutation is drawn at random, the amount of permutation rounds T necessary for each model to observe a minimum number of distinct datasets k with probability 1 − ρ can be given with high probability via a variation of the coupon collector problem as T ≥ d m ρ 1 m ( H m − H m − k ) , where H m is the m -th harmonic number—see Lm. 5 in

Original: [Equation]
Revised: App.

Original: then each local model has seen at least k distinct datasets with probability 1 − γ . From Lem.
Revised: A.5 for details.

Original: 4 it follows that when we perform daisy-chaining with m clients and local datasets of size n for at least d ln γ ((ln( m − 1) − ln ( m)) (  m − k + 1) m) − 1 rounds, then each local model will with probability at least 1 − γ be trained on at least kn distinct samples.
Revised: It follows that when we perform daisy-chaining with m clients and local datasets of size n for at least dmρ − 1 m ( H m − H m − k ) rounds, then each local model will with probability at least 1 − ρ be trained on at least kn distinct samples.

Original: For an (cid:15), δ -guarantee, we thus need to set b large enough so that kn ≥ n 0 ( (cid:15), √ δ ) with probability at least 1 − √ δ .
Revised: For an (cid:15), δ -guarantee, we thus need to set b large enough so that kn ≥ n 0 ( (cid:15), √ δ ) with probability at least 1 − √ δ .

Original: This way, the failure probability is the product of not all clients observing k distinct datasets and the model having a risk larger than (cid:15) , which is √ δ √ δ = δ .
Revised: This way, the failure probability is the product of not all clients observing k distinct datasets and the model having a risk larger than (cid:15) , which is √ δ √ δ = δ .

Original: Proposition 5.
Revised: Proposition 4.

Original: Let H be a model space with Radon number r ∈ N , ε a convex risk , and A a learning algorithm with sample size n 0 ( (cid:15), δ ) .
Revised: Let H be a model space with Radon number r ∈ N , ε a convex risk , and A a learning algorithm with sample size n 0 ( (cid:15), δ ) .

Original: Given (cid:15) > 0 , δ ∈ (0 , r − 1 ) and any h ∈ N , and local datasets
Revised: Given (cid:15) > 0 , δ ∈ (0 , r − 1 ) and any h ∈ N , and local datasets

Original: D 1 , . . .
Revised: D 1 , . . .

Original: , D m of size n ∈ N with m ≥ r h , then Alg.
Revised: , D m of size n ∈ N with m ≥ r h , then Alg.

Original: 1 using the Radon point with aggr.
Revised: 1 using the Radon point with aggr.

Original: period
Revised: period

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: improves model quality in terms of ( (cid:15), δ ) -guarantees.
Revised: improves model quality in terms of ( (cid:15), δ ) -guarantees.

Original: This result implies that if enough daisy-chaining rounds are performed in-between aggregation rounds, federated learning via the iterated Radon point improves model quality in terms of ( (cid:15), δ ) -guarantees: the resulting model has generalization error smaller than (cid:15) with probability at least 1 − δ .
Revised: This result implies that if enough daisy-chaining rounds are performed in-between aggregation rounds, federated learning via the iterated Radon point improves model quality in terms of ( (cid:15), δ ) -guarantees: the resulting model has generalization error smaller than (cid:15) with probability at least 1 − δ .

Revised: Note that the aggregation period cannot be arbitrarily increased without harming convergence.

Revised: To illustrate the interplay between these variables, we provide a numerical analysis of Prop. 4 in App. A.5.1.

Original: This theoretical result is also evident in practice, as we show in Fig.
Revised: This theoretical result is also evident in practice, as we show in Fig.

Original: There, we compare F ED DC with standard federated learning and equip both with the iterated Radon point on the SUSY binary classiﬁcation dataset (Baldi et al., 2014).
Revised: There, we compare F ED DC with standard federated learning and equip both with the iterated Radon point on the SUSY binary classiﬁcation dataset (Baldi et al., 2014).

Original: We train a linear model on 441 clients with onlysamples per client.
Revised: We train a linear model on 441 clients with only 2 samples per client.

Original: After 500 rounds F ED DC daisy-chaining every round ( d = 1 ) and aggregating every  b = 50 rounds reached the test accuracy of a gold-standard model that has been trained on the centralized dataset (ACC= 0 . 77 ).
Revised: After 500 rounds F ED DC daisy-chaining every round ( d = 1 ) and aggregating every ﬁfty rounds ( b = 50 ) reached the test accuracy of a gold-standard model that has been trained on the centralized dataset (ACC= 0 . 77 ).

Original: Standard federated learning with the same communication complexity using b = 1 is outperformed by a large margin (ACC= 0 . 68 ).
Revised: Standard federated learning with the same communication complexity using b = 1 is outperformed by a large margin (ACC= 0 . 68 ).

Original: We additionally provide results of standard federated learning with b = 50 (ACC= 0 . 64 ), which shows that while the aggregated models perform reasonable, the standard approach heavily overﬁts on local datasets if not pulled to a global average in every round.
Revised: We additionally provide results of standard federated learning with b = 50 (ACC= 0 . 64 ), which shows that while the aggregated models perform reasonable, the standard approach heavily overﬁts on local datasets if not pulled to a global average in every round.

Original: More details on this experiment can be found in App. A.1.2.
Revised: More details on this experiment can be found in App. A.3.2.

Original: In Sec.we show that the empirical results for averaging as aggregation operator are similar to those for the
Revised: In Sec.we show that the empirical results for averaging as aggregation operator are similar to those for the

Original: Radon machine.
Revised: Radon machine.

Original: First, we discuss the privacy-aspects of F ED DC.
Revised: First, we discuss the privacy-aspects of F ED DC.

Original:  D ATA P RIVACY
Revised:  D ATA P RIVACY

Original: [Figure]
Revised: [Figure]

Original: A major advantage of federated over centralized learning is that local data remains undisclosed to anyone but the local client, only model parameters are exchanged.
Revised: A major advantage of federated over centralized learning is that local data remains undisclosed to anyone but the local client, only model parameters are exchanged.

Original: This provides a natural beneﬁt to data privacy, which is the main concern in applications such as healthcare.
Revised: This provides a natural beneﬁt to data privacy, which is the main concern in applications such as healthcare.

Original: However, an attacker can make inferences about local data from model parameters (Ma et al., 2020) and model updates or gradients (Zhu and Han, 2020).
Revised: However, an attacker can make inferences about local data from model parameters (Ma et al., 2020) and model updates or gradients (Zhu and Han, 2020).

Original: In the daisychaining rounds of F ED DC clients receive a model that was directly trained on the local data of another client, instead of a model aggregate, potentially facilitating membership inference attacks (Shokri et al., 2017)—reconstruction attacks (Zhu and Han, 2020) remain difﬁcult because model updates cannot be inferred since the server randomly permutes the order of clients in daisy-chaining rounds.
Revised: In the daisy-chaining rounds of F ED DC clients receive a model that was directly trained on the local data of another client, instead of a model aggregate, potentially facilitating membership inference attacks (Shokri et al., 2017)—reconstruction attacks (Zhu and Han, 2020) remain difﬁcult because model updates cannot be inferred since the server randomly permutes the order of clients in daisy-chaining rounds.

Original: Should a malicious client obtain model updates through additional attacks, a common defense is applying appropriate clipping and noise before sending models.
Revised: Should a malicious client obtain model updates through additional attacks, a common defense is applying appropriate clipping and noise before sending models.

Original: This guarantees (cid:15), δ -differential privacy for local data (Wei et al., 2020) at the cost of a slight-to-moderate loss in model quality.
Revised: This guarantees (cid:15), δ -differential privacy for local data (Wei et al., 2020) at the cost of a slight-to-moderate loss in model quality.

Original: This technique is also proven to defend against backdoor and poisoning attacks (Sun et al., 2019).
Revised: This technique is also proven to defend against backdoor and poisoning attacks (Sun et al., 2019).

Original: Moreover, F ED DC is compatible with standard defenses against such attacks, such as noisy or robust aggregation (Liu et al., 2022)—F ED DC with the Radon machine is an example ofsuch a robust aggregation.
Revised: Moreover, F ED DC is compatible with standard defenses against such attacks, such as noisy or robust aggregation (Liu et al., 2022)—F ED DC with the Radon machine is an example of robust aggregation.

Original: We illustrate the effectiveness of F ED DC  with differential privacytechniques in the following experiment.
Revised: We illustrate the effectiveness of F ED DC [Figure] with differential privacy in the following experiment.

Original: We train a small ResNet on 250 clients using F ED DC with d = 2 and b = 10 , postponing the details on the experimental setup to App. 
Revised: We train a small ResNet on 250 clients using F ED DC with d = 2 and b = 10 , postponing the details on the experimental setup to App. A.1.

Original: A.7.1 and A.7.2.
Revised: and A.1.2.

Original: Differential privacy is achieved by clipping local model updates and adding Gaussian noise as proposed by Geyer et al.
Revised: Differential privacy is achieved by clipping local model updates and adding Gaussian noise as proposed by Geyer et al.

Original: The results as shown in Figure 3 indicate that the standard trade-off between model quality and privacy holds for F ED DC as well.
Revised: The results as shown in Figure 3 indicate that the standard trade-off between model quality and privacy holds for F ED DC as well.

Original: Moreover, for mild privacy settings the model quality does not decrease.
Revised: Moreover, for mild privacy settings the model quality does not decrease.

Original: That is, F ED DC is able to robustly predict even under differential privacy.
Revised: That is, F ED DC is able to robustly predict even under differential privacy.

Original: We provide an extended discussion on the privacy aspects of F ED DC in App.A.6.
Revised: We provide an extended discussion on the privacy aspects of F ED DC in App.

Original: [Figure]
Revised: A.7.

Original:  E XPERIMENTS ON D EEP L EARNING
Revised:  E XPERIMENTS ON D EEP L EARNING

Original: Our approach F ED DC, both provably and empirically, improves model quality when using Radon points as aggregation which, however, require convex problems.
Revised: Our approach F ED DC, both provably and empirically, improves model quality when using Radon points as aggregation which, however, require convex problems.

Original: For non-convex problems, in particular deep learning, averaging is the state-of-the-art aggregation operator.
Revised: For non-convex problems, in particular deep learning, averaging is the state-of-the-art aggregation operator.

Original: We, hence, evaluate
Revised: We, hence, evaluate

Original: F ED DC with averaging against the state of the art in federated learning on synthetic and real world data using neural networks.
Revised: F ED DC with averaging against the state of the art in federated learning on synthetic and real world data using neural networks.

Original: As baselines, we consider federated averaging (F ED A VG )
Revised: As baselines, we consider federated averaging (F ED A VG )

Original: (McMahan et al., 2017) with optimal communication, F ED A VG with equal communication as F ED DC, and simple daisy-chaining without aggregation.
Revised: (McMahan et al., 2017) with optimal communication, F ED A VG with equal communication as F ED DC, and simple daisy-chaining without aggregation.

Original: We further consider the 4 state-of-the-art methods F ED P ROX (Li et al., 2020a), F ED A DAGRAD , F ED Y OGI , and F ED A DAM (Reddi et al., 2020).
Revised: We further consider the 4 state-of-the-art methods F ED P ROX (Li et al., 2020a), F ED A DAGRAD , F ED Y OGI , and F ED A DAM (Reddi et al., 2020).

Original: As datasets we consider a synthetic classiﬁcation dataset as a simple test for vanilla neural networks, the image classiﬁcation [-problem CIFAR
-] (Krizhevsky, 2009), and two real-world applications: (i) publicly available MRI scans for brain tumors 2, and(ii) chest X-rays for pneumonia 3 (e.g., from COVID-19).
Revised: As datasets we consider a synthetic classiﬁcation dataset, image classiﬁcation [-problem CIFAR
-]in CIFAR10 (Krizhevsky, 2009), and two real medical datasets: MRI scans for brain tumors, 2 and chest X-rays for pneumonia 3 .

Original: We present resultswith networks that achieve state-of-the-art accuracy on MNIST in App. A.1.7.
Revised: We provide additional results on MNIST in App. A.3.8.

Original: For reproducibility, we provide details on architectures and experimental setup  in App.
Revised: Details on the experimental setup are in App.

Original: A.7.1,A.7.2. The code is publicly available at https://anonymous.4open.science/r/FedDC-19E0 .
Revised: A.1.1,A.1.2, code is publicly available at https://github.com/kampmichael/FedDC .

Original: Synthetic Data: We ﬁrst investigate the potential of F ED DC on a synthetic binary classiﬁcation dataset generated by the sklearn (Pedregosa et al., 2011) make_classification function with 100 features.
Revised: Synthetic Data: We ﬁrst investigate the potential of F ED DC on a synthetic binary classiﬁcation dataset generated by the sklearn (Pedregosa et al., 2011) make_classification function with 100 features.

Original: On this dataset, we train a simple fully connected neural network with 3 hidden layers on m = 50 clients with n = 10 samples per client.
Revised: On this dataset, we train a simple fully connected neural network with 3 hidden layers on m = 50 clients with n = 10 samples per client.

Original: We compare F ED DC with daisy-chaining period d = 1 and aggregation period b = 200 to F ED A VG with the same amount of communication b
Revised: We compare F ED DC with daisy-chaining period d = 1 and aggregation period b = 200 to F ED A VG with the same amount of communication b

Original: =and the same averaging period b = 200 .
Revised: =and the same averaging period b = 200 .

Original: The results presented in Fig.
Revised: The results presented in Fig.

Original: 4 show that F ED DC achieves a test accuracy of 0 . 89 .
Revised: 4 show that F ED DC achieves a test accuracy of 0 . 89 .

Original: This is comparable to centralized training on all data which achieves a test accuracy of 0 . 88 .
Revised: This is comparable to centralized training on all data which achieves a test accuracy of 0 . 88 .

Original: It substantially outperforms both F ED A VG setups, which result in an accuracy of 0 .
Revised: It substantially outperforms both F ED A VG setups, which result in an accuracy of 0 .

Original: 80 and 0 .
Revised: 80 and 0 .

Original: Investigating the training of local models between aggreation periods reveals that the main issue of F ED A VG is overﬁtting of local clients, where F ED A VG train accuracy reaches 1 .quickly after each averaging step.
Revised: Investigating the training of local models between aggreation periods reveals that the main issue of F ED A VG is overﬁtting of local clients, where F ED A VG train accuracy reaches 1 .quickly after each averaging step.

Original: With these promising results on vanilla neural networks, we next turn to real-world image classiﬁcation problems typically solved with CNNs.
Revised: With these promising results on vanilla neural networks, we next turn to real-world image classiﬁcation problems typically solved with CNNs.

Original: CIFAR10: As a ﬁrst challenge for image classiﬁcation, we consider the well-known CIFAR image benchmark.
Revised: CIFAR10: As a ﬁrst challenge for image classiﬁcation, we consider the well-known CIFAR image benchmark.

Original: We ﬁrst investigate the effect of the aggregation period b on F ED DC and F ED A VG , separately optimizing for an optimal period for both methods.
Revised: We ﬁrst investigate the effect of the aggregation period b on F ED DC and F ED A VG , separately optimizing for an optimal period for both methods.

Original: We use a setting of 250 clients with a small version of ResNet, and 64 local samples each, which simulates our small sample setting, drawn at random without replacement (details in App. A.7.2).
Revised: We use a setting of 250 clients with a small version of ResNet, and 64 local samples each, which simulates our small sample setting, drawn at random without replacement (details in App. A.1.2).

Original: We report the results in Figure 5 and set the period for F ED DC to b = 10 , and consider federated averaging with periods of both b =(equivalent communication to F ED DC with d = 1 , b = 10 ) and b = 10 (less communication than
Revised: We report the results in Figure 5 and set the period for F ED DC to b = 10 , and consider federated averaging with periods of both b =(equivalent communication to F ED DC with d = 1 , b = 10 ) and b = 10 (less communication than

Original: F ED DC by a factor of 10 ) for all subsequent experiments.
Revised: F ED DC by a factor of 10 ) for all subsequent experiments.

Original: [Figure]
Revised: [Figure]

Original: Next, we consider a subset of 9600 samples spread across 150 clients (i.e. 64 samples per client), which corresponds to our small sample setting.
Revised: Next, we consider a subset of 9600 samples spread across 150 clients (i.e. 64 samples per client), which corresponds to our small sample setting.

Original: Now,however, each client is equipped with a larger, untrained ResNet18.
Revised: Now, each client is equipped with a larger, untrained ResNet18.

Original: 4 Note that the combined amount of examples is only one ﬁfth of the original training data, hence we cannot expectthe typical performance on this dataset.
Revised: 4 Note that the combined amount of examples is only one ﬁfth of the original training data, hence we cannot expect typical CIFAR10 performance.

Original: To investigate the extent of what is possible on this subset of CIFAR10, we run centralized learning CENTRAL , separately optimizing its hyperparameters, which sets the baseline at an accuracy of around 0 .
Revised: To obtain a gold standard for comparison, we run centralized learning CENTRAL , separately optimizing its hyperparameters, yielding an accuracy of around 0 .

Original: All results are reported in Table 1. We always report F ED A VG with b =and b = 10 , as these were the best performing settings, and b = 1 corresponds to equal amounts of communication as F ED DC.
Revised: All results are reported in Table 1, where we report F ED A VG with b = 1 and b = 10 , as these were the best performing settings and b = 1 corresponds to equal amounts of communication as F ED DC.

Original: We use a daisy chaining period of d = 1 for F ED DC throughout all experiments for consistency, and provide results for larger daisy chaining periods in App. A.1.5, noting that depending on the data distribution,larger periods might be favorable.
Revised: We use a daisy chaining period of d = 1 for F ED DC throughout all experiments for consistency, and provide results for larger daisy chaining periods in App. A.3.5, which, depending on the data distribution, might be favorable.

Original: We observe that F ED DC achieves substantially higher accuracyof more than 6 percentage points over the baseline set by federated averaging. 
Revised: We observe that F ED DC achieves substantially higher accuracy over the baseline set by federated averaging. In App.

Revised: A.3.7 we show that this holds also for client subsampling.

Original: Upon further inspection of the results, we see that F ED A VG drastically overﬁts, achieving training accuracies of 0 .
Revised: Upon further inspection, we see that F ED A VG drastically overﬁts, achieving training accuracies of 0 .

Original: 97 (see App. A.1.1), a similar trend as reported in Figurefor synthetic data.
Revised: (App. A.3.1), a similar trend as on the synthetic data before.

Original: We also see that daisy-chaining alone, besides its privacy issues,  performs worse than F ED DC.
Revised: Daisy-chaining alone, apart from privacy issues, also performs worse than F ED DC.

Original: Intriguingly, also the state of the art shows similarperformance trends.
Revised: Intriguingly, also the state of the art shows similar trends.

Original: F ED P ROX , run with optimal b = 10 and µ = 0 .
Revised: F ED P ROX , run with optimal b = 10 and µ = 0 .

Original: 1 , only achieves an accuracy of 0 .
Revised: 1 , only achieves an accuracy of 0 .

Original: The methods F ED A DAGRAD , F ED Y OGI , and F ED A DAM show even worse performance ofaccuracies around 0 .
Revised: 51 and F ED A DAGRAD , F ED Y OGI , and F ED A DAM show even worse performance of around 0 .

Original: 22 , 0 . 31 , and 0 . 34 , respectively.
Revised: 22 , 0 . 31 , and 0 . 34 , respectively.

Original: While applied successfully on large-scale data, these methods seem to have shortcomings when it comes to small sample regimes.
Revised: While applied successfully on large-scale data, these methods seem to have shortcomings when it comes to small sample regimes.

Original: To model different data distributions  that could occur in for example our healthcare setting, we ran further experiments simulating non-iid data, gradually increasing the locally available data, and non-privacy preserving decentralized learning.
Revised: To model different data distributions across clients that could occur in for example our healthcare setting, we ran further experiments on simulated non-iid data, gradually increasing the locally available data, as well as on non-privacy preserving decentralized learning.

Original: We investigate the effect of non-iid data on F ED DC by studying the “pathological non-IID partition of the data” (McMahan et al., 2017).
Revised: We investigate the effect of non-iid data on F ED DC by studying the “pathological non-IID partition of the data” (McMahan et al., 2017).

Original: Here, each client only sees examples from 2 out of the 10 classes of CIFAR10.
Revised: Here, each client only sees examples from 2 out of the 10 classes of CIFAR10.

Original: We again use a subset of the dataset.
Revised: We again use a subset of the dataset.

Original: The results in Tab. 2 show that F ED DC outperforms F ED A VG by a wide margin.
Revised: The results in Tab. 2 show that F ED DC outperforms F ED A VG by a wide margin.

Original: It also outperforms F ED P ROX , a method specialized on heterogeneous datasets in our considered small sample setting.
Revised: It also outperforms F ED P ROX , a method specialized on heterogeneous datasets in our considered small sample setting.

Original: For a similar training setup as before, we show results for gradually increasing local datasets in App.
Revised: For a similar training setup as before, we show results for gradually increasing local datasets in App.

Original: A.1.1.
Revised: A.3.4.

Original: Most notably, F ED DC outperforms F ED A VG even with 150 samples locally.
Revised: Most notably, F ED DC outperforms F ED A VG even withsamples locally.

Original: Only when the full CIFAR10 dataset is distributed across the clients, F ED A VG is on par with F ED DC (see App.
Revised: Only when the full CIFAR10 dataset is distributed across the clients, F ED A VG is on par with F ED DC (see App.

Original: Fig.
Revised: Fig.

Original: We also compare with distributed training  (App. A.1.5) without considering privacy, implemented by mini-batch SGD with parameter settings corresponding tothe same settings as our federated setup as well as a separately optimized version.
Revised: We also compare with distributed training through gradient sharing (App. A.3.3), which discards any privacy concerns, implemented by mini-batch SGD with parameter settings corresponding to our federated setup as well as a separately optimized version.

Original: The results show that such an approach is outperformed by both F ED A VG as well as F ED DC, which is in line with previous ﬁndings and emphasize the importance of model aggregation.
Revised: The results show that such an approach is outperformed by both F ED A VG as well as F ED DC, which is in line with previous ﬁndings and emphasize the importance of model aggregation.

Original: As a ﬁnal experiment on CIFAR10, we consider daisy-chaining with different combinations of aggregation methods, and hence its ability to serve as a building block that can be combined with other federated learning approaches.
Revised: As a ﬁnal experiment on CIFAR10, we consider daisy-chaining with different combinations of aggregation methods, and hence its ability to serve as a building block that can be combined with other federated learning approaches.

Original: In particular, we consider the same setting as before and combine
Revised: In particular, we consider the same setting as before and combine

Original: F ED P ROX with daisy chaining.
Revised: F ED P ROX with daisy chaining.

Original: The results, reported in Tab. 2, show that this combination is not only successful, but also outperforms all others in terms of accuracy.
Revised: The results, reported in Tab. 2, show that this combination is not only successful, but also outperforms all others in terms of accuracy.

Original: Medical image data: Finally, we consider two real medical image datasets representing actual health related machine learning tasks, which are naturally of small sample size.
Revised: Medical image data: Finally, we consider two real medical image datasets representing actual health related machine learning tasks, which are naturally of small sample size.

Original: For the brain MRI scans, we simulate 25 clients (e.g., hospitals) with 8 samples each.
Revised: For the brain MRI scans, we simulate 25 clients (e.g., hospitals) with 8 samples each.

Original: Each client is equipped with a CNN (see App. A.7.1).
Revised: Each client is equipped with a CNN [Table] [Table]

Original: The results for brain tumor prediction evaluated on a test set of 53 of these scans are reported in Table 1.
Revised: The results for brain tumor prediction evaluated on a test set of 53 of these scans are reported in Table 1.

Original: Overall, F ED DC performs best among the federated learning approaches and is close to the centralized model.
Revised: Overall, F ED DC performs best among the federated learning approaches and is close to the centralized model.

Original: Whereas F ED P ROX performed comparably poorly on CIFAR10, it now outperforms F ED A VG .
Revised: Whereas F ED P ROX performed comparably poorly on CIFAR10, it now outperforms F ED A VG .

Original: Also F ED Y OGI and F ED A DAM fare much better than on

Original: CIFAR10, although for all these methods there is still a considerable margin towards F ED DC.To [Table] [Table]
Revised: Similar to before, we observe a considerable margin between all competing methods and F ED DC.

Original:  investigate the effect of skewed distributions of sample sizes across clients, such as smaller hospitals having less data than larger ones, we provide additional experiments in App. A.1.5.
Revised: To investigate the effect of skewed distributions of sample sizes across clients, such as smaller hospitals having less data than larger ones, we provide additional experiments in App. A.3.5.

Original: The key insight is that also in these settings, F ED DC outperforms F ED A VG considerably, and is close to its performance on the unskewed datasets.
Revised: The key insight is that also in these settings, F ED DC outperforms F ED A VG considerably, and is close to its performance on the unskewed datasets.

Original: For the pneumonia dataset, we simulate 150 clients training ResNet18 (seeagain App.
Revised: For the pneumonia dataset, we simulate 150 clients training ResNet18 (see App.

Original: A.7.1) withsamples per client, the hold out test set are 624 images.
Revised: A.1.1) withsamples per client, the hold out test set are 624 images.

Original: The results, reported in Table 1, showagain similar trends as for the other datasets, with F ED DC outperforming all baselines and the state of the art, and being within the performance of the centrally trained model.
Revised: The results, reported in Table 1, show similar trends as for the other datasets, with F ED DC outperforming all baselines and the state of the art, and being within the performance of the centrally trained model.

Original: Moreover it highlights that F ED DC enables us to train a ResNet18 to high accuracy with as little as 8 samples per client.
Revised: Moreover it highlights that F ED DC enables us to train a ResNet18 to high accuracy with as little as 8 samples per client.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: We propose to combine daisy-chaining and aggregation to effectively learn high quality models in a federated setting where only little data is available locally.
Revised: We propose to combine daisy-chaining and aggregation to effectively learn high quality models in a federated setting where only little data is available locally.

Original: We formally prove convergence of our approach F ED DC, and for convex settings provide PAC-like generalization guarantees when aggregating by iterated Radon points.
Revised: We formally prove convergence of our approach F ED DC, and for convex settings provide PAC-like generalization guarantees when aggregating by iterated Radon points.

Original: Empirical results on the SUSY benchmark underline these theoretical guarantees, with F ED DC matching the performance of centralized learning.
Revised: Empirical results on the SUSY benchmark underline these theoretical guarantees, with F ED DC matching the performance of centralized learning.

Original: Extensive empirical evaluation shows that the proposed combination of daisy-chaining and aggregation enables federated learning from small datasets in practice.
Revised: Extensive empirical evaluation shows that the proposed combination of daisy-chaining and aggregation enables federated learning from small datasets in practice.

Original: When using averaging, we improve upon the state of the art for federated deep learning by a large margin for the considered small sample settings.
Revised: When using averaging, we improve upon the state of the art for federated deep learning by a large margin for the considered small sample settings.

Original: Last but not least, we show that daisy-chaining is not restricted to F ED DC, but can be straight-forwardly included in F ED A VG , Radon machines, and F ED P ROX as a building block, too.
Revised: Last but not least, we show that daisy-chaining is not restricted to F ED DC, but can be straight-forwardly included in F ED A VG , Radon machines, and F ED P ROX as a building block, too.

Original: F ED DCnaturally permits differential privacy mechanisms that introduce noise on model parameters, e.g., to safeguard against membership inference, poisoning and backdoor attacks.
Revised: F ED DC permits differential privacy mechanisms that introduce noise on model parameters, offering protection against membership inference, poisoning and backdoor attacks.

Original: As theserver assigns random permutations for daisy-chaining rounds, F ED DC is also robust against reconstruction attacks.
Revised: Through the random permutations in daisy-chaining rounds, F ED DC is also robust against reconstruction attacks.

Original: F ED DC uses additional daisy-chaining rounds, resulting in a linear increase in communication.
Revised: Through the daisy-chaining rounds, we see a linear increase in communication.

Original: As we are primarily interested in applications in healthcare, we consider the cross-SILO setting. Here, communication is not a bottleneck, so this slight increase in communication is negligible.
Revised: As we are primarily interested in healthcare applications, where communication is not a bottleneck, such an increase in communication is negligible.

Original: Furthermore, F ED DC outperforms F ED A VG in practice, even whenproviding both with the same amount of communication.
Revised: Importantly, F ED DC outperforms F ED A VG in practice also when both use the same amount of communication.

Original: Improving the communication efﬁciency considering settings where bandwidth is limited, e.g., model training on mobile devices, would make for engaging future work.
Revised: Improving the communication efﬁciency considering settings where bandwidth is limited, e.g., model training on mobile devices, would make for engaging future work.

Original: We conclude that daisy-chaining lends itself as a simple, yet effective building block to improve federated learning, complementing existing work to extend to settings where little data is available per client.
Revised: We conclude that daisy-chaining lends itself as a simple, yet effective building block to improve federated learning, complementing existing work to extend to settings where little data is available per client.

Original: F ED DC, thus, might offer a solution to the open problem of federated learning in healthcare, where very few, undisclosable samples are available at each site. 
Revised: F ED DC, thus, might offer a solution to the open problem of federated learning in healthcare, where very few, undisclosable samples are available at each site. A CKNOWLEDGMENTS

Revised: The authors thank Sebastian U. Stich for his detailed comments on an earlier draft.

Revised: This work received support from the Cancer Research Center Cologne Essen (CCCE).