Original: A N A GNOSTIC A PPROACH TO F EDERATED L EARNING WITH C LASS I MBALANCE
Revised: A N A GNOSTIC A PPROACH TO F EDERATED L EARNING WITH C LASS I MBALANCE

Original: Anonymous authors Paper under double-blind review
Revised: edu

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated Learning (FL) has emerged as the tool of choice for training deep models over heterogeneous and decentralized datasets.
Revised: Federated Learning (FL) has emerged as the tool of choice for training deep models over heterogeneous and decentralized datasets.

Original: As a reﬂection of the experiences from different clients, severe class imbalance issues are observed in realworld FL problems.
Revised: As a reﬂection of the experiences from different clients, severe class imbalance issues are observed in realworld FL problems.

Original: Moreover, there exists a drastic mismatch between the imbalances from the local and global perspectives, i.e. a local majority class can be the minority of the population.
Revised: Moreover, there exists a drastic mismatch between the imbalances from the local and global perspectives, i.e. a local majority class can be the minority of the population.

Original: Additionally, the privacy requirement of FL poses an extra challenge, as one should handle class imbalance without identifying the minority class.
Revised: Additionally, the privacy requirement of FL poses an extra challenge, as one should handle class imbalance without identifying the minority class.

Original: In this paper we propose a novel agnostic constrained learning formulation to tackle the class imbalance problem in FL without requiring further information beyond the standard FL objective.
Revised: In this paper we propose a novel agnostic constrained learning formulation to tackle the class imbalance problem in FL without requiring further information beyond the standard FL objective.

Original: A meta algorithm, CLIMB , is designed to solve the target optimization problem, with its convergence property analyzed under certain oracle assumptions.
Revised: A meta algorithm, CLIMB , is designed to solve the target optimization problem, with its convergence property analyzed under certain oracle assumptions.

Original: Through an extensive empirical study over various data heterogeneity and class imbalance conﬁgurations, we showcase that CLIMB considerably improves the performance in the minority class without compromising the overall accuracy of the classiﬁer, which signiﬁcantly outperforms previous arts.
Revised: Through an extensive empirical study over various data heterogeneity and class imbalance conﬁgurations, we showcase that CLIMB considerably improves the performance in the minority class without compromising the overall accuracy of the classiﬁer, which signiﬁcantly outperforms previous arts.

Original: In fact, we observe the greatest performance boost in the most difﬁcult scenario where every client only holds data from one class.
Revised: In fact, we observe the greatest performance boost in the most difﬁcult scenario where every client only holds data from one class.

Revised: The code can be found here.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Class imbalance is ubiquitous in real world supervised learning problems.
Revised: Class imbalance is ubiquitous in real world supervised learning problems.

Original: Examples span from medical applications (Lee & Shin, 2020; Roy et al., 2019; Choudhury et al., 2019), fraud detection (Yang et al., 2019;
Revised: Examples span from medical applications (Lee & Shin, 2020; Roy et al., 2019; Choudhury et al., 2019), fraud detection (Yang et al., 2019;

Original: Chan et al., 1999), to consumer based applications (Wang et al., 2021b; Wu et al., 2020; Long et al., 2020).
Revised: Chan et al., 1999), to consumer based applications (Wang et al., 2021b; Wu et al., 2020; Long et al., 2020).

Original: In these scenarios, data belonging to a subset of classes constitute a great proportion of the population while data from the minority classes, generated by uncommon events, are scarce (He & Garcia, 2009).
Revised: In these scenarios, data belonging to a subset of classes constitute a great proportion of the population while data from the minority classes, generated by uncommon events, are scarce (He & Garcia, 2009).

Original: Having a non-uniform number of samples per class deteriorates the performance of the classiﬁer in the minority class (Huang et al., 2016), resulting in low training and testing accuracy.
Revised: Having a non-uniform number of samples per class deteriorates the performance of the classiﬁer in the minority class (Huang et al., 2016), resulting in low training and testing accuracy.

Original: More importantly, unintended consequences of mistreating the minority class can be catastrophic (Van Hulse et al., 2007)
Revised: More importantly, unintended consequences of mistreating the minority class can be catastrophic (Van Hulse et al., 2007)

Original: if the problem is not handled appropriately.
Revised: if the problem is not handled appropriately.

Original: From the perspectives of data heterogeneity and privacy , the issue of class imbalance is even more signiﬁcant in the setting of Federated Learning (FL).
Revised: From the perspectives of data heterogeneity and privacy , the issue of class imbalance is even more signiﬁcant in the setting of Federated Learning (FL).

Original: Due to the heterogeneity of the local data distributions, there can be a signiﬁcant mismatch between the local and global imbalance, i.e. the class that is a minority locally can actually be a majority class globally.
Revised: Due to the heterogeneity of the local data distributions, there can be a signiﬁcant mismatch between the local and global imbalance, i.e. the class that is a minority locally can actually be a majority class globally.

Original: Moreover, for the purpose of privacy protection in FL, one should tackle the class-imbalance problem in an agnostic manner, i.e. the proposed algorithms should not require the minority class to be identiﬁed.
Revised: Moreover, for the purpose of privacy protection in FL, one should tackle the class-imbalance problem in an agnostic manner, i.e. the proposed algorithms should not require the minority class to be identiﬁed.

Original: In the centralized training setting, techniques like balanced sampling, loss re-weighting, and gradient tuning have achieved many successes (Johnson & Khoshgoftaar, 2019).
Revised: In the centralized training setting, techniques like balanced sampling, loss re-weighting, and gradient tuning have achieved many successes (Johnson & Khoshgoftaar, 2019).

Original: However, due to the extra difﬁculty of handling the class imbalance problem in FL, previous arts that rely on the explicit identiﬁcation of the minority class do not directly apply.
Revised: However, due to the extra difﬁculty of handling the class imbalance problem in FL, previous arts that rely on the explicit identiﬁcation of the minority class do not directly apply.

Original: While research along this line is quite limited, a commonly used heuristic is to estimate the portion of a class using the norm of the gradient per class.
Revised: While research along this line is quite limited, a commonly used heuristic is to estimate the portion of a class using the norm of the gradient per class.

Original: When used as proxies, these quantities are then utilized to reweight the losses corresponding to different classes (Wang et al., 2021a; Yang et al., 2020).
Revised: When used as proxies, these quantities are then utilized to reweight the losses corresponding to different classes (Wang et al., 2021a; Yang et al., 2020).

Original: A key drawback of (Wang et al., 2021a) is that a subset of the client’s local dataset needs to be shared, which is not ideal for privacy preserving purposes.
Revised: A key drawback of (Wang et al., 2021a) is that a subset of the client’s local dataset needs to be shared, which is not ideal for privacy preserving purposes.

Original: Moreover, the effectiveness of such approaches degrades when there is a notable mismatch between the local imbalance and the global imbalance (Wang et al., 2021a).
Revised: Moreover, the effectiveness of such approaches degrades when there is a notable mismatch between the local imbalance and the global imbalance (Wang et al., 2021a).

Original: In this work, we target the most difﬁcult yet possibly most interesting setting in FL, where there is a signiﬁcant mismatch between the local and global imbalance.
Revised: In this work, we target the most difﬁcult yet possibly most interesting setting in FL, where there is a signiﬁcant mismatch between the local and global imbalance.

Original: Concretely, on certain clients, a minority class from a global perspective is in the majority locally.
Revised: Concretely, on certain clients, a minority class from a global perspective is in the majority locally.

Original: Practitioners often encounters such a mismatch due to the highly heterogeneous data conﬁguration in FL and the principle of locality (Wang et al., 2021a).
Revised: Practitioners often encounters such a mismatch due to the highly heterogeneous data conﬁguration in FL and the principle of locality (Wang et al., 2021a).

Original: For example, when the local data are produced by a minority user, the corresponding minority data could occupy a large portion of the local distribution.
Revised: For example, when the local data are produced by a minority user, the corresponding minority data could occupy a large portion of the local distribution.

Original: In this scenario, due to the scarceness of the minority data from a global perspective, the corresponding underrepresented client will usually experience poor service quality from the trained model, giving rise to potentially severe fairness issue.
Revised: In this scenario, due to the scarceness of the minority data from a global perspective, the corresponding underrepresented client will usually experience poor service quality from the trained model, giving rise to potentially severe fairness issue.

Original: To overcome the aforementioned challenges, we propose a constrained learning formulation to handle the class imbalance issue in FL while accounting for both heterogeneity and privacy .
Revised: To overcome the aforementioned challenges, we propose a constrained learning formulation to handle the class imbalance issue in FL while accounting for both heterogeneity and privacy .

Original: In brief, we impose constraints on the standard FL formulation so that the empirical loss on every client should not overly exceed the average empirical loss.
Revised: In brief, we impose constraints on the standard FL formulation so that the empirical loss on every client should not overly exceed the average empirical loss.

Original: Such constraints are shown to force the classiﬁer to account for all classes equally and hence mitigate the detrimental effect of class imbalance, under a type of heterogeneous data conﬁguration that captures the mismatch between the local and global imbalance.
Revised: Such constraints are shown to force the classiﬁer to account for all classes equally and hence mitigate the detrimental effect of class imbalance, under a type of heterogeneous data conﬁguration that captures the mismatch between the local and global imbalance.

Original: The advantages of our formulation are threefold: First, in contrast to previous arts which usually rely on some heuristics to reweight the loss functions for different classses, our approach is principled, yielding a simple optimization interpretation.
Revised: The advantages of our formulation are threefold: First, in contrast to previous arts which usually rely on some heuristics to reweight the loss functions for different classses, our approach is principled, yielding a simple optimization interpretation.

Original: Second, unlike existing methods, our formulation requires no additional information compared to the original FL formulation and it treats data from different classes agnostically, and hence is less likely to leak the private information of the clients.
Revised: Second, unlike existing methods, our formulation requires no additional information compared to the original FL formulation and it treats data from different classes agnostically, and hence is less likely to leak the private information of the clients.

Original: Third, from our extensive empirical study, our formulation can signiﬁcantly improve the testing accuracy on the minority class, without compromising the overall performance.
Revised: Third, from our extensive empirical study, our formulation can signiﬁcantly improve the testing accuracy on the minority class, without compromising the overall performance.

Original: Contribution
Revised: Contribution.

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original: • Problem Formulation: To address the challenge of class imbalance in FL with data heterogeneity, we propose a novel constrained FL formulation with explicit enforcement on the similarity between the local empirical losses.
Revised: Problem Formulation: To address the challenge of class imbalance in FL with data heterogeneity, we propose a novel constrained FL formulation with explicit enforcement on the similarity between the local empirical losses.

Original: Using only information from the standard FL objective, our approach is completely agnostic to class distribution of the client data and its proxies, as opposed to the existing literature (which mostly violate privacy requirements of FL).
Revised: Using only information from the standard FL objective, our approach is completely agnostic to class distribution of the client data and its proxies, as opposed to the existing literature (which mostly violate privacy requirements of FL).

Original: • Meta-Algorithm: We solve our constrained optimization problem via a primal-dual approach.
Revised: Meta-Algorithm: We solve our constrained optimization problem via a primal-dual approach.

Original: For a ﬁxed dual variable, the corresponding Lagrangian function enjoys a similar structure as the standard FL loss, but with non-uniform weights on the local objectives.
Revised: For a ﬁxed dual variable, the corresponding Lagrangian function enjoys a similar structure as the standard FL loss, but with non-uniform weights on the local objectives.

Original: Accordingly, we propose an efﬁcient (meta-) algorithm called CLIMB that can use any FL optimization method as a subroutine, with nearly negligible communication overhead.
Revised: Accordingly, we propose an efﬁcient (meta-) algorithm called CLIMB that can use any FL optimization method as a subroutine, with nearly negligible communication overhead.

Original: Furthermore, we analyze the convergence properties of CLIMB under certain oracle assumptions.
Revised: Furthermore, we analyze the convergence properties of CLIMB under certain oracle assumptions.

Original: • Accuracy Improvement in the minority class: On benchmark datasets, CLIMB with Fed-Avg as base solver achieves signiﬁcant enhancement of the accuracy in the minority class without compromising the overall accuracy under various data heterogeneity and class imbalance scenarios.
Revised: Accuracy Improvement in the minority class: On benchmark datasets, CLIMB with Fed-Avg as base solver achieves signiﬁcant enhancement of the accuracy in the minority class without compromising the overall accuracy under various data heterogeneity and class imbalance scenarios.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Class Imbalance in Centralized Setting.
Revised: Class Imbalance in Centralized Setting.

Original: In the centralized learning setting, the number of samples per class is known.
Revised: In the centralized learning setting, the number of samples per class is known.

Original: We categorize existing solutions that exploit such information as follows.
Revised: We categorize existing solutions that exploit such information as follows.

Original: Balanced sampling.
Revised: Balanced sampling.

Original: In order to balance the data used for gradient calculation, the most common approaches are either to under sample from the majority classes (Liu et al., 2008), or to augment the minority class (Chawla et al., 2002; Guo & Viktor, 2004).
Revised: In order to balance the data used for gradient calculation, the most common approaches are either to under sample from the majority classes (Liu et al., 2008), or to augment the minority class (Chawla et al., 2002; Guo & Viktor, 2004).

Original: Both methods seek to generate an artiﬁcial uniform distribution of data (Buda et al., 2018; Pouyanfar et al., 2018).
Revised: Both methods seek to generate an artiﬁcial uniform distribution of data (Buda et al., 2018; Pouyanfar et al., 2018).

Original: Loss reweighting.
Revised: Loss reweighting.

Original: Methods in this category focus on re-weighting the loss functions for different classes, with extra emphases on the mistakes made in the minority class (Cui et al., 2019;
Revised: Methods in this category focus on re-weighting the loss functions for different classes, with extra emphases on the mistakes made in the minority class (Cui et al., 2019;

Original: Ling &
Revised: Ling &

Original: Sheng, 2008; Sun et al., 2007; Wang et al., 2016).
Revised: Sheng, 2008; Sun et al., 2007; Wang et al., 2016).

Original: There also exist works like (Lin et al., 2017) that adjust the scale of the loss sample-wise, without exploiting the global data distribution.
Revised: There also exist works like (Lin et al., 2017) that adjust the scale of the loss sample-wise, without exploiting the global data distribution.

Original: Gradient tuning.
Revised: Gradient tuning.

Original: In the context of neural networks, some other works focus on tuning the gradient per class (Anand et al., 1993) 
Revised: In the context of neural networks, some other works focus on tuning the gradient per class (Anand et al., 1993) showing faster rates of convergence in the class imbalance setting.

Original: To do this, the gradient is reshaped in order to have an equal magnitude when projected to all the gradients per class, while keeping the same norm as the original gradient.
Revised: To do this, the gradient is reshaped in order to have an equal magnitude when projected to all the gradients per class, while keeping the same norm as the original gradient.

Original: Class Imbalance in Federated Learning.
Revised: Class Imbalance in Federated Learning.

Original: Combating with the issue of class imbalance is more challenging in FL since the data composition in such a setting is generally unknown and the minority classes are often difﬁcult to identify Li et al.
Revised: Combating with the issue of class imbalance is more challenging in FL since the data composition in such a setting is generally unknown and the minority classes are often difﬁcult to identify Li et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020b).
Revised: (2020b).

Original: While research in this direction are quite limited, to the best of our knowledge, we classify them into the following two classes based on whether a proxy of the data composition is explicitly built.
Revised: While research in this direction are quite limited, to the best of our knowledge, we classify them into the following two classes based on whether a proxy of the data composition is explicitly built.

Original: (i)
Revised: (i)

Original: A common strategy in existing methods is to explicitly build proxies of the data composition based on some heuristics.
Revised: A common strategy in existing methods is to explicitly build proxies of the data composition based on some heuristics.

Original: These proxies are dynamically adjusted during the learning procedure and allow the aforementioned three techniques, balanced sampling, loss re-weighting and gradient tuning, to be utilized in the FL setting: Wang et al.
Revised: These proxies are dynamically adjusted during the learning procedure and allow the aforementioned three techniques, balanced sampling, loss re-weighting and gradient tuning, to be utilized in the FL setting: Wang et al.

Original: (2021a) suggests that there exists a proportional relation between the magnitude of the gradients the corresponds to the last layer of the neural network and the sample quantity.
Revised: (2021a) suggests that there exists a proportional relation between the magnitude of the gradients the corresponds to the last layer of the neural network and the sample quantity.

Original: Based on such observation, the proposed the Ratio-Loss , a class-wise re-weighted version of the standard cross entropy loss.
Revised: Based on such observation, the proposed the Ratio-Loss , a class-wise re-weighted version of the standard cross entropy loss.

Original: We note that to deﬁne the
Revised: We note that to deﬁne the

Original: Ratio-Loss , one needs to collect a subset of client data as “auxiliary data”, which may not be ideal in the setting of FL.
Revised: Ratio-Loss , one needs to collect a subset of client data as “auxiliary data”, which may not be ideal in the setting of FL.

Original: On the same vein, Yang et al.
Revised: On the same vein, Yang et al.

Original: (2020) argues that the gradient per class can be used as a proxy to infer the imbalances in the distributions of the clients, and thus clients should be selected according to how uniform the magnitude of gradient per class is.
Revised: (2020) argues that the gradient per class can be used as a proxy to infer the imbalances in the distributions of the clients, and thus clients should be selected according to how uniform the magnitude of gradient per class is.

Original: We also notice the work Astraea which introduces extra virtual components called mediators between the FL server and clients (Duan et al., 2019).
Revised: We also notice the work Astraea which introduces extra virtual components called mediators between the FL server and clients (Duan et al., 2019).

Original: The mediator is assumed to have access to the local data distributions of the clients so that client rescheduling and data augmentation can be carried out accordingly.
Revised: The mediator is assumed to have access to the local data distributions of the clients so that client rescheduling and data augmentation can be carried out accordingly.

Original: Through the prepossessing on the mediator, the gradients communicated to the server are made balanced class-wise.
Revised: Through the prepossessing on the mediator, the gradients communicated to the server are made balanced class-wise.

Original: However, when privacy is taken into consideration, methods like Astraea may not be appropriate as in essence it is directly built on the global data distribution.
Revised: However, when privacy is taken into consideration, methods like Astraea may not be appropriate as in essence it is directly built on the global data distribution.

Original: (ii) Without explicitly constructing an estimation of the data composition, there are works that resort to techniques like active learning and reinforcement learning to implicitly learn the data composition as the optimization progresses.
Revised: (ii) Without explicitly constructing an estimation of the data composition, there are works that resort to techniques like active learning and reinforcement learning to implicitly learn the data composition as the optimization progresses.

Original: Works along this research line usually rely on client selection to mitigate the effects of class imbalances.
Revised: Works along this research line usually rely on client selection to mitigate the effects of class imbalances.

Original: In (Goetz et al., 2019), to address the class imbalance problem, there will a higher probability to sampling clients that posses the global minority class.
Revised: In (Goetz et al., 2019), to address the class imbalance problem, there will a higher probability to sampling clients that posses the global minority class.

Original: Other works, leverage client selection as a multi-armed bandit problem, and design a client selection policy to balance the gradients (Xia et al., 2020).
Revised: Other works, leverage client selection as a multi-armed bandit problem, and design a client selection policy to balance the gradients (Xia et al., 2020).

Original: Other arts leverage Q-learning techniques to select clients in order to minimize the overall loss Wang et al.
Revised: Other arts leverage Q-learning techniques to select clients in order to minimize the overall loss Wang et al.

Original: (2020a).
Revised: (2020a).

Original: However, we believe in the most practical setting of FL, clients that are available per communication round is incoercible and hence these strategies have limited applicability.
Revised: However, we believe in the most practical setting of FL, clients that are available per communication round is incoercible and hence these strategies have limited applicability.

Original: Comparison with Previous Works.
Revised: Comparison with Previous Works.

Original: Our method signiﬁcantly differs from the above strategies in the following ways: (1) Our method requires no knowledge of the data composition, nor any proxy of such information.
Revised: Our method signiﬁcantly differs from the above strategies in the following ways: (1) Our method requires no knowledge of the data composition, nor any proxy of such information.

Original: Consequently, our approach is agnostic to the minority class and better preserves clients’ privacy.
Revised: Consequently, our approach is agnostic to the minority class and better preserves clients’ privacy.

Original: Such a property draws clear distinction between our work and the works listed in class (i) above.
Revised: Such a property draws clear distinction between our work and the works listed in class (i) above.

Original: (2) As will be more clear in the following section, we only perform clientwise re-weighting as opposed to the class-wise re-weighting schemes used in previous works, which further emphasize the agnostic nature of our approach.
Revised: (2) As will be more clear in the following section, we only perform clientwise re-weighting as opposed to the class-wise re-weighting schemes used in previous works, which further emphasize the agnostic nature of our approach.

Original: In contrast to the methods in class (ii) above, our approach does not require active client selection, enjoying a broader applicability in the most practical and interesting settings.
Revised: In contrast to the methods in class (ii) above, our approach does not require active client selection, enjoying a broader applicability in the most practical and interesting settings.

Original:  F EDERATED L EARNING WITH E MPIRICAL L OSS C ONSTRAINTS
Revised:  F EDERATED L EARNING WITH E MPIRICAL L OSS C ONSTRAINTS

Original: We consider the problem of multi-class classiﬁcation.
Revised: We consider the problem of multi-class classiﬁcation.

Original: Let x ∈ X ⊆
Revised: Let x ∈ X ⊆

Original: R
Revised: R

Original: d be the input and use y ∈ Y = { 1 , . . .
Revised: d be the input and use y ∈ Y = { 1 , . . .

Original: , C } to denote the target label, where C is the total number of classes.
Revised: , C } to denote the target label, where C is the total number of classes.

Original: In the context of FL, we assume to have N clients, each of which posses its own private data distribution p i ( x, y ) .
Revised: In the context of FL, we assume to have N clients, each of which posses its own private data distribution p i ( x, y ) .

Original: Here, p i ( x, y ) is a joint probability distribution of the input x and the output label y .
Revised: Here, p i ( x, y ) is a joint probability distribution of the input x and the output label y .

Original: One can decompose p i ( x, y ) as p i ( x, y )
Revised: One can decompose p i ( x, y ) as p i ( x, y )

Original: =
Revised: =

Original: p ( x | y ) p
Revised: p ( x | y ) p

Original: i ( y ) , where p ( x | y ) is the conditional distribution of the input x given class y and p i ( y ) is the marginal distribution of class y on client i .
Revised: i ( y ) , where p ( x | y ) is the conditional distribution of the input x given class y and p i ( y ) is the marginal distribution of class y on client i .

Original: We assume that the conditional distribution p ( x | y ) is identical on all devices, but the marginal distribution p i ( y ) can vary signiﬁcantly due to the heterogeneity of the data conﬁguration.
Revised: We assume that the conditional distribution p ( x | y ) is identical on all devices, but the marginal distribution p i ( y ) can vary signiﬁcantly due to the heterogeneity of the data conﬁguration.

Original: Let H =
Revised: Let H =

Original: { φ : X × Θ → R C } be a family of parameterized predictors with parameter θ ∈ Θ ⊆ R Q .
Revised: { φ : X × Θ → R C } be a family of parameterized predictors with parameter θ ∈ Θ ⊆ R Q .

Original: Let (cid:96) : R C × Y
Revised: Let (cid:96) : R C × Y

Original: → R + be the loss function, then the local objective function of client i is deﬁned as
Revised: → R + be the loss function, then the local objective function of client i is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For multi-class classiﬁcation, (cid:96) is often chosen to be the cross entropy loss.
Revised: For multi-class classiﬁcation, (cid:96) is often chosen to be the cross entropy loss.

Original: In the standard formulation of FL, the goal is to minimize the global average of local objectives
Revised: In the standard formulation of FL, the goal is to minimize the global average of local objectives

Original: [Equation]
Revised: [Equation]

Original: While it is well known that, in the presence of class imbalance, the above vanilla formulation will produce models that perform poorly on the minority data, we consider the following conﬁguration of heterogeneous local class distributions in order to make a quantitative analysis of such a phenomenon for a concrete class imbalance setting.
Revised: While it is well known that, in the presence of class imbalance, the above vanilla formulation will produce models that perform poorly on the minority data, we consider the following conﬁguration of heterogeneous local class distributions in order to make a quantitative analysis of such a phenomenon for a concrete class imbalance setting.

Original: More importantly, such a setting will also motivate our constrained FL formulation.
Revised: More importantly, such a setting will also motivate our constrained FL formulation.

Original: We emphasize that the following setting is just used as a motivating example to showcase our claims, and our results apply generally to any FL setting.
Revised: We emphasize that the following setting is just used as a motivating example to showcase our claims, and our results apply generally to any FL setting.

Original: Motivating Example.
Revised: Motivating Example.

Original: Let u be the uniform distribution over the classes, i.e. for y ∼ u , Pr( y = c ) = 1 C , ∀ c ∈ Y and let δ c be the Dirac distribution of class c .
Revised: Let u be the uniform distribution over the classes, i.e. for y ∼ u , Pr( y = c ) = 1 C , ∀ c ∈ Y and let δ c be the Dirac distribution of class c .

Original: We assume for some small but ﬁxed α ∈ [0 , 1] , the local class distribution of the client i is a mixture of the uniform distribution over all classes and the Dirac distribution of a ﬁxed class
Revised: We assume for some small but ﬁxed α ∈ [0 , 1] , the local class distribution of the client i is a mixture of the uniform distribution over all classes and the Dirac distribution of a ﬁxed class

Original: c i , i.e. p i
Revised: c i , i.e. p i

Original: = αu
Revised: = αu

Original: + (1 − α )
Revised: + (1 − α )

Original: δ c i .
Revised: δ c i .

Original: We use N c to denote the number clients with c i
Revised: We use N c to denote the number clients with c i

Original: = c .
Revised: = c .

Original: Note that in the limit setting when α = 0 , the aforementioned conﬁguration captures the most heterogeneous setting: clients only have data from a single class.
Revised: Note that in the limit setting when α = 0 , the aforementioned conﬁguration captures the most heterogeneous setting: clients only have data from a single class.

Original: We consider an extreme case of class imbalance under the above conﬁguration.
Revised: We consider an extreme case of class imbalance under the above conﬁguration.

Original: Without loss of generality, we consider the binary classiﬁcation problem, i.e. C = 2 , and we assume class 1 to be the minority class with N 1 = 1 .
Revised: Without loss of generality, we consider the binary classiﬁcation problem, i.e. C = 2 , and we assume class 1 to be the minority class with N 1 = 1 .

Original: We deﬁne g
Revised: We deﬁne g

Original: i ( θ ) :=
Revised: i ( θ ) :=

Original: E x ∼ p ( x | y = i )
Revised: E x ∼ p ( x | y = i )

Original: [ (cid:96) ( φ ( x, θ ) , i )] as the loss of the predictor φ ( · , θ ) on the data with y = i .
Revised: [ (cid:96) ( φ ( x, θ ) , i )] as the loss of the predictor φ ( · , θ ) on the data with y = i .

Original: We can calculate that
Revised: We can calculate that

Original: [Equation]
Revised: [Equation]

Original: Clearly, when α , the portion of data with uniform label distribution, is small, e.g. α = 0 , and N , the total number of clients, is large, the loss of the predictor on class 1 has negligible weight, which often leads to the poor performance on the minority class in the trained classiﬁer as the majority classes dominate the gradient.
Revised: Clearly, when α , the portion of data with uniform label distribution, is small, e.g. α = 0 , and N , the total number of clients, is large, the loss of the predictor on class 1 has negligible weight, which often leads to the poor performance on the minority class in the trained classiﬁer as the majority classes dominate the gradient.

Original: Moreover, observe that when α is small, there is a signiﬁcant mismatch between the local and global class imbalance: the global minority class 1 is in the majority on the corresponding client locally.
Revised: Moreover, observe that when α is small, there is a signiﬁcant mismatch between the local and global class imbalance: the global minority class 1 is in the majority on the corresponding client locally.

Original: Such phenomenon is pertinent to the FL setting due to the data heterogeneity and poses a great challenge for tackling the issue of class imbalance in FL.
Revised: Such phenomenon is pertinent to the FL setting due to the data heterogeneity and poses a great challenge for tackling the issue of class imbalance in FL.

Original: 3. C ONSTRAINED FL F ORMULATION
Revised: 3. C ONSTRAINED FL F ORMULATION

Original: In our work, to address the class imbalance challenge, we propose to minimize the following constrained FL (CFL) formulation
Revised: In our work, to address the class imbalance challenge, we propose to minimize the following constrained FL (CFL) formulation

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:15) is a tolerance constant that controls the enforced closeness in the training loss among clients, and we emphasize the dependence of the optimal value P ∗ (cid:15) on (cid:15) by encoding it in the subscript.
Revised: Here, (cid:15) is a tolerance constant that controls the enforced closeness in the training loss among clients, and we emphasize the dependence of the optimal value P ∗ (cid:15) on (cid:15) by encoding it in the subscript.

Original: As a motivation, we show that the constraint in our formulation (CFL) can be translated to a constraint on the performance of the minority class, under the above class imbalance setting: Consider the setting where the tolerance constant is very small, i.e. (cid:15) is close to zero.
Revised: As a motivation, we show that the constraint in our formulation (CFL) can be translated to a constraint on the performance of the minority class, under the above class imbalance setting: Consider the setting where the tolerance constant is very small, i.e. (cid:15) is close to zero.

Original: WLOG, we assume that the ﬁrst device is the one that has c i = 1 .
Revised: WLOG, we assume that the ﬁrst device is the one that has c i = 1 .

Original: Recall the deﬁnition of g i above Eq.(3).
Revised: Recall the deﬁnition of g i above Eq.(3).

Original: One can compute that
Revised: One can compute that

Original: [Equation]
Revised: [Equation]

Original: Therefore, our formulation (CFL) enjoys a clear class-balancing interpretation in the highly heterogeneous setting when α is small and N is large.
Revised: Therefore, our formulation (CFL) enjoys a clear class-balancing interpretation in the highly heterogeneous setting when α is small and N is large.

Original: Note that this is achieved without the identiﬁcation of the minority class and server collects no additional information compared to the vanilla FL.
Revised: Note that this is achieved without the identiﬁcation of the minority class and server collects no additional information compared to the vanilla FL.

Original: While the above nice interpretation of the proposed constraint may not hold exactly for general class imbalance settings, in spirit, we want the resulting classiﬁer from our algorithm to perform similarly on every class, or in other words to account for the minority class and majority class equally.
Revised: While the above nice interpretation of the proposed constraint may not hold exactly for general class imbalance settings, in spirit, we want the resulting classiﬁer from our algorithm to perform similarly on every class, or in other words to account for the minority class and majority class equally.

Original: In the following section, we discuss how to solve the proposed formulation by alternating the primal and dual updates of an equivalent Lagrangian formulation.
Revised: In the following section, we discuss how to solve the proposed formulation by alternating the primal and dual updates of an equivalent Lagrangian formulation.

Original: 3. A LGORITHM C ONSTRUCTION
Revised: A LGORITHM C ONSTRUCTION

Original: In order to solve problem (CFL), we resort to the method of Lagrange multipliers.
Revised: In order to solve problem (CFL), we resort to the method of Lagrange multipliers.

Original: By introducing the dual variables λ
Revised: By introducing the dual variables λ

Original: = [ λ 1 , . . .
Revised: = [ λ 1 , . . .

Original: λ N ] ∈ R N
Revised: λ N ] ∈ R N

Original: + , we deﬁne the Lagrangian function as,
Revised: + , we deﬁne the Lagrangian function as,

Original: [Equation]
Revised: [Equation]

Original: With this in hand, we can construct a lower bound of the above constrained optimization problem using the Lagrangian L ( θ, λ ) as follows:
Revised: With this in hand, we can construct a lower bound of the above constrained optimization problem using the Lagrangian L ( θ, λ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where we often refer to D ∗ (cid:15) as the dual problem.
Revised: where we often refer to D ∗ (cid:15) as the dual problem.

Original: To solve the dual problem, we propose CLIMB , a method described in Algorithm 1, which is discussed in detail as follows.
Revised: To solve the dual problem, we propose CLIMB , a method described in Algorithm 1, which is discussed in detail as follows.

Original: CLIMB proceeds by alternatingly optimizing over the primal variable θ and the dual variable λ .
Revised: CLIMB proceeds by alternatingly optimizing over the primal variable θ and the dual variable λ .

Original: Primal Update.
Revised: Primal Update.

Original: For a ﬁxed λ , the minimization of L with respect to θ is equivalent to a re-weighted version of the standard unconstrained FL objective (2):
Revised: For a ﬁxed λ , the minimization of L with respect to θ is equivalent to a re-weighted version of the standard unconstrained FL objective (2):

Original: [Equation]
Revised: [Equation]

Original: Importantly, this simple and canonical form allows us to perform the update on θ using any FL solver as the base optimizer.
Revised: Importantly, this simple and canonical form allows us to perform the update on θ using any FL solver as the base optimizer.

Original: For ﬂexibility, we do not explicitly choose the base optimizer in our algorithm description, but refer to the update on θ as the subroutine ClientUpdate ( { w i } Ni =1 , θ t )
Revised: For ﬂexibility, we do not explicitly choose the base optimizer in our algorithm description, but refer to the update on θ as the subroutine ClientUpdate ( { w i } Ni =1 , θ t )

Original: → θ t +1 .
Revised: → θ t +1 .

Original: Such a subroutine takes the non-uniform weights { w i } Ni =1 on the local objectives and the current consensus model θ t as inputs, and returns an updated model θ t +1 .
Revised: Such a subroutine takes the non-uniform weights { w i } Ni =1 on the local objectives and the current consensus model θ t as inputs, and returns an updated model θ t +1 .

Original: Note that every call to
Revised: Note that every call to

Original: ClientUpdate may consists of multiple communication rounds.
Revised: ClientUpdate may consists of multiple communication rounds.

Original: Dual Update.
Revised: Dual Update.

Original: Once we have obtained the new consensus model θ t +1 , we perform dual update on λ by taking a single dual ascent step of the following equivalent objective (given some ﬁxed θ ),
Revised: Once we have obtained the new consensus model θ t +1 , we perform dual update on λ by taking a single dual ascent step of the following equivalent objective (given some ﬁxed θ ),

Original: [Equation]
Revised: [Equation]

Original: To evaluate the gradient of L with respect to λ , we need to ﬁrst broadcast the consensus model θ t +1 and then aggregate the function values f ( θ t +1 ) .
Revised: To evaluate the gradient of L with respect to λ , we need to ﬁrst broadcast the consensus model θ t +1 and then aggregate the function values f ( θ t +1 ) .

Original: Since the broadcast model can be used for the next round of primal update, the only overhead of CLIMB compared to the standard FL solver
Revised: Since the broadcast model can be used for the next round of primal update, the only overhead of CLIMB compared to the standard FL solver

Original: ClientUpdate , is to transmit the functional value, which is negligible.
Revised: ClientUpdate , is to transmit the functional value, which is negligible.

Revised: Remark 3.1 We emphasize that CLIMB can be implemented in a privacy-preserving manner:

Revised: A client can carry out its update locally given the access to the global average of the dual variables ¯ λ and the global average of the loss functions ¯ f ( θ ) .

Revised: These quantities can be computed via the standard

Revised: FL technique of Homomorphic Encryption without revealing the exact value of the dual variable λ i and local loss f i ( θ ) to the server, as elaborated in Appendix D.

Original: 3. T HEORETICAL G UARANTEES
Revised: 3. T HEORETICAL G UARANTEES

Original: Based on the recent progress in constrained learning (Chamon et al., 2021), we show that underthe following mild regularity conditions, the duality gap between the dual problem D ∗ (cid:15) and the primal problem P ∗ (cid:15) can be controlled by some quantity that describes the capability of the parametric function class H .
Revised: Based on the recent progress in constrained learning (Chamon et al., 2021), we show that under mild regularity conditions, the duality gap between the dual problem D ∗ (cid:15) and the primal problem P ∗ (cid:15) can be controlled by some quantity that describes the capability of the parametric function class H .

Original: Assumption 3.1 The loss function (cid:96) in the deﬁnition of the local objective (1) is L -Lipschitz, i.e. (cid:107) (cid:96) ( x, · ) − (cid:96) ( z, · ) (cid:107) ≤ L (cid:107) x − z (cid:107) , and bounded by B .
Revised: Assumption 3.1 The loss function (cid:96) in the deﬁnition of the local objective (1) is L -Lipschitz, i.e. (cid:107) (cid:96) ( x, · ) − (cid:96) ( z, · ) (cid:107) ≤ L (cid:107) x − z (cid:107) , and bounded by B .

Original: Assumption 3.2 The conditional distribution p ( x | y ) is non-atomic for all y ∈ R C .
Revised: Assumption 3.2 The conditional distribution p ( x | y ) is non-atomic for all y ∈ R C .

Original: While usually the local data distribution is discrete, we can always augment it by randomly perturbing the data points with white noise, this is often used as data augmentation in vision tasks.
Revised: While usually the local data distribution is discrete, we can always augment it by randomly perturbing the data points with white noise, this is often used as data augmentation in vision tasks.

Original: Assumption 3.3 There exists a convex hypothesis class ˆ H such that H ⊆
Revised: Assumption 3.3 There exists a convex hypothesis class ˆ H such that H ⊆

Original: ˆ H , and there exists a constant ξ > 0 such that ∀ ˆ φ ∈
Revised: ˆ H , and there exists a constant ξ > 0 such that ∀ ˆ φ ∈

Original: ˆ H , there exists θ ∈ Θ such that sup x ∈X (cid:107) ˆ φ ( x ) − φ ( x, θ ) (cid:107) ≤ ξ .
Revised: ˆ H , there exists θ ∈ Θ such that sup x ∈X (cid:107) ˆ φ ( x ) − φ ( x, θ ) (cid:107) ≤ ξ .

Original: A simple strategy to construct ˆ H is to take the convex hull of H .
Revised: A simple strategy to construct ˆ H is to take the convex hull of H .

Original: When H is sufﬁciently rich, ξ can be expected to be small.
Revised: When H is sufﬁciently rich, ξ can be expected to be small.

Original: Notice that this bound can be decreased by increasing the richness of the function class H .
Revised: Notice that this bound can be decreased by increasing the richness of the function class H .

Original: All the proofs can be found in the Appendix.
Revised: All the proofs can be found in the Appendix.

Original: Theorem 3.1 (Near Zero Duality Gap) Under Assumptions 3.1, 3.2,3.3, and the Contrained Federated Learning problem is feasible in ˆ H with constraint (cid:15) − 2 Lξ , the Constrained Federated Learning problem has near zero-duality gap,
Revised: Theorem 3.1 (Near Zero Duality Gap) Under Assumptions 3.1, 3.2,3.3, and the Contrained Federated Learning problem is feasible in ˆ H with constraint (cid:15) − 2 Lξ , the Constrained Federated Learning problem has near zero-duality gap,

Original: [Equation]
Revised: [Equation]

Original: , (8) where λ ∗ (cid:15) − 2 Lξ is the optimal dual variable associated with the Constrained Federated Learning problem equation CFL with constraints (cid:15) − 2 Lξ over the space of functions ˆ H .
Revised: ) where λ ∗ (cid:15) − 2 Lξ is the optimal dual variable associated with the Constrained Federated Learning problem (CFL) with constraints (cid:15) − 2 Lξ over the space of functions ˆ H .

Original: Theorem 3.1 establishes an upper bound on the duality gap of the Constrained Federated Learning
Revised: Theorem 3.1 establishes an upper bound on the duality gap of the Constrained Federated Learning

Original: Problem CFL.
Revised: Problem CFL.

Original: The gap depends on the Lipschitz constant of the loss function L , the richness of the function class ξ , and the optimal dual variable of a more restrictive problem.
Revised: The gap depends on the Lipschitz constant of the loss function L , the richness of the function class ξ , and the optimal dual variable of a more restrictive problem.

Original: Note that we required the Constrained Federated Learning problem to be feasible for constraint (cid:15) − 2 Lξ .
Revised: Note that we required the Constrained Federated Learning problem to be feasible for constraint (cid:15) − 2 Lξ .

Original: In the case of the cross-entropy loss, as long as (cid:15) − 2 Lξ > 0 , this can be satisﬁed by a classiﬁer that assigns a uniform label for every sample, as each individual loss will be equal to each other.
Revised: In the case of the cross-entropy loss, as long as (cid:15) − 2 Lξ > 0 , this can be satisﬁed by a classiﬁer that assigns a uniform label for every sample, as each individual loss will be equal to each other.

Original: Since the minimization of the Lagrangian function L is non-convex, to show the convergence of
Revised: Since the minimization of the Lagrangian function L is non-convex, to show the convergence of

Original: CLIMB , we need an additional oracle assumption as follows.
Revised: CLIMB , we need an additional oracle assumption as follows.

Original: Assumption 3.4 (Approximate Solution)
Revised: Assumption 3.4 (Approximate Solution)

Original: For every dual variable λ ∈ R N + , and precision δ >there exists an oracle approximate solution θ λ such that L ( θ λ , λ ) ≤ min θ ∈ R Q L ( θ, λ ) + δ .
Revised: For every dual variable λ ∈ R N + , and precision δ >there exists an oracle approximate solution θ λ such that L ( θ λ , λ ) ≤ min θ ∈ R Q L ( θ, λ ) + δ .

Original: Theorem 3.2 (Convergence) Deﬁne the dual function d ( λ )
Revised: Theorem 3.2 (Convergence) Deﬁne the dual function d ( λ )

Original: =
Revised: =

Original: min θ ∈ Θ L ( θ, λ ) .
Revised: min θ ∈ Θ L ( θ, λ ) .

Original: Under Assumptions 3.1 to 3.4, for a ﬁxed tolerance r > 0 , the iterates generated by Algorithm 1, achieve a neighborhood of the dual problem D ∗ (cid:15) in at most T r 
Revised: Under Assumptions 3.1 to 3.4, for a ﬁxed tolerance r > 0 , the iterates generated by Algorithm 1 converge to a neighborhood of the dual problem D ∗ (cid:15) in at most T r =

Original: = O (1 /r ) steps, i.e.,
Revised: O (1 /r ) steps, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Due to space limitation, a detailed discussion of the convergence theorem is deferred to the appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate our formulation (CFL) against the competitors on various FL scenarios at the presence of class imbalance.
Revised: In this section, we evaluate our formulation (CFL) against the competitors on various FL scenarios at the presence of class imbalance.

Original: Our results highlight the beneﬁts of our approach especially when the local data distributions are severely heterogeneous and there is a signiﬁcant mismatch between local and global imbalance.
Revised: Our results highlight the beneﬁts of our approach especially when the local data distributions are severely heterogeneous and there is a signiﬁcant mismatch between local and global imbalance.

Original: To ensure a fair comparison, we adopt the same base optimizer, Fed-Avg [Table] base FL solver is Fed-Avg with partial-participation : 100 devices participate in every communication round. in the current experiment1 , for all formulations: the standard FL objective in Eq.(2), the proposed formulation in Eq.(CFL), Ratio-Loss (Wang et al., 2021a), and Focal-Loss (Lin et al., 2017).
Revised: To ensure a fair comparison, we use Fed-Avg as the base optimizer in the current experiment for all formulations : the standard FL objective in Eq.(2), the proposed formulation in Eq.(CFL), Ratio-Loss (Wang et al., 2021a), and Focal-Loss (Lin et al., 2017).

Original: We emphasize that the novelty of our work lies in the new constrained FL formulation (CFL) and is orthogonal tothe research of how the formulation is solved.
Revised: We emphasize that the novelty of our work lies in the new constrained FL formulation (CFL) and is orthogonal to how the formulation is solved.

Original: In the following, we describe the datasets and models used in our experiments where more detailsare provided inthe Appendix A.
Revised: We now describe the datasets and models used in our experiments with more details provided in Appendix A.

Original: Datasets Two benchmark datasets are used in our experiments with the default train/test splits,  which are MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky et al., 2009).
Revised: Datasets Three benchmark datasets are used in our experiments with the default train/test splits, [Table] -Avg with partial-participation : 100 devices participate in every communication round. which are MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky et al., 2009) and Fashion-MNIST (Xiao et al., 2017).

Revised: The results on the last dataset are deferred to the appendix due to space limitation.

Original: Heterogeneity.
Revised: Heterogeneity.

Original: We generate heterogeneity in the local data distributions according to the strategy from (Karimireddy et al., 2020;
Revised: We generate heterogeneity in the local data distributions according to the strategy from (Karimireddy et al., 2020;

Original: Hsu et al., 2019):
Revised: Hsu et al., 2019):

Original: Let α ∈ [0 , 1] be some constant that determines the level of heterogeneity.
Revised: Let α ∈ [0 , 1] be some constant that determines the level of heterogeneity.

Original: For a ﬁxed α , we divide the dataset among N = 100 (moderate) or
Revised: For a ﬁxed α , we divide the dataset among N = 100 (moderate) or

Original: N = 500 (massive) clients as follows: for we allocate to each client a portion of α i.i.d.
Revised: N = 500 (massive) clients as follows: for we allocate to each client a portion of α i.i.d.

Original: data and the remaining portion of (1 − α ) by sorting according to label.
Revised: data and the remaining portion of (1 − α ) by sorting according to label.

Revised: In our appendix, we also consider the Dirichlet type heterogeneous data allocation scheme which is wildly used in the literature of

Revised: Federated Learning, for example (Hsu et al., 2019; Acar et al., 2020).

Original: Data Imbalance.
Revised: Data Imbalance.

Original: We simulate the phenomenon of class imbalance by removing data belong to the minority classes: Observe that the datasets included in our experiments both have 10 perfectly balanced classes.
Revised: We simulate the phenomenon of class imbalance by removing data belong to the minority classes: Observe that the datasets included in our experiments both have 10 perfectly balanced classes.

Original: For the minority class(es), we retain only 1 /ρ portion of the corresponding data.
Revised: For the minority class(es), we retain only 1 /ρ portion of the corresponding data.

Original: Here, ρ ≥ 1 the ratio between the numbers of data in the majority class and in the minority class and is termed the imbalance ratio .
Revised: Here, ρ ≥ 1 the ratio between the numbers of data in the majority class and in the minority class and is termed the imbalance ratio .

Original: For example, when there are 3 minority classes with ρ = 10 , 90% of the data belong to classes 0, 1, 2 (without loss of generality) are manually removed.
Revised: For example, when there are 3 minority classes with ρ = 10 , 90% of the data belong to classes 0, 1, 2 (without loss of generality) are manually removed.

Original: In our experiments, we consider the setting of 1 or 3 minority classes and we take ρ = 5 , 10 , 20 .
Revised: In our experiments, we consider the setting of 1 or 3 minority classes and we take ρ = 5 , 10 , 20 .

Original: Models We follow the choice of model architectures in (Acar et al., 2020; McMahan et al., 2017).
Revised: Models We follow the choice of model architectures in (Acar et al., 2020; McMahan et al., 2017).

Original: Speciﬁcally, we use a 2 hidden layer fully-connected neural network for MNIST, where the numbers of neurons are (128, 128).
Revised: Speciﬁcally, we use a 2 hidden layer fully-connected neural network for MNIST, where the numbers of neurons are (128, 128).

Original: For CIFAR10, we use a CNN model consisting of 2 convolutional layers with 64 5 × 5 ﬁlters followed by 2 fully connected layers with 394 and 192 neurons.
Revised: For CIFAR10, we use a CNN model consisting of 2 convolutional layers with 64 5 × 5 ﬁlters followed by 2 fully connected layers with 394 and 192 neurons.

Original: We note that higher testing accuracy on the included datasets can be obtain by using models with high capacity, but is orthogonal to our research.
Revised: We note that higher testing accuracy on the included datasets can be obtain by using models with high capacity, but is orthogonal to our research.

Original: 4. R ESULTS S UMMARY
Revised: 4. R ESULTS S UMMARY

Original: To evaluate the effectiveness of an approach against the challenge of class imbalance, one needs to take into consideration both the performance on the minority class(es) and the average performance on all the classes.
Revised: To evaluate the effectiveness of an approach against the challenge of class imbalance, one needs to take into consideration both the performance on the minority class(es) and the average performance on all the classes.

Original: We report both quantities after sufﬁcient communication rounds (5000 rounds for CIFAR10 and 1000 rounds for MNIST) under various experiment settings in Tables 1 and 2.
Revised: We report both quantities after sufﬁcient communication rounds (5000 rounds for CIFAR10 and 1000 rounds for MNIST) under various experiment settings in Tables 1 and 2.

Original: In every cell of these tables, the quantity above denote the minority class testing accuracy and the quantity in the parentheses is the average performance on all the classes.
Revised: In every cell of these tables, the quantity above denote the minority class testing accuracy and the quantity in the parentheses is the average performance on all the classes.

Original: We also considered the case where there are multiple minority classes and we report the worst accuracy among the minority classes.
Revised: We also considered the case where there are multiple minority classes and we report the worst accuracy among the minority classes.

Original: Our approach outperforms previous arts in all cases, often by a large margin.
Revised: Our approach outperforms previous arts in all cases, often by a large margin.

Original: Imbalance Ratio and Number of Minority Classes.
Revised: Imbalance Ratio and Number of Minority Classes.

Original: The imbalance ratio ρ and the number of minority classes are two important quantities to measure the difﬁculty of a class imbalance problem.
Revised: The imbalance ratio ρ and the number of minority classes are two important quantities to measure the difﬁculty of a class imbalance problem.

Original: Under all the included choices, CLIMB consistently beats previous arts in both minority class testing accuracy and average performance on all the classes.
Revised: Under all the included choices, CLIMB consistently beats previous arts in both minority class testing accuracy and average performance on all the classes.

Original: Therefore, we conclude that CLIMB is able to boost the performance on the minority class without compromising the performance on the other classes.
Revised: Therefore, we conclude that CLIMB is able to boost the performance on the minority class without compromising the performance on the other classes.

Original: This is a rare merit when addressing the class imbalance problem.
Revised: This is a rare merit when addressing the class imbalance problem.

Original: In fact, methods like
Revised: In fact, methods like

Original: Ratio-Loss is able to improve the testing accuracy on the minority class, but it also sacriﬁces the performance on the rest classes, leading to an inferior average testing accuracy.
Revised: Ratio-Loss is able to improve the testing accuracy on the minority class, but it also sacriﬁces the performance on the rest classes, leading to an inferior average testing accuracy.

Original: Level of Heterogeneity.
Revised: Level of Heterogeneity.

Original: We test the performance of CLIMB under different levels of heterogeneity and observe that CLIMB outperforms the included methods considerably in all settings.
Revised: We test the performance of CLIMB under different levels of heterogeneity and observe that CLIMB outperforms the included methods considerably in all settings.

Original: It has the biggest advantage over the competitors in the most heterogeneous setting, α = 0 .
Revised: It has the biggest advantage over the competitors in the most heterogeneous setting, α = 0 .

Original: There are situations that existing methods completely fail in the minority class with less than 10% accuracy, but CLIMB is still able to correctly classify most of the minority data, e.g. see Table 2 → ρ = 5 →
Revised: There are situations that existing methods completely fail in the minority class with less than 10% accuracy, but CLIMB is still able to correctly classify most of the minority data, e.g. see Table 2 → ρ = 5 →

Original: MNIST → 3 minority classes → α = 0 .
Revised: MNIST → 3 minority classes → α = 0 .

Original: Moderate vs. Large Number of Devices An important goal of FL is to exploit the computational resources of the IoT devices.
Revised: Moderate vs. Large Number of Devices An important goal of FL is to exploit the computational resources of the IoT devices.

Original: Hence, the scalability to a large number of devices is a critical property of an FL method.
Revised: Hence, the scalability to a large number of devices is a critical property of an FL method.

Original: We hence test CLIMB on both moderate ( N = 100 , see Table 2) and massive devices ( N = 500 , see Table 1) settings.
Revised: We hence test CLIMB on both moderate ( N = 100 , see Table 2) and massive devices ( N = 500 , see Table 1) settings.

Original: To make things more practical, we instantiate the subroutine ClientUpdate using Fed-Avg with the partial-participation scheme in the massive device setting.
Revised: To make things more practical, we instantiate the subroutine ClientUpdate using Fed-Avg with the partial-participation scheme in the massive device setting.

Original: Speciﬁcally, 100 devices participate model training after every Fed-Avg global communication round.
Revised: Speciﬁcally, 100 devices participate model training after every Fed-Avg global communication round.

Original: We clearly observe the advantage of CLIMB in all of these setups.
Revised: We clearly observe the advantage of CLIMB in all of these setups.

Original: C ONCLUSION
Revised: C ONCLUSION

Original: In this paper we proposed a novel agnostic constrained learning formulation to tackle the problem of class imbalance in the Federated Learning setting.
Revised: In this paper we proposed a novel agnostic constrained learning formulation to tackle the problem of class imbalance in the Federated Learning setting.

Original: By introducing constraints in the learning procedure we enforced the performance to be similar in all clients, thus accounting for the class imbalances.
Revised: By introducing constraints in the learning procedure we enforced the performance to be similar in all clients, thus accounting for the class imbalances.

Original: In terms of privacy protection, our formulation requires no further information than the standard FL objective to be collected in the server and it never estimates the data composition as opposed to all previous approaches.
Revised: In terms of privacy protection, our formulation requires no further information than the standard FL objective to be collected in the server and it never estimates the data composition as opposed to all previous approaches.

Original: Moreover, compared with previous arts which are usually heuristic based, our approach is principled as it is purely optimization based and can be efﬁciently solved via the proposed meta-algorithm CLIMB , yielding major practical beneﬁts.
Revised: Moreover, compared with previous arts which are usually heuristic based, our approach is principled as it is purely optimization based and can be efﬁciently solved via the proposed meta-algorithm CLIMB , yielding major practical beneﬁts.

Original: Our extensive empirical study showcases the superiority of proposed constrained formulation over previous arts.
Revised: Our extensive empirical study showcases the superiority of proposed constrained formulation over previous arts.

Revised: A CKNOWLEDGMENTS

Revised: This research is supported by AFOSR Award 19RT0726, NSF HDR TRIPODS award 1934876,

Revised: NSF award CPS-1837253, NSF award CIF-1910056, NSF CAREER award CIF-1943064, and NSF award CCF-2112665.

Original: [Table]
Revised: [Table]