Paper: FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data (ICLR-2024)

Reviewer comment (Presentation):
The content of the paper is not very easy to follow. For example, the message from Figure 1 is a bit hard to interpret. In the first paragraph of the third page (where Figure 1 (a) is explained), it is not quite clear what large mean and small mean imply.

Author response:
Q1. What does Figure 1 signify, and what are the implications of large and small means?

Thanks for the question and we are glad to clarify what Fig. 1 signifies: Compared to Fig. 1 (a), **two significant changes** are evident in Fig. 1 (b) after implementing FedLoGe: **1. Noisy features are masked.** The small mean features with bigger variances are located in the sparsely allocated area, marked by grey vertical shadows. 2. **The quality of dominant features is further enhanced**. This is evidenced by the decreased variance in features with larger means, indicating they have become more concise and effective. 

Implications of large and small means: Features with both **large and small means** are crucial for a network. However, training directly with a fixed ETF classifier can lead to poor quality of small mean features. This is reflected in the relative variance of smaller mean features within the same class being much higher compared to large mean features.

Below is a detailed interpretation of Fig. 1:

Neural collapse reveals a principle, namely feature collapse, which shows that in a properly trained network, features should effectively collapse into the class mean (**where all relative variances in Fig. 1 (a) are expected to be zero**). Yet, when trained with a vallina fixed ETF classifier, **features, particularly those with smaller means, do not collapse effectively, namely feature degeneration. FedLoGe is designed to mitigate feature degeneration by squeezing poor-quality and noisy features into these sparsely allocated positions.** Overall, as shown in Fig. 1, after training with $\psi_{SSE-C}$, FedLoGe inherently filters out noisy smaller mean features, making features more concise (Appendix A.4) and effective (Table 3 in the main paper), in which the variance of dominant features becomes smaller. 

We give the detailed computation process for the data in Fig. 1:
Given that $n_c$ is the sample number of class $c$.  $\mu_c$ is the class mean of class $c$, and $h_{i,c}$ is the feature of class $c$ on sample $i$, the features $h_{i,c}$ will collapse to the within-class mean $\mu_c = \frac{1}{n_c} \sum_{i=1}^{n_c}h_{i,c}$. This indicates that the covariance $\frac{1}{n_c} \sum_{i=1}^{n_c}(h_{i,c}-\mu_c)(h_{i,c}-\mu_c)^T$ will converge to $0$. We investigate whether $h_{i,c} - \mu_c$ tends toward zero when training with a fixed ETF classifier, as shown in Fig. 1 (a). 

The steps to obtain the data in Fig. 1 are：

1. Select class $c$.
2. During global test, compute all sample features $h_{i,c}$, with dimension $D$, representing the number of features. Use $h_{i,c}[d]$ to denote the $d^{th}$ feature in $h_{i,c}$.

 3.  Compute the mean and relative variance of $h_{c}$, sorting them in descending order of the mean, and accordingly adjust the variance positions.

It becomes evident that not all features, $h_{i,c}[d] - \mu_c[d]$, converge to zero, and smaller $\mu_c[d]$ values exhibit greater variance.

The process illustrated in **Fig.1 (b)** follows the same steps as **Fig.1 (a)**, but with grey vertical shadows for the sparsely positioned features. It can be observed that sparse ETF training effectively squeezes poor-quality and noisy features into these sparsely allocated positions.

**All the calculation details have been updated in the revised appendix A.2 marked in red.**

Reviewer comment (Writing):
Notations seem to be inconsistent, the notation for the number of clients is K in Section 3, but N in Section 4.

Author response:
Q2. Notations and typos?

Thank you for the detailed feedback! We have corrected all the inconsistent usage of K and N and revised “…. personalized classifier by multiple the norm of … “ to “by multiplying… “ in Section 3.3. and have uploaded the revised version and **marked the modifications in red color.**

Reviewer comment (Writing):
Probably a typo: “…. personalized classifier by multiple the norm of … “ Should it be “by multiplying… “?

Author response:
Q2. Notations and typos?

Thank you for the detailed feedback! We have corrected all the inconsistent usage of K and N and revised “…. personalized classifier by multiple the norm of … “ to “by multiplying… “ in Section 3.3. and have uploaded the revised version and **marked the modifications in red color.**

Reviewer comment (Novelty):
The method of fixing the classifier and backbone alternatively for training is proposed in previous papers on personalized FL.

Author response:
Q3. The method of fixing the classifier and backbone alternatively for training is proposed in previous papers on personalized FL.

We need to clarify that the alternative training framework is a well-established training paradigm, inspired by decoupling learning [4]. **However, the main contribution of FedLoGe lies in being the first to achieve joint global and local training in Federated long-tailed learning, approached from the perspective of neural collapse**, utilizing the novel methods, $\psi_{SSE-C}$ and GLA-FR.

We detail the differences between FedLoGe and traditional alternative training methods in personalized FL: Traditional alternative training for personalized FL **ignores representation learning for backbone**, making it **challenging to obtain global model and personalized models simultaneously**, as a strong backbone is foundational for both. Moreover, other personalized FL methods are **not specifically tailored for long-tail and heterogeneous data**.

For example, FedRep [1] alternatively trains the classifier and backbone within a single local round. However, it does not specifically address imbalanced and long-tailed learning.

Reviewer comment (Reproducibility):
Some settings are not clear: e.g., the number of clients participating in training each round and the number of local epochs.

Author response:
Q4. Some settings are not clear: e.g., the number of clients participating in training each round and the number of local epochs. How many clients participate in training each round? Does the paper focus on cross-silo FL, cross-device FL, or both?

Thank you for detailed suggestions that really help in refining our work. FedLoGe covers diverse system scale in different dataset. For CIFAR10/100-LT datasets, we engaged 40 clients with full participation in each training round, setting the number of local epochs to 5. For ImageNet/iNat datasets, the experiments involved 20 clients, with a 40% participation rate in each training round, and the number of local epochs was set to 3.  **We have updated Section A.1, Detailed Setup, and marked the modifications in red for clarity.**

To assess FedLoGe's resilience and effectiveness in diverse settings,  we conducted experiments on CIFAR100-LT with imbalance factor of 100 and heterogeneity coefficient of 0.5, while lowering the participation rate to 30%. The results, documented in the table below, shows that our approach remains superior, particularly in terms of personalized performance. 

To further explore the effectiveness of FedLoGe in a cross-device FL setting, we increased the number of clients to 100. The results demonstrate that FedLoGe significantly outperforms other algorithms.

| Method | 40 clients(100% participate) | 40(30%) | 100(30%) |
| --- | --- | --- | --- |
| FedAvg | GM0.3818/PM0.6214 | 0.3746/0.6205 | 0.3718/0.6384 |
| FedRoD | 0.3919/0.6919 | 0.3758/0.7156 | 0.3727/0.7665 |
| FedETF | 0.3825/0.6421 | 0.4180/0.7083 | 0.3847/0.7822 |
| FedLoGe | 0.4233/0.7285 | 0.4206/0.7524 | 0.4159/0.8008 |

Reviewer comment (Reproducibility):
How many clients participate in training each round?

Author response:
Q4. Some settings are not clear: e.g., the number of clients participating in training each round and the number of local epochs. How many clients participate in training each round? Does the paper focus on cross-silo FL, cross-device FL, or both?

Thank you for detailed suggestions that really help in refining our work. FedLoGe covers diverse system scale in different dataset. For CIFAR10/100-LT datasets, we engaged 40 clients with full participation in each training round, setting the number of local epochs to 5. For ImageNet/iNat datasets, the experiments involved 20 clients, with a 40% participation rate in each training round, and the number of local epochs was set to 3.  **We have updated Section A.1, Detailed Setup, and marked the modifications in red for clarity.**

To assess FedLoGe's resilience and effectiveness in diverse settings,  we conducted experiments on CIFAR100-LT with imbalance factor of 100 and heterogeneity coefficient of 0.5, while lowering the participation rate to 30%. The results, documented in the table below, shows that our approach remains superior, particularly in terms of personalized performance. 

To further explore the effectiveness of FedLoGe in a cross-device FL setting, we increased the number of clients to 100. The results demonstrate that FedLoGe significantly outperforms other algorithms.

| Method | 40 clients(100% participate) | 40(30%) | 100(30%) |
| --- | --- | --- | --- |
| FedAvg | GM0.3818/PM0.6214 | 0.3746/0.6205 | 0.3718/0.6384 |
| FedRoD | 0.3919/0.6919 | 0.3758/0.7156 | 0.3727/0.7665 |
| FedETF | 0.3825/0.6421 | 0.4180/0.7083 | 0.3847/0.7822 |
| FedLoGe | 0.4233/0.7285 | 0.4206/0.7524 | 0.4159/0.8008 |

Reviewer comment (Experiments):
Does the paper focus on cross-silo FL, cross-device FL, or both?

Author response:
Q4. Some settings are not clear: e.g., the number of clients participating in training each round and the number of local epochs. How many clients participate in training each round? Does the paper focus on cross-silo FL, cross-device FL, or both?

Thank you for detailed suggestions that really help in refining our work. FedLoGe covers diverse system scale in different dataset. For CIFAR10/100-LT datasets, we engaged 40 clients with full participation in each training round, setting the number of local epochs to 5. For ImageNet/iNat datasets, the experiments involved 20 clients, with a 40% participation rate in each training round, and the number of local epochs was set to 3.  **We have updated Section A.1, Detailed Setup, and marked the modifications in red for clarity.**

To assess FedLoGe's resilience and effectiveness in diverse settings,  we conducted experiments on CIFAR100-LT with imbalance factor of 100 and heterogeneity coefficient of 0.5, while lowering the participation rate to 30%. The results, documented in the table below, shows that our approach remains superior, particularly in terms of personalized performance. 

To further explore the effectiveness of FedLoGe in a cross-device FL setting, we increased the number of clients to 100. The results demonstrate that FedLoGe significantly outperforms other algorithms.

| Method | 40 clients(100% participate) | 40(30%) | 100(30%) |
| --- | --- | --- | --- |
| FedAvg | GM0.3818/PM0.6214 | 0.3746/0.6205 | 0.3718/0.6384 |
| FedRoD | 0.3919/0.6919 | 0.3758/0.7156 | 0.3727/0.7665 |
| FedETF | 0.3825/0.6421 | 0.4180/0.7083 | 0.3847/0.7822 |
| FedLoGe | 0.4233/0.7285 | 0.4206/0.7524 | 0.4159/0.8008 |

Reviewer comment (Theory):
Why the proposed method can improve performance on FL with long-tailed problem?

Author response:
>Q5. Why the proposed method can improve performance on FL with long-tailed problem? （GLA-FR）

Thanks for the question. Here's a detailed explanation of the mechanisms involved:

**The classifier's class vector norm positively correlates with its predictive power and the proportion of that class's samples in the overall training set.**  The pattern has been noted in references [4, 5, 6]. Reference [7] offers an empirical explanation for this pattern in terms of decision boundaries. Additionally, within the neural collapse framework, references [9, 10] provide theoretical justification for the relationship between the norm and data distribution, based on the peer model.

**The global model trained on long-tailed often exhibits biases**, favoring predictions for head classes. GA-FR addresses this by implementing self-normalization, dividing each class vector by its own norm to make them unit vectors in order to encourage the model to make more fair judgments. 

As seen in **Fig.2, the norm of $\psi$ becomes more balanced distributed after Global Adaptive Feature Realignment (GA-FR).** For experiments, we have reported improvements in Tables 1 and 2 and there are incredible performance enhancements for few and medium classes.

It is important to note that Local Adaptive Feature Realignment (LA-FR) is designed to tailor the local model to fit local data, which is heterogeneous rather than typically long-tailed.


More discussion on the motivation and effectiveness of using the integrated personalized classifier for each client as well as the feature alignment can be found in the response of Reviewer Nmh4, Q2, and Reviewer YHb3, Q3.

[1] Exploiting Shared Representations for Personalzied Federated Learning. PMLR 2021.

[2] Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features. IJCAI 2022.

[3] Fedic: Federated learning on non-iid and long-tailed data via calibrated distillation. IEEE ICME 2022.

[4] Decoupling representation and classifier for long-tailed recognition. ICLR 2020.

[5] Equalization loss v2: A new gradient balance approach for long-tailed object detection. CVPR 2021.

[6] Over-coming classifier imbalance for long-tail object detection with balanced group softmax. CPVR 2020.

[7] Adjusting decision boundary for class imbalanced learning. IEEE Access 2020.

[8] Prevalence of neural collapse during the terminal phase of deep learning training. PNAS 2020.

[9] Neural collapse in deep linear network: From balanced to imbalanced data. ICML 2023.

[10] Imbalance trouble: Revisiting neural-collapse geometry. NIPS 2022.

Reviewer comment (Theory):
Convergence analysis is missing, which is very important for the optimization process of federated learning. Please analyze it from the experimental and theoretical point of view.

Author response:
We thank the reviewer for the thoughtful suggestion. We agree that the theoretical analysis would benefit the significance of the proposed method. However, given the limited time for rebuttal, we could not include a complete convergence analysis in this response. Drawing inspiration from neural collapse, a rich body of derivations about convergence proofs, based on the unconstrained features model and peel model, can be found in references [14, 15, 16, 17, 18, 19, 20, 21]. Based on this, we can extend these proofs to the federated framework and we shall provide the analysis in our revised supplementary material.


While we are currently unable to provide formal proof of convergence, we have conducted experimental investigations into the convergence. Specifically, we explored the number of rounds required to reach an accuracy of 0.33 on CIFAR100 with an imbalance factor (IF) of 100 and alpha=0.5, as documented in the table below. It is evident that the convergence rate of FedLoGe is almost on par with FedAvg and surpasses the current state-of-the-art (SOTA) methods.

| Methods | FedETF  | FedROD | FedLoGe (Ours) |
| --- | --- | --- | --- |
| Convergence Round | 83 | 68 | 62 |

Reviewer comment (Theory):
Lack of discussion about privacy. Federated learning is proposed to protect the privacy of the client, but the method in this paper has the risk of gradient disclosure, please add the discussion of privacy in this work.

Author response:
Thank you for highlighting the importance of discussing privacy aspects in our work. **In the entire training process of FedLoGe, no extra information compared with FedAvg is transmitted across clients and the server:** 1. During the 'Representation Learning with SSE-C' phase, FedLoGe transmits local model weights as FedAvg necessitates. 2. In the 'Feature Realignment' phase, the local models are already downloaded for norm analysis, and there are no further information-sharing operations required.

As the norm analysis is performed on the client side or global side, there would still be a few privacy concerns. However, noteworthy that the potential privacy issue exists in the general FL frameworks rather than specific to our proposed Global and Local Adaptive Feature Realignment (GLA-FR). For instance, gradient inversion[1] can pose a threat to almost all gradient transmission-based FL methods without any privacy-preservation techniques.  Additional defense techniques such as Differential Privacy [22] and Homomorphic Encryption [23] could be considered to further enhance the performance of our work. As the privacy issue of FL framework is beyond the scope of this work, **we have included the discussion about privacy in the paper A.10, marked in red for clarity.**

Reviewer comment (Experiments):
The communication overhead of the model seems to be very large, which will restrict the practical application. Please increase the experiment and analysis of the communication overhead.

Author response:
**In fact, FedLoGe ensures minimal communication overhead, as it does not require the transfer of any additional information beyond what is already exchanged in FedAvg.**

For FedLoGe's computational cost, please refer to the experimental analysis presented in **Appendix A.3**. During the backbone training phase, using frozen $\psi_{SSE-C}$ means there’s no need to compute gradients in the fixed dense linear layer. Furthermore, feature realignment is a one-shot operation that requires only basic arithmetic computations. We conducted feature realignment (GLA-FR) for 40 clients and one global model on an NVIDIA GeForce RTX 3090, which took only 1.876e-2 seconds. Additionally, we assessed the computational expense of a single round of FedLoGe training on the PyTorch platform. The results presented in the table below show that FedLoGe is a lightweight algorithm like FedAvg and FedBABU.

| Method | FedAvg | FedProx | FedRep | FedBABU | FedLoGe |
| --- | --- | --- | --- | --- | --- |
| Time Cost/Round | 3min11s | 5min24s | 4min07s | 3min14s | 3min19s |

Reviewer comment (Novelty):
The related work is simply a list of existing methods. Please add a discussion of the differences between this work and previous work to further clarify the contribution of this paper.

Author response:
Thank you for pointing out the need for a more detailed discussion of related work comparisons. Your feedback will help us clarify the distinct contributions of our paper and **we have updated it in A.11** in red in the paper. To the best of our knowledge, w**e are the first to achieve joint global and local training in Federated long-tailed learning from the perspective of neural collapse.** In the introduction, we demonstrated the improvements of FedLoGe over some existing methods, and here we further provide the discussion:

1. Most existing methods in personalized federated learning [2, 3, 4, 5, 6] utilize the global model to regularize or construct personalized models but **do not evaluate them in a generic setup**, failing to yield a strong global model without long-tail bias. This makes them unsuitable as a strong incentive for attracting new clients. In contrast, our approach learns models that **excel in both personalized (LA-FR) and generic (GA-FR) setups** without compromising either.
2. Existing methods in federated long-tailed learning [7, 8, 9, 10, 11], focusing only on calibrating long-tail bias for a single global model, **overlook the aspect of representation learning (backbone) under federated long-tailed settings** and fail to provide personalized models. For instance, CreFF involves post-classifier retraining but does not advance in boosting the backbone and does not offer personalized models.  In contrast, Our FedLoGe **achieves strong representation learning with SSE-C** to support both global model and local models.
3. Existing work involving fixed classifiers has not delved deeply into the analysis of the initialization of these fixed classifiers. FedBABU[12], for instance, merely employs a randomly initialized fixed linear layer. For FedETF[13], the vallina fixed ETF classifier has **suffered from feature degeneration**, characterized by significant fluctuations between features and the class mean, while **FedLoGo alleviates this issue** by filtering out noisy smaller mean features (Fig. 1), making features more concise and effective (Fig. 3 (b, c) and Fig. 6). Besides, **FedLoGe is less computationally expensive and easier to adapt to existing algorithms.** DR loss is necessary for FedETF, which means some modifications are required when integrating with existing algorithms. Besides, FedETF needs a projection layer for optimization. For ResNet18 with four blocks, a 512x512 projection layer would be approximately 1MB, almost doubling the parameter size of Block 1, which is 0.56MB.

Reviewer comment (Theory):
(quality) The generic-local setting looks self-contradictive to me. When the personalized federated learning targets training models personalized for each client, is it necessary to consider a model adapted for global or generic distribution? In personalized federated learning, the assumption is that each client has its own data distribution and it aims to learn a model adapted for the specific distribution. If the global or generic distribution is considered, which client will be expected to use the model?

Author response:
Thank you for raising this important question about the necessity and role of a global model in personalized federated learning. We shall outline our motivation for the generic-local setting as follows:

1. **The global model plays a crucial role as it forms a synergistic and progressively enhancing relationship with the local models, leading to mutual improvements and advancements**. The global model becomes more robust and versatile by aggregating the learning outcomes of various local models, while the local models gain a better starting point from the updated global model, enabling them to learn the unique characteristics of their local data more effectively.
2.  **A high-quality global model acts as a strong incentive for attracting new clients to join the federated training**. When new clients join the training, they can directly access the quick benefits of a good generic model and continuously share local knowledge in the following training. Otherwise, training from scratch could be a cost-intensive option. Without well trained global model, existing local clients are unwilling to share their own private personalized models with the new clients who make no contributions to the training process.
3. **A high-quality global model supports faster local model adaptation**.  Taking into account scenarios where users opt to **purchase the global model for local adaptation**, rather than joining the federated training process. We conduct an experiment to investigate the local adaptation efficiency with a high-quality global model. Assuming 20 clients take a trained global model for local fast adaptation, we have documented the average accuracy of local models across various local adaptation epochs **with and without Global Adaptive Feature Realignment (GA-FR)**, as shown in the table below. It shows that employing GA-FR can achieve better performance in the same training epochs.
4. **Other personalized federated learning papers have also considered the generic-local setting [1, 2]**, though they do not specifically designed for long-tailed and imbalanced data.

|  | epoch 15 | epoch 20 | epoch 25 |
| --- | --- | --- | --- |
| Vallina Global Model | 0.6117 | 0.6331 | 0.6441 |
| Global Model with GA-FR | 0.6544 | 0.6606 | 0.6642 |

Reviewer comment (Novelty):
(significance) Because of my concerns about the problem setting, I am afraid that the proposed method may have a limited impact on a few specific problems.

Author response:
Thank you for your concern regarding the potential scope of impact of our proposed method. **Data privacy and long-tailed distribution are more common than exceptional in many real-world scenarios, and FedLoGe is applicable in these situations.** As more clients join the training process, FedLoGe not only offers excellent personalized solutions but also provides a superior global model that attracts even more clients, thereby creating a positive feedback loop.

For instance, patients’ diagnosis varies substantially across medical centers but collaboratively form long-tailed distributions[3, 4]. FedLoGe can provide personalized models for specific local diagnostics, while also offering a high-quality global model that incentivizes other medical centers to participate and contribute to the training process.

Reviewer comment (Theory):
(clarity) The technical motivation is not clear. The neural collapse has been used for handling data heterogeneity and classifiers should be fixed by the neural collapse theorem. It is unclear why the realignment is still needed. Why the fixed classifier cannot be used for all clients?

Author response:
**The realignment is still needed because a single fixed classifier is not suitable for all clients with diverse local data**. The fixed classifier is employed to train a robust backbone, while realignment further customizes the classifier, tailoring the model for specific local data. As illustrated in the table below, we experimented with various classifiers and the results clearly show that Local Adaptive Feature Realignment (LA-FR) is most effective.

In the table, $\psi$ denotes the global classifier and $\phi_k$ denotes the classifier of the $k^{th}$ client. We derive the final prediction head for the $k^{th}$ client $\phi_k^{\prime}$ combine with $\phi_k$  and $\psi$ with $\phi'_{k,c}=\psi_c$

$*\left\|\phi_{k,c}\right\|_2$.

| $\phi_k$ | $\psi$ | $\psi_{SSE-C}$ | $\phi_k^{\prime}$ |
| --- | --- | --- | --- |
| 0.6986 | 0.7128 | 0.7146 | 0.7341 |

Furthermore, the table also indicates that only relying on a personalized classifier does not yield optimal performance either. The local head, lacking global knowledge, tends to overfit to local data. In contrast, the combination of global and local heads, as implemented with our LA-FR, can leverage both the extensive global knowledge and the specific data statistics of the local context.

Reviewer comment (Novelty):
Please provide clear motivation for the generic-local setting. When the personalized federated learning targets training models personalized for each client, is it necessary to consider a model adapted for global or generic distribution? In personalized federated learning, the assumption is that each client has its own data distribution and it aims to learn a model adapted for the specific distribution. If the global or generic distribution is considered, which client will be expected to use the model?

Author response:
Thank you for raising this important question about the necessity and role of a global model in personalized federated learning. We shall outline our motivation for the generic-local setting as follows:

1. **The global model plays a crucial role as it forms a synergistic and progressively enhancing relationship with the local models, leading to mutual improvements and advancements**. The global model becomes more robust and versatile by aggregating the learning outcomes of various local models, while the local models gain a better starting point from the updated global model, enabling them to learn the unique characteristics of their local data more effectively.
2.  **A high-quality global model acts as a strong incentive for attracting new clients to join the federated training**. When new clients join the training, they can directly access the quick benefits of a good generic model and continuously share local knowledge in the following training. Otherwise, training from scratch could be a cost-intensive option. Without well trained global model, existing local clients are unwilling to share their own private personalized models with the new clients who make no contributions to the training process.
3. **A high-quality global model supports faster local model adaptation**.  Taking into account scenarios where users opt to **purchase the global model for local adaptation**, rather than joining the federated training process. We conduct an experiment to investigate the local adaptation efficiency with a high-quality global model. Assuming 20 clients take a trained global model for local fast adaptation, we have documented the average accuracy of local models across various local adaptation epochs **with and without Global Adaptive Feature Realignment (GA-FR)**, as shown in the table below. It shows that employing GA-FR can achieve better performance in the same training epochs.
4. **Other personalized federated learning papers have also considered the generic-local setting [1, 2]**, though they do not specifically designed for long-tailed and imbalanced data.

|  | epoch 15 | epoch 20 | epoch 25 |
| --- | --- | --- | --- |
| Vallina Global Model | 0.6117 | 0.6331 | 0.6441 |
| Global Model with GA-FR | 0.6544 | 0.6606 | 0.6642 |

Reviewer comment (Evaluation):
How do the mean and variance of features relate to the feature collapse in the quoted content?  
> However, preliminary experiments benchmarking ETF with CIFAR-100 in Fed-LT suggest that only a few features have relatively large means, while most of the small-mean features are contaminated by severe noise, as shown in Fig. 1(a). Such observations are inconsistent with the feature collapse property, and we coin it as feature degeneration.

Author response:
Thanks for the questions. The objective of analyzing variance and means of features is to examine whether the properties revealed by neural collapse theory are maintained when using a fixed ETF classifier.

**Relations bewteen feature collapse with mean and variance of features:** Neural collapse reveals a principle, namely feature collapse, which shows that in a properly trained network, features should effectively collapse into the class mean (**where all relative variances in Fig. 1 (a) are expected to be zero)**. Yet, when trained with a vallina fixed ETF classifier, **features, particularly those with smaller means, do not collapse effectively, namely feature degeneration. FedLoGe is designed to mitigate feature degeneration by squeezing poor-quality and noisy features into these sparsely allocated positions.** Overall, as shown in Fig. 1, after training with $\psi_{SSE-C}$, FedLoGe inherently filters out noisy smaller mean features, making features more concise (Appendix A.4) and effective (Table 3 in the main paper), in which the variance of dominant features becomes smaller. 

**Main changes between Fig. 1 (a) and (b):** Compared to Fig. 1 (a), **two significant changes** are evident in Fig. 1 (b) after implementing FedLoGe: **1. Noisy features are masked.** The small mean features with bigger variances are located in the sparsely allocated area, marked by grey vertical shadows. 2. **The quality of dominant features is further enhanced**. This is evidenced by the decreased variance in features with larger means, indicating they have become more concise and effective.

We give the detailed computation process for the data in Fig. 1:
Given that $n_c$ is the sample number of class $c$.  $\mu_c$ is the class mean of class $c$, and $h_{i,c}$ is the feature of class $c$ on sample $i$, the features $h_{i,c}$ will collapse to the within-class mean $\mu_c = \frac{1}{n_c} \sum_{i=1}^{n_c}h_{i,c}$. This indicates that the covariance $\frac{1}{n_c} \sum_{i=1}^{n_c}(h_{i,c}-\mu_c)(h_{i,c}-\mu_c)^T$ will converge to $0$. We investigate whether $h_{i,c} - \mu_c$ tends toward zero when training with a fixed ETF classifier, as shown in Fig. 1 (a). 

The steps to obtain the data in Fig. 1 are：

1. Select class $c$.
2. During global test, compute all sample features $h_{i,c}$, with dimension $D$, representing the number of features. Use $h_{i,c}[d]$ to denote the $d^{th}$ feature in $h_{i,c}$.

 3.  Compute the mean and relative variance of $h_{c}$, sorting them in descending order of the mean, and accordingly adjust the variance positions.

It becomes evident that not all features, $h_{i,c}[d] - \mu_c[d]$, converge to zero, and smaller $\mu_c[d]$ values exhibit greater variance.

The process illustrated in **Fig.1 (b)** follows the same steps as **Fig.1 (a)**, but with grey vertical shadows for the sparsely positioned features. It can be observed that sparse ETF training effectively squeezes poor-quality and noisy features into these sparsely allocated positions.

**All the calculation details have been updated in the revised appendix A.2 marked in red for clarity.**

Reviewer comment (Theory):
Authors need to provide detailed motivation of each step in the proposed algorithm, such as the motivation of keeping two classifier heads for clients: $\phi_k$ and $\psi_k$, and realignment of the global classifier $\psi$ in Eq 6.

Author response:
Thank you for your insightful questions regarding the detailed motivations behind the steps in our proposed algorithm. I am pleased to explain the motivations point by point. FedLoGe achieves joint training of generic and local models by decoupling representation learning (backbone) and classifier learning [1, 2] in sequential order:

Step 1: During the backbone training phase, we employ the fixed SSE-C classifier to address the divergence caused by data heterogeneity. A well-trained shared backbone is essential, as it lays the foundation for enhancing the potential of both the global model and local models.

Step 2: In the feature realignment stage, Global and Local Adaptive Feature Realignment (GLA-FR) focuses on aligning the model's capabilities with the corresponding data distributions. Building on a robust backbone, we further customize the classifiers to fit corresponding test data. The global model is designed for fairly treating all classes (GA-LR), **highlighted in line 8 of the Revised Algorithm**. Meanwhile, the personalized models are aligned with the local statistics **(LA-FR, Eq. 7)**, seen in **Revised Algorithm, line 11.**

Specifically, we explain the motivation and objective of the feature alignment of FedLoGe. **The objective of feature realignment (FR) is to address challenges brought by the heterogeneous and imbalanced data.** In the context of long-tailed learning, it is commonly acknowledged that the classifier's class vector norm positively correlates with its predictive power and the proportion of that class's samples in the overall training set[2, 3, 4, 5, 6, 7]. In Fed-LT, the long-tailed data would lead to a highly imbalanced classifier throughout the training process. Therefore, we use FR to re-balance the norm of the classifier for a balanced representation learning with heterogeneous global/local data.

GA-FR is effective because it aligns the long-tailed norm to the balanced distribution caused by a long-tailed training set, thus balancing the predictive abilities of all classifiers. Due to the inconsistency of local and global data distribution, GA-FR fails to fit the local statistic. **We design LA-FR, which adapts the global classifier to fit the local statistics by local classifier norms.** We have provided a detailed explanation of keeping two classifier heads in response to Question 2.

Reviewer comment (Presentation):
Writing: Certain citation formats need to be fixed; Figure 1 is difficult to comprehend.

Author response:
Thanks for the question and we are glad to clarify what Fig. 1 implies. Compared to Fig. 1 (a), **two significant changes** are evident in Fig. 1 (b) after implementing FedLoGe: **1. Noisy features are masked.** The small mean features with bigger variances are located in the sparsely allocated area, marked by grey vertical shadows. 2. **The quality of dominant features is further enhanced**. This is evidenced by the decreased variance in features with larger means, indicating they have become more concise and effective.

Below is a detailed interpretation of Fig. 1:

Neural Collapse (NC) theory highlights a key characteristic of successful neural network training, which is the convergence of all sample features (outputs of the backbone) within the same category to the class mean of that category.
**Yet, when trained with a vallina fixed ETF classifier (Fig. 1 (a)), features with smaller means do not collapse effectively, namely feature degeneration. FedLoGe(Fig. 1 (b)) is designed to mitigate feature degeneration by squeezing poor-quality and noisy features into these sparsely allocated positions.**

Overall, as shown in Fig. 1, after training with $\psi_{SSE-C}$, FedLoGe inherently filters out noisy smaller mean features, making features more concise (Appendix A.4) and effective (Table 3 in the main paper), in which the variance of dominant features becomes smaller.

We give the detailed computation process for the data in Fig. 1:
Given that $n_c$ is the sample number of class $c$. $\mu_c$ is the class mean of class $c$, and $h_{i,c}$ is the feature of class $c$ on sample $i$, the features $h_{i,c}$ will collapse to the within-class mean $\mu_c = \frac{1}{n_c} \sum_{i=1}^{n_c}h_{i,c}$. This indicates that the covariance $\frac{1}{n_c} \sum_{i=1}^{n_c}(h_{i,c}-\mu_c)(h_{i,c}-\mu_c)^T$ will converge to $0$. We investigate whether $h_{i,c} - \mu_c$ tends toward zero when training with a fixed ETF classifier, as shown in Fig. 1 (a).

The steps to obtain the data in Fig. 1 are：

1. Select class $c$.
2. During global test, compute all sample features $h_{i,c}$, with dimension $D$, representing the number of features. Use $h_{i,c}[d]$ to denote the $d^{th}$ feature in $h_{i,c}$.

3. Compute the mean and relative variance of $h_{c}$, sorting them in descending order of the mean, and accordingly adjust the variance positions.

It becomes evident that not all features, $h_{i,c}[d] - \mu_c[d]$, converge to zero, and smaller $\mu_c[d]$ values exhibit greater variance.

The process illustrated in **Fig.1 (b)** follows the same steps as **Fig.1 (a)**, but with grey vertical shadows for the sparsely positioned features. It can be observed that sparse ETF training effectively squeezes poor-quality and noisy features into these sparsely allocated positions.

**All the calculation details have been updated in the revised appendix A.2 marked in red.**

Reviewer comment (Novelty):
Missing discussion and comparison with related work on long-tailed FL. Authors should specify the technical contributions of the proposed method over other prior work, especially FedETF, which shows much resemblance to the proposed work.

Author response:
Other long-tailed FL approaches [8, 9, 10, 11] focus on a single global model, overlooking personalized client needs. In contrast, FedLoGe is capable of simultaneously training both generic and local models. Notably, FedLoGe integrates representation learning and feature realignment under the unified framework of neural collapse.
Below are the detailed differences between FedLoGe and FedETF:

1. FedETF is only designed to optimize a **single global** model, while our FedLoGe optimizes **both global model and client models** which is more appealing in practice.
2. The ETF classifier has been noted to **suffer from feature degeneration**, characterized by significant fluctuations between features and the class mean, while **FedLoGo alleviates this issue** by filtering out noisy smaller mean features, making features more concise and effective.
3. **FedLoGe is less computationally expensive and easier to adapt to existing algorithms.** DR loss is necessary for FedETF, which means some modifications are required when integrating with existing algorithms. Besides, FedETF needs a projection layer for optimization. For ResNet18 with four blocks, a 512x512 projection layer would be approximately 1MB, almost doubling the parameter size of Block 1, which is 0.56MB. We conducted an ablation study on DR Loss and the Projection Layer, as shown in **the table below**. The results demonstrate that FedLoGe can achieve good performance even without the DR Loss and Projection Layer.

| Method  | DR Loss | Projection Layer | Many   | Medium | Few    | All    |
| ------- | ------- | ---------------- | ------ | ------ | ------ | ------ |
| CreFF   | -       | -                | 0.684  | 0.440  | 0.146  | 0.401  |
| FedGraB | -       | -                | 0.683  | 0.553  | 0.221  | 0.411  |
| FedETF  | ×       | ×                | 0.6904 | 0.4751 | 0.1820 | 0.3825 |
| FedETF  | √       | ×                | 0.6882 | 0.4607 | 0.2059 | 0.3932 |
| FedETF  | √       | √                | 0.6909 | 0.5011 | 0.2310 | 0.4109 |
| FedLoGe | ×       | ×                | 0.7137 | 0.4989 | 0.2179 | 0.4249 |

**Please refer to the detailed discussion of the differences between our work and previous work in the responses to Reviewer tSL5 (W4), or in Section A.11 of our paper.**

Reviewer comment (Theory):
How to optimize Eq (4) that contains a maximization over $j$?

Author response:
Thank you for your question regarding the details of Equation (4). We select two classifier vectors $\mathbf{\hat{\mathbf{\psi}}^{\prime}}_{:,i}$ 

and $\mathbf{\hat{\mathbf{\psi}}^{\prime}}_{:,j}$ $(i \neq j)$ which have maximum cosine similarity. Then we transfer cosine similarity to degree and optimize it. The calculation steps for Maximal Angle Loss are shown in the code below:

```python
# Constraint 2: Maximize the angle between vectors (minimize cosine similarity)
normalized_etf = sparse_etf / row_norms
cos_sim = torch.mm(normalized_etf.t(), normalized_etf)
torch.diagonal(cos_sim).fill_(-1)
angle_loss = -torch.acos(cos_sim.max(dim=1)[0].clamp(-0.99999, 0.99999)).mean()
```

We shall provide a more detailed explanation of the details of the optimization process.

Please access more details about the Maximal Angle Loss in **Supplementary Material** ( ./util/etf_methods.py).

Reviewer comment (Theory):
What is the purpose of introducing sparsity to the SSE-C classifier, besides its empirical effectiveness?

Author response:
Thank you for your insightful inquiry about the role of sparsity in the SSE-C classifier. The introduction of sparsity serves several key purposes beyond its empirical effectiveness, as outlined below:

1. Non-sparse ETF results in unstable distances between features with smaller samples means and the class means, while enforcing **sparsity would mitigate feature degeneration** by squeezing poor-quality and noisy features into these sparsely allocated positions. We demosntrate this in **Fig. 1.** Please check Q3 for more details.
2. Introducing sparsity helps **learn more concise and dominant features** for better classification, which is evident in **Figures 3(b) and 3(c)**. It shows that after training with SSE-C, pruning dominant features drastically decreases performance, while pruning negligible features barely affects performance.
3. Sparsity can help **learn more discriminant decision boundaries**. We have conducted a T-SNE visualization of the class means before and after sparsification, detailed in **Appendix A.4**. This visualization reveals that the decision boundaries of class mean post-sparsification are more distinct.