Paper: FedNovel: Federated Novel Class Learning (ICLR-2024)

Reviewer comment (Presentation):
The method lacks sufficient elaboration and requires additional effort to comprehend the underlying technique. For instance, it encompasses multiple design elements, yet Figure 1 fails to offer adequate details to facilitate a clear understanding of the approach. And it is difficult to visualize the algorithm. It would be better if the paper provided an algorithm block. (Q1, Q2, Q4)

Author response:
Thanks for your suggestion. We have revised Figure 1 in our manuscript to include more information and details, enabling it to more clearly express the method we propose. As for the algorithm flowchart, we have provided the optimization pipeline for the novel class learning stage in the appendix (Section C). In our revision, we present the complete GAL algorithm process, encompassing both the known class learning and novel class learning stages.

Reviewer comment (Theory):
Some of the assertions made in the paper need additional justifications. (Q3)

Author response:
Thank you for your comment. We have answered your questions in detail in the subsequent Question Section. Please refer to the responses provided below.

Reviewer comment (Experiments):
Section 3.2 mentions the unlabeled testing datasets "belongs to a unified novel label space", does it assume the number of classes of novel samples is fixed? If this number is not fixed, then how to decide the number of clusters? If this number is fixed, how does it apply to the setting where the novel data are continually emerging?

Author response:
In FedNovel, we assume that **novel class learning can be conducted periodically**. During each novel class learning stage, we only invite clients who are active and available at the time to participate, and we pause the update of their data memories. Therefore, in this case, the number of novel classes in each novel class learning stage remains constant. For the continuously incoming novel data, we can initiate another stage of novel class learning when the novel data memory of the majority of clients is nearly full.

Reviewer comment (Theory):
Section 3.3.2 chooses neuron weights as prototypes since the data from other clients is unavailable in the FL setting. However, there is still a lot of difference between weights and features. Any justification to support that using weights as prototypes is as effective as data/features?

Author response:
Your understanding is correct. The weights of a standard linear fully connected layer indeed differ significantly from features. However, under specific conditions, if we ensure that **the bias of the linear fully connected layer are zero**, and train the neural network using the cross-entropy loss, the weights of the linear fully connected layer can be viewed as prototypes [1]. Moreover, the classifier consists solely of a single unbiased linear fully connected layer, which is put right after the feature extractor. Consequently, **the dimensionality of this layer's weights is the same as that of the extracted representations**. With the bias consistently set to zero, the computation of cross-entropy loss is based on the softmax probabilities derived from the inner products between the representations and the weights of that linear layer. Minimizing cross-entropy loss increases the inner product between the sample representation and the corresponding neuron weight (associated with the sample's ground truth label). The inner product between vectors is proportionally related to the cosine similarity between them. Thus, **an increase in the inner product equals an increase in cosine similarity**. Therefore, we can consider the neuron weights of the linear fully connected layer as prototypes based on cosine similarity.

Reviewer comment (Writing):
The "anchor sample" first appears in Section 3.3.1. How are they selected, or what is their definition?

Author response:
Thanks for your question and careful reading. The term 'anchor sample' was a typo. Our original intention was to refer to each regular training sample. We have removed the term 'anchor' in our revision.

Reviewer comment (Reproducibility):
In Section 3.3.2, there is a data memory storing filtered-out data. Are these data filtered out because they are known class data?

Author response:
Yes, your understanding is correct. Before the formal start of novel class learning, clients' novel data memory will collect continuously incoming streaming data. The streaming data includes both known class data and potentially data from unseen novel classes. **Our current design does not utilize known class data in novel class learning**, thus we need to filter it out. Then, clients' novel data memory will store only novel class data. We have also conducted **experiments with the presence of known class data**; please refer to Section A.3 in the appendix. According to the experimental results, using known class data in the novel class learning stage does not lead to performance improvement. Therefore, to further reduce the cost of storage and computational resources, we choose to filter out known class data.

Reviewer comment (Novelty):
Lacks of crucial reference literatures [A][B]. Thus, Federated New Class Discovery/Learning is not a new research problem. In contribution, authors say, ”we are the first to focus on this problem and propose an effective solution”. It might be somewhat overclaiming.

Author response:
>1. Lacks of crucial reference literature.

To prevent misunderstandings, we need to clarify that Federated OpenSet/OpenWorld Learning (FOL) is related to but distinctly different from, the FedNovel problem we are focusing on. Firstly, **their application scenarios differ**. FOL is typically applied in static federated training, where the simultaneous learning of known and novel classes is achieved within a single training phase. In contrast, FedNovel focuses on dynamic federated training, where the federated model, after having been employed for a period following the training of known classes, is then found to require the learning of novel classes. **For dynamic federated training, the goal extends beyond learning novel classes to effectively mitigate the forgetting of known classes. FOL, which only considers single-phase training, is not applicable in continual dynamic federated learning.**

Secondly, **the training settings differ**. For FOL, given its focus on static federated training, clients' local data is also static, meaning there is no continual influx of streaming data, and **labeled data can be statically retained for training**. Furthermore, static federated training does not allow the participation of new clients. In contrast, new clients in dynamic federated training often have no data at all. Although their streaming data continues to arrive, it also lacks any labels. FedNovel specifically addresses the challenges posed by the continual arrival of streaming data. It assumes that during the novel class learning stage, **there will be no labeled data available, whether for known or novel classes**.

Certainly, we understand your concern. We have already discussed the similarities and differences between FedNovel and FOL in the related work section and **removed the claim of the first work in our revised manuscript**. Furthermore, we have **modified the title of our paper to "Federated Continual Novel Class Learning"** to avoid any confusion and misunderstanding.

Reviewer comment (Novelty):
Insufficient comparison and discussion. From my understanding, the proposed method is similar to commonly-used semi-supervised learning methods.

Author response:
>2. Insufficient comparison and discussion.

Commonly used semi-supervised learning (SSL) typically assumes that the training data is partially labeled and that **both labeled and unlabeled data belong to the same label space**. The most prevalent method in SSL is progressive learning. This approach initially trains the model on labeled data, then tests it on unlabeled data, selecting high-confidence data to add back into the training set. As training goes on, an increasing amount of unlabeled data is incorporated, gradually improving the model's performance. In contrast, **the novel class learning problem that FedNovel focuses on has no labeled data available, and the novel classes do not overlap with the known classes**. Under such settings, even if a model trained on known classes is tested on novel classes, the confidence becomes meaningless due to the completely different label spaces.

We understand your concerns. During the rebuttal phase, we conducted comparison experiments between our GAL approach and a federated semi-supervised learning work, SemiFL [F]. The experiment results are provided below, and we can observe that GAL can still substantially outperform SemiFL.

| Dataset |  | CIFAR-100 |  |  |  | Tiny-ImageNet |  |  |  | ImageNet-Subset |  |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|  | known | novel | all |  | known | novel | all |  | known | novel | all |
| SemiFL | 42.3 | 28.8 | 39.6 |  | 33.2 | 25.5 | 32.5 |  | 37.1  | 27.5 | 35.2 |
| Our GAL | 72.6 | 45.8 | 67.3 |  | 57.6 | 32.7 | 55.0 |  | 55.8 | 29.7 | 50.5 |

Reviewer comment (Novelty):
Limited novelty. From the perspective of NCD methods, prototype-based contrastive learning and low confidence sample rejection have been explored in [C]. Why is the proposed method superior to [C]? From the perspective of semi-supervised federated learning methods, the federated prototype learning has been studied in [D][E]. What’s the novelty of the proposed methods compared with [D][E]?

Author response:
>3. Limited novelty.

Regarding your first question, **Prototype Contrastive Learning (PCL) is utilized in the known class learning stage** of our GAL, solely to enhance the model's capability for representation extraction. Moreover, **we have modified the standard PCL** by incorporating sample-wise comparison in the negative pair comparison. This strengthens the model's inter-class representation extraction ability. The **low-confidence sample rejection strategy is applied only in the data filtering of GAL** for both known and novel classes, and this filtering is only used before the initiation of novel class learning. In summary, the application of PCL and the low-confidence sample rejection strategy are merely preparatory steps before GAL initiates novel class learning. We need to mention that **our primary contributions lie in the global novel class number estimation and discriminative novel class learning** that occur after the initiation of novel class learning, corresponding to the Potential Prototype Merge (PPM) and Semantic Weighted Loss (SWL) in our GAL framework. As for why Opencon [C] may underperform compared to GAL, a plausible explanation is that although local contrastive learning enables local models to learn distinguishable clustering representations, in FedNovel, the novel label spaces differ across clients. This leads to various levels of shifts and rotations in the clustering representation distribution learned by local models, resulting in significant location differences of the same novel classes in the representation space.

As for your second question, first, for the FedNovel problem, our proposed **GAL is not based on prototypical networks or prototype learning.** In GAL, to protect privacy, prototype communication is conducted only once, utilizing PPM to construct global prototypes. Once the global prototypes are constructed, they are used to initialize the novel class classifier, which is then updated with local training. After this, GAL does not include any prototype communication. The two works [D, E] you mentioned involve prototype sharing and communication in every global training round. In this setting, **GAL is more secure**. Furthermore, regarding the construction of prototypes, works D and E can reliably use labeled data, and as the training progresses, the prototypes constructed by them become increasingly reliable. However, **for GAL, there are no labeled data available to guide the construction of prototypes**, thus we use unsupervised clustering. Consequently, the prototypes constructed by GAL are highly unreliable. Therefore, **GAL treats the use of prototypes as a weakly supervised learning problem**, leading to the design of SWL. SWL innovatively optimizes the Euclidean distance between samples and prototypes, rather than cosine similarity, because the prototypes in GAL are constructed using k-means based on Euclidean distance. In addition, SWL does not arbitrarily consider only a single prototype but optimizes the relationships of a sample with all prototypes, effectively mitigating the impact of the unreliability of prototypes. In contrast, works D and E can directly utilize reliable and accurate prototypes for single pseudo labeling learning. From the problem setting to the specific solutions, GAL differs significantly from works D and E, and every aspect of GAL's design is logical and cohesively integrated.

Reviewer comment (Experiments):
From my understanding, [A][B] can also be used in federated new class learning. It is better to discuss and compare the proposed method with [A][B].

Author response:
>4. Comparison with works A, B.

Thank you for your suggestion. We have already provided a detailed discussion of the differences between our work and works A and B in the response to Question 1, and we have also included these discussions in our revised manuscript. Of course, we have conducted comparison experiments between works A, B and GAL within the FedNovel setting, with detailed results as follows. From the experimental results, it is evident that our proposed GAL still significantly outperforms works A and B.

| Dataset |  | CIFAR-100 |  |  |  | Tiny-ImageNet |  |  |  | ImageNet-Subset |  |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|  | known | novel | all |  | known | novel | all |  | known | novel | all |
| FedoSSL [A] | 62.1 | 16.6 | 53.0 |  | 53.0 | 8.1 | 48.5 |  | 55.0  | 17.7 | 47.5 |
| AGCL [B] | 63.6 | 35.2 | 57.9 |  | 40.2 | 24.4 | 38.6 |  | 46.7 | 20.0 | 41.4 |
| Our GAL | 72.6 | 45.8 | 67.3 |  | 57.6 | 32.7 | 55.0 |  | 55.8 | 29.7 | 50.5 |

Reviewer comment (Theory):
Lack of theoretical proof why the estimation method is better than other competitors.

Author response:
>5. Lack of theoretical proof for PPM.

a) Firstly, we need to clarify that, to our best knowledge, **there currently isn't a rigorous framework available for the theoretical analysis of novel class number estimation** (NCNE) in fields of novel class discovery, openworld semi-supervised learning, and deep clustering. Now, let us attempt to analyze why existing NCNE methods perform poorly in FedNovel. Existing NCNE approaches are usually based on the assumption that if labeled known class data and unlabeled novel class data are mixed and clustered using an unsupervised clustering algorithm, when the labeled known class data is effectively separated, the unlabeled novel class data will also be effectively separated. **This assumption is problematic because for it to hold, two preconditions must be met: first, in the representation space, the distributions of known classes must be similar to those of novel classes; and second, the unsupervised clustering algorithm used must be capable of handling different data distributions.** Regarding the first precondition, **work [G]'s Theorem 1** states that when a learning model is trained using the EM (Expectation-Maximization) algorithm based on limited observation of training data distribution, it will map different labels to distinct representational distributions. In FedNovel, NCNE occurs before novel class learning. Therefore, only known classes undergo EM algorithm training based on limited observations, while novel classes do not undergo such training. **Therefore, there is a significant difference in the representation distribution of known and novel classes.** Regarding the second precondition, existing methods typically employ the k-means algorithm for unsupervised clustering. However, **k-means struggles with clustering tasks where cluster distributions vary significantly [H]**. There are also other influencing factors, such as that the representation augmentation performed on known class prototypes cannot accurately generate representations that conform to the original distribution statistics. Additionally, **local prototypes are a further abstraction on limited observation of novel class data distribution** (the last challenge mentioned on page 5 of the main paper when conducting NCNE in FedNovel). 

b) In contrast, our designed PPM can effectively work without the concern of the two aforementioned preconditions. Firstly, **PPM does not use known class data**, thus the difference in representational distributions between known and novel classes does not affect PPM's effectiveness. Moreover, upon careful comparison, we found that PPM shares the same design principle with incremental DBSCAN [I]. According to **Corollary 1 from work [I]**, as the parameter $\epsilon$ of DBSCAN increases, **PPM can identify and separate clusters of different densities.** Therefore, PPM can effectively locate high-density regions in FedNovel, leading to accurate NCNE.

c) In addition to the experiments presented in the main text, we conducted **extensive experiments to validate the effectiveness of PPM**, including various scenarios of different novel class numbers across six datasets. For detailed experimental results, please refer to section B.5 in the appendix. According to these results, **PPM consistently provides the most accurate estimations,** with an estimation error significantly smaller than other methods. Certainly, in future work, we plan to explore more rigorous theoretical support for the effectiveness of GAL.

Reviewer comment (Experiments):
The reasonableness of the experiment setting still needs to be considered. As authors claimed in the paper, they use Dirichlet Distribution to control data heterogeneity. (1) It makes client labeled and unlabeled data highly-unbalanced. However, in [F], the labeled and unlabeled data are kept fixed partitioning. It is better to discuss the relationship between proposed method and data distribution. (2) It might lead clients to have some sharing categories or have non-overlapping categories. How does the proposed method solve both situations?

Author response:
>6. The reasonableness of the experiment setting still needs to be considered. 

We chose to use the Dirichlet distribution to set the non-IID heterogeneity of FL because it aligns with the real-world FL scenarios and is currently the most commonly used, most popular, and most challenging non-IID distribution in state-of-the-art FL research. As you mentioned, applying the Dirichlet distribution in FedNovel does lead to data imbalance and variation in the degree of label overlap. Regarding your questions, after careful review, our response is as follows, assuming our understanding is correct. Of course, if we have misunderstood, please do inform us further.

(1) Regarding your first question, we need to clarify that the Dirichlet distribution will cause imbalances in the labeled data during the known class learning stage, as well as in the unlabeled novel data during the novel class learning stage. However, it's important to mention that **there will be no labeled data during the novel class learning stage, from either known or novel classes**. **Therefore, the coexisting of imbalanced labeled and unlabeled data you mentioned is impossible**. Indeed, during the novel class learning, each client's novel class data distribution is different. Thus, the FedNovel solution needs to effectively address varying data distributions. Our GAL leverages PPM to construct global prototypes, providing a uniform standard for each client's local training. This enables GAL to effectively handle diverse data distributions. 

(2) Expanding the data imbalance caused by the Dirichlet distribution to extreme cases results in a scenario where different clients share some labels while having others that do not overlap. Therefore, **FedNovel deals with a hybrid situation where both these scenarios coexist, representing a more challenging problem**. In such a mixed scenario, GAL's PPM can effectively work. For example, for labels shared among clients, their corresponding local prototypes will be located in close proximity, forming high-density areas. PPM can accurately identify these areas and merge the prototypes contained within each area. As for labels that do not overlap between clients, PPM will segregate the corresponding local prototypes into separate areas. Thus, PPM can construct effective global prototypes in such mixed scenarios, allowing the subsequent novel class discriminative learning to naturally handle this case as well.

(3) In our appendix, we have included additional experiments, which are relevant to data distribution, including a) experiments in the centralized setting (Section B.1); b) experiments with much more heterogeneous non-IID settings (Section B.2); c) experiments in settings with different data partitions (Section B.3). Please refer to these experimental results, as they may be helpful in addressing your concerns.

Reviewer comment (Novelty):
It seems that open-world semi-supervised learning [1][2] also considers classifying both seen and unseen classes during the testing phase. Please compare it in related work.

Author response:
Thank you for your suggestion, we have discussed open-world semi-supervised learning (OSSL) in the Related Work of the revised paper. Although the setup of OSSL does share certain similarities with ours, there are still key differences: (1) **The data distribution differs**. In OSSL, the training data is centralized to locate a machine during training. However, in FedNovel, the data is distributed to multiple clients following the non-IID setting, which means the local data distributions of clients are distinct from each other. (2) **The data availability differs**. The data in OSSL are static, which means all labeled and unlabeled data can be accessed at the same time, thus they can be learned within a single training phase. In contrast, the data in FedNovel are dynamic. FedNovel first requires the model to learn only on labeled known classes, then moves to learn on unlabeled novel classes without access to labeled known class data. Therefore, in the novel class learning stage, there is one more challenge in FedNovel than OSSL, which is to preserve good performance on known classes.

Reviewer comment (Experiments):
In section 4, the baselines on federated self-supervised learning methods are insufficient. It is suggested to add more FedSSL methods in the experimental part.

Author response:
Thanks for your suggestion. Similar to what we mentioned before, the key difference between SemiFL and FedNovel is also the **data availability**. The data in SemiFL are static while the data in FedNovel are dynamic. SemiFL can train on all labeled and unlabeled data in the same training phase, while FedNovel can only train on labeled classes in the first training phase and unlabeled classes in the following training phase. To further evaluate the effectiveness of our GAL, we also carry out the comparison experiments with SemiFL as below. Based on the results, GAL still achieves better performance than SemiFL.

| Dataset |  | CIFAR-100 |  |  |  | Tiny-ImageNet |  |  |  | ImageNet-Subset |  |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|  | known | novel | all |  | known | novel | all |  | known | novel | all |
| SemiFL | 42.3 | 28.8 | 39.6 |  | 33.2 | 25.5 | 32.5 |  | 37.1  | 27.5 | 35.2 |
| Our GAL | 72.6 | 45.8 | 67.3 |  | 57.6 | 32.7 | 55.0 |  | 55.8 | 29.7 | 50.5 |

Reviewer comment (Experiments):
It is unclear why the value of $n_{size}$ is set as 2. A more detailed explanation should be added.

Author response:
In our Potential Prototype Merging (PPM), $n_{size}$ is used to control the data scale of high-density regions. As the data in FedNovel are non-IID distributed among clients, **a particular client may have only a few data samples for certain novel classes. Besides, for a particular novel class, it may be possible that only a few clients own this class**. Setting a small $n_{size}$ can help our PPM not ignore the novel classes that correspond to a small number of local prototypes.

Reviewer comment (Theory):
The definition of private information is vague. It seems that the number of novel classes is thought to be private, which however is not necessarily true.

Author response:
>1. The number of novel classes

**In FedNovel, the number of novel classes is not private information.** This can be verified by works [1] in federated open-world semi-supervised learning, where the server is assumed to know the exact number of unseen classes before training. In contrast, the proposed GAL can estimate the number of novel classes by using our designed Potential Prototype Merging (PPM).

[1] Towards Unbiased Training in Federated Open-world Semi-supervised Learning. ICML, 2023.

Reviewer comment (Theory):
Without specification on the privacy definition, there also lacks sufficient justification for how the proposed method will protect privacy.

Author response:
>2. Definition of private information

**The private information that needs to be protected in FedNovel is the meaningful semantic information of raw data for every client.** If existing data reconstruction attacks can't recover such semantic information from the prototypes sent by each client, we can say that our GAL does not leak private information. We have carried out experiments of launching data reconstruction attacks on the prototypes in GAL. The detailed results are shown in Appendix B.8, and the reconstructed images shown in Figure 3 look very similar to random noise in human eyes. To further verify that semantic information is not contained in the reconstructed images, we compute the Structural Similarity (SSIM) between reconstructed images and all images from each class. Then we check whether there is a set of classes that always share higher similarity with the reconstructed images and whether this class set is major included in the clusters corresponding to the attacked prototypes. Specifically, we choose the top 50% classes (10 out of 20) according to the average SSIM and set their corresponding dimensions as 1 in a zero vector. We also create another zero vector with the major component classes of a prototype as 1. Then we calculate the Manhattan Distance (MD) between these two vectors. For a better comparison, we also obtain the same vectors for random noise images and calculate the MD between vectors of reconstructed and random noise images. **According to the following results, we can conclude that data reconstruction attacks cannot recover meaningful semantic information, thus our GAL is privacy-preserving.**

|Manhattan Distance|CIFAR-100|Tiny-ImageNet|ImageNet-Subset|
|:---:|:---:|:---:|:---:|
|with the vectors of major cluster components|9.7|10.1|10.5|
|with the vectors of gaussian noise images|10.1|9.9|10.2|

Reviewer comment (Evaluation):
Though the empirical results of privacy evaluation are appreciated, the authors should clarify the meaning of privacy and adjust the claim of privacy.

Author response:
>2. Definition of private information

**The private information that needs to be protected in FedNovel is the meaningful semantic information of raw data for every client.** If existing data reconstruction attacks can't recover such semantic information from the prototypes sent by each client, we can say that our GAL does not leak private information. We have carried out experiments of launching data reconstruction attacks on the prototypes in GAL. The detailed results are shown in Appendix B.8, and the reconstructed images shown in Figure 3 look very similar to random noise in human eyes. To further verify that semantic information is not contained in the reconstructed images, we compute the Structural Similarity (SSIM) between reconstructed images and all images from each class. Then we check whether there is a set of classes that always share higher similarity with the reconstructed images and whether this class set is major included in the clusters corresponding to the attacked prototypes. Specifically, we choose the top 50% classes (10 out of 20) according to the average SSIM and set their corresponding dimensions as 1 in a zero vector. We also create another zero vector with the major component classes of a prototype as 1. Then we calculate the Manhattan Distance (MD) between these two vectors. For a better comparison, we also obtain the same vectors for random noise images and calculate the MD between vectors of reconstructed and random noise images. **According to the following results, we can conclude that data reconstruction attacks cannot recover meaningful semantic information, thus our GAL is privacy-preserving.**

|Manhattan Distance|CIFAR-100|Tiny-ImageNet|ImageNet-Subset|
|:---:|:---:|:---:|:---:|
|with the vectors of major cluster components|9.7|10.1|10.5|
|with the vectors of gaussian noise images|10.1|9.9|10.2|

Reviewer comment (Writing):
What is the definition of private information in the paper?

Author response:
>2. Definition of private information

**The private information that needs to be protected in FedNovel is the meaningful semantic information of raw data for every client.** If existing data reconstruction attacks can't recover such semantic information from the prototypes sent by each client, we can say that our GAL does not leak private information. We have carried out experiments of launching data reconstruction attacks on the prototypes in GAL. The detailed results are shown in Appendix B.8, and the reconstructed images shown in Figure 3 look very similar to random noise in human eyes. To further verify that semantic information is not contained in the reconstructed images, we compute the Structural Similarity (SSIM) between reconstructed images and all images from each class. Then we check whether there is a set of classes that always share higher similarity with the reconstructed images and whether this class set is major included in the clusters corresponding to the attacked prototypes. Specifically, we choose the top 50% classes (10 out of 20) according to the average SSIM and set their corresponding dimensions as 1 in a zero vector. We also create another zero vector with the major component classes of a prototype as 1. Then we calculate the Manhattan Distance (MD) between these two vectors. For a better comparison, we also obtain the same vectors for random noise images and calculate the MD between vectors of reconstructed and random noise images. **According to the following results, we can conclude that data reconstruction attacks cannot recover meaningful semantic information, thus our GAL is privacy-preserving.**

|Manhattan Distance|CIFAR-100|Tiny-ImageNet|ImageNet-Subset|
|:---:|:---:|:---:|:---:|
|with the vectors of major cluster components|9.7|10.1|10.5|
|with the vectors of gaussian noise images|10.1|9.9|10.2|