Paper: Flashback: Understanding and Mitigating Forgetting in Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
Some experimental details seem to be missing. e.g., what is the public dataset that is used for experiments on CIFAR, CINIC, and FEMNIST?

Author response:
We kindly refer the reviewer to section 5.1 and B.1 in the appendix.

Reviewer comment (Experiments):
The comparison with other methods may not be fair as the proposed method leverages a shared public dataset in the server while compared methods may not use it. Some papers on FL and KD also use a public dataset. e.g., [1][2].  
[1] Ensemble distillation for robust model fusion in federated learning. NeurIPS’20  
[2] Performance optimization of federated person re-identification via benchmark analysis. ACMMM’20

Author response:
This is a valid concern. Indeed, for this reason we compare with [1]. Moreover, FedDF [1] uses the all of CIFAR100 as the public dataset, when experimenting with CIFAR10, while we only use 2.5% of CIFAR10 as the public dataset, and use the remaining 97.5% for the clients. This is a much smaller public dataset.  
Thanks for the reply!

Apologies for the confusion I caused by my comment. [1] in their own experiments, they used CIFAR100 as the public dataset, when training on CIFAR10. In our experiments, we used the same public dataset for all the methods that use a public dataset. So for both Flashback and FedDF [1] we used the exact same small fraction of CIFAR10 as the public dataset.

Reviewer comment (Experiments):
A straightforward baseline to consider is fine-tuning with the public dataset in the server, using soft labels from clients or ground truth labels. It would provide more insights into the significance of the proposed method. The reviewer would consider raising the rating if some of the concerns can be addressed.

Author response:
This a very good suggestion, we will add such an experiment in the revised manuscript.

Reviewer comment (Experiments):
What is the impact of different selection choices of public datasets? Would the method still work if the data distribution of the public dataset is different from the client’s data distribution?

Author response:
We experimented with a very small public dataset, we would report on additional experiments to answer this question in the revised manuscript.

Reviewer comment (Experiments):
It is unclear if the other baseline methods benefit from the public dataset. Their performance can improve if the server can train on the centralized public dataset as well.

Author response:
We include a comparison against FedDF, which uses a public dataset. FedDF has poorer performance compared to some other baselines (see Figure 3). This indicates that using a public dataset doesn’t necessarily give a performance boost.

Reviewer comment (Experiments):
The algorithm has two parts: local and global KD. An ablation study on each part needs to be included.

Author response:
We did an additional experiment that explored this question, and we found out that, using both is necessary. And doing KD on one side only doesn’t yield a good performance and training stability. This is experiment will be included in the revised manuscript. Check this image for the result: https://ibb.co/sWdp6JQ

Reviewer comment (Experiments):
One of the stated contributions of the paper is to "show how and where forgetting happens in FL". I'm not convinced this question is answered by the manuscript. In particular, only two possible causes are explored: local training and server aggregation. Other possible factors are not considered, e.g., the ordering of the clients, the ordering of the data in the clients [2]. I believe the manuscript should be more specific in this statement or, hopefully, perform a more systematic exploration of what really affects forgetting in FL.

Author response:
That is a very interesting point, and can be an additional dimension of exploring forgetting in FL. However, we believe that the first example is implicitly included in the **server forgetting,** where generally given a random order of clients of the rounds forgetting would occur, therefore, constructing a difficult ordering of clients will make the **server forgetting** more severe. This is not to say that this specific example does not need to be studied but to say that our characterization is more general. As an additional supporting material, we will update the manuscript to make this clear and potentially include experiments for other possible causes of forgetting.  We would appreciate it if the reviewer could give advice on how such an experiment can be set up. Regarding the 2nd example, client-local training essentially is a single training process of a deep learning model, therefore, any possible forgetting in a deep learning training can happen during the client local training, which makes it not specific to the intricacy of FL.

Reviewer comment (Presentation):
In the same line, the paper defines, measures and tests local forgetting. Later on, it concludes that some amount of it is necessary for learning. This conclusion is valuable, but this nuance is not reflected in the introduction nor in the motivation of the manuscript.

Author response:
Thank you for this feedback, we will improve the manuscript, to reflect that.

Reviewer comment (Experiments):
I am surprised that several baselines did not converge for FEMNIST in Table 1. This is a fairly simple benchmark that should achieve good performance with a CNN.

Author response:
We found that in general FL experiments are tricky and very sensitive to the experiment setup, such as client selection, and number of clients. Can you please refer us to similar experiments with the same setup, where FL algorithms converged to better results?

Reviewer comment (Presentation):
I found Figures 2 and 5 confusing. I'm not sure what the colors refer to, and why Flashback is performing better according to these figures. Please clarify.

Author response:
The green color is to distinguish Flashback from the other baselines. The intensity of the color in the heatmap reflects the accuracy. Flashback is performing better in Figure 2 because it shows less drop in accuracy (forgetting) from the first row (global model at round t-1) to the rows in the middle, which represents the accuracy of the clients that participated after their local training. Furthermore, from the rows in the middle to the last row (global model at round t) Flashback shows better aggregation results, with less forgetting than the other baselines.  
As for Figure 5, we show that Flashback is more consistent across rounds because it has fewer drops of accuracies of the global model per-class accuracy over all the rounds, which indicates forgetting. Thanks for the feedback, we will revise the figure and update the figures for more clarity in the revised manuscript

Reviewer comment (Experiments):
In line with the connections between distillation and algorithmic churn, and with other possible causes of forgetting, future versions of the manuscript would benefit from studying forgetting at the example level (see [2]) for a given test dataset at the server.

Author response:
That is a good suggestion thanks. But can the reviewer please clarify the question here?