Paper: CoSDA: Continual Source-Free Domain Adaptation (ICLR-2024)

Reviewer comment (Novelty):
Another concern pertains to the originality of CoSDA, as it appears to amalgamate various existing strategies, including the teacher-student model, mixup, and information maximization. While the use of the exponential moving average method is a common and sensible strategy to prevent overfitting, the rationale behind employing mixup and information maximization to tackle this issue isn't clear or intuitive.

Author response:
The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy. The rationale for choosing Mixup as augmentation and integrating information maximization is discussed in section 3.1 , appendix A.1 and A.2.

Reviewer comment (Presentation):
Lastly, the presentation of results in the tables is somewhat overwhelming and perplexing. The abundance of statistics, compounded by the use of multiple colors, makes it difficult to interpret the data and grasp the essential outcomes. Simplifying these tables for clarity and ease of understanding would be highly beneficial.

Author response:
Thanks for your advice. We will try to improve the presentation of results. Also thanks for pointing out typos.

Reviewer comment (Writing):
Some typos in this manuscript: 
1. "by by consolidating data"
2. "In this section, We ..."

Author response:
Thanks for your advice. We will try to improve the presentation of results. Also thanks for pointing out typos.

Reviewer comment (Novelty):
Although the studied new setting is of high practical value, and the experiments are abundant, the major concern is that the proposed method(s) is not new/novel, the proposed modules are quite popular in the related areas.

Author response:
**[W1]** The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy.

Reviewer comment (Novelty):
The proposed CoSDA approach lacks novelty and bears a *significant* similarity to CoTTA [1]. CoTTA tackles the closely related problem of test-time adaptation (TTA) and utilizes the exact same concept of a student-teacher framework trained using consistency regularization. CoSDA simply replaces the general augmentation set in CoTTA with mixup. The authors do compare with CoTTA and mention that "... CoTTA (Wang et al., 2022) ensures knowledge preservation by stochastically preserving a subset of the source model’s parameters during each update" but fail to mention this other crucial aspect of this paper which is directly related to their method. The inclusion of the mutual information regularization loss for better pseudo-labels is also borrowed directly from a previous SFDA approach SHOT [2]. Finally, the claimed student-teacher EMA framework is very common in the SFDA literature, [3, 4], none of which are mentioned in the paper.

Author response:
**[W1]** CoSDA is clearly different from CoTTA in three aspects:
1. The setting of continual test-time adaptation is not about catastrophic forgetting. It computes the average of results over multiple itrations. Whereas in our continual SFDA setting data from each domain is seen only once.
2. The dual-speed optimization strategy is the module designed for continual SFDA. While in CoTTA, both teacher model and student model are updated after each epoch.
3. CoTTA needs to inference $k$ times(typically $k=32$), while CoSDA only needs $k=1$.
We discuss knowledge distillation-based methods in related works section, which is more relative than EMA.

Reviewer comment (Novelty):
A primary goal of this work is to prevent catastrophic forgetting of previously seen domains, however, there is no discussion of existing continual learning approaches and what distinguishes this work from these papers.

Author response:
**[W2]** We do discuss Continual DA methods in the last part of related works section.

Reviewer comment (Experiments):
The experiments on sequential target domains (Sec 4.3) is limited, despite being a prominent claim of this paper: (a) the max number of domains is only 4, while TTA methods experiment with up to 15, and (b) no experiments on effects of a domain being repeated.

Author response:
**[W3]** The max number of domain is not 4, but 6. DomainNet has 6 domains, and the results are shown in Figure 2. The domains in the studied datasets have distinct covariate shift, while in continual test-time adaptation setting, all domains are augmented with 15 types of corruptions from the same images, therefore sharing great similarity. Furthermore, we implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:
| Method        | Mean |  
| ------------- | ---- |  
| Source        | 43.5 |  
| BN Adapt      | 20.4 |  
| Pseudo-label  | 19.8 |  
| TENT          | 18.6 |  
| CoTTA         | 16.2 |  
| CoSDA         | 15.1 |

Reviewer comment (Experiments):
Why was mixup chosen as the preferred augmentation? Are there experiments with other augmentations? The authors mention that mixup can be applied to other domains (NLP, Audio), but no evidence is provided to prove the efficacy of CoSDA on other domains, let alone more challenging tasks on images, such as semantic segmentation.

Author response:
**[Q1]** The rationale for choosing Mixup as augmentation is discussed in Section 3.1 and Appendix A.1. We have tried CutMix and the result is not as well.

Reviewer comment (Experiments):
How does the performance vary with the number of unlabeled samples per domain?

Author response:
**[Q2]** By default, all current SFDA works discussed in the paper train on the whole unlabeled dataset, so we follow the setting.

Reviewer comment (Experiments):
CoTTA utilizes a stochastic restore of the source weights since it tackles the more challenging TTA setup (performing adaptation with very few images) - was this removed in the experiments? Furthermore, how well does CoSDA perform on the TTA benchmarks?

Author response:
**[Q3]** The stochastic restore part is not removed, which can varified from our source code(in supplementary materials). We implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:
| Method        | Mean |  
| ------------- | ---- |  
| Source        | 43.5 |  
| BN Adapt      | 20.4 |  
| Pseudo-label  | 19.8 |  
| TENT          | 18.6 |  
| CoTTA         | 16.2 |  
| CoSDA         | 15.1 |

Reviewer comment (Novelty):
The technique contribution is somewhat limited. The proposed method can be easily derived from existing methods. For example, the idea of the teacher-student framework [1] and the mix-up strategy [2] for continual source-free domain adaptation is nothing new.

Author response:
The technique contribution is somewhat limited. The proposed method can be easily derived from existing methods. For example, the idea of the teacher-student framework[1] and the mix-up strategy[2] for continual source-free domain adaptation is nothing new.

Reviewer comment (Experiments):
There is a need for deeper experiments. Could the authors test their proposed method on a variety of adaptation sequences, especially those that are more complex, instead of relying solely on one fixed combination sequence? In real-world scenarios, sequential target data might span a longer duration and encompass a wider range of distributions. For instance, the authors could establish longer sequences to evaluate the method for mitigating forgetting.

Author response:
We implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:  
| Method        | Mean |  
| ------------- | ---- |  
| Source        | 43.5 |  
| BN Adapt      | 20.4 |  
| Pseudo-label  | 19.8 |  
| TENT          | 18.6 |  
| CoTTA         | 16.2 |  
| CoSDA         | 15.1 |

Reviewer comment (Experiments):
In continual source-free domain adaptation, a primary challenge lies in managing the tradeoff between adapting to the target domain and preventing the forgetting of previous domains. However, the proposed methods give limited attention to this challenge. The tradeoff is achieved through the EMA momentum m within a teacher-student learning framework in this paper. Thus, a hyper-parameter experiment of m should at least be conducted.

Author response:
The dual-speed optimization strategy is the module designed for continual SFDA. The student model captures short-term features and update to adapt to target domains, while the teacher model filters out long-term domain-invariant features. The EMA momentum does not have much impact, so we fix the parameter over all experiments. Specifically, we follow the settings in MoCo and BYOL and increase the momemtum from 0.9 to 0.99 using a cosine schedule, as stated at the end of Section 3. We will carry out experiments to demonstrate that afterwards. Thanks for your advice!

Reviewer comment (Theory):
I have concerns regarding the formulation of consistency loss which is the main contribution of the work. Equation 1 is confusing. From my understanding, h_{\psi}(\tilde{x}) should refer to the output logits from the student network. If so, minimizing the divergence between two distributions from different mathematical spaces does not make any sense. To be specific, \tilde{p} is a softmax probability vector with each element in the range [0,1], while h_{\psi}(\tilde{x}) has elements ranging from negative infinity to positive infinity.

Author response:
**[W1]** Both $\tilde{p}$ and $h_{\psi}(\tilde{x})$ are in the range [0, 1]. The hard-label from teacher is computed as $p := softmax(h_{\theta}(X)/\tau)$, while $h_{\psi}(\tilde{x})$ denotes logits normalized by softmax function.

Reviewer comment (Experiments):
Please add an ablation study to demonstrate the potential issue of the proposed consistency loss collapsing. My suspicion is that this collapse occurs due to the divergence between two distinct mathematical spaces.

Author response:
**[W2]** As stated above, there is no distinct divergence in mathematical spaces. Also, ablation study is already provided as results (-) MI. You can find collapse in Table1 between columns "CoSDA" and "CoSDA (-) MI".

Reviewer comment (Theory):
In batch normalization, the mean and variance are computed based on the activations within each mini-batch, which is a subset of the entire dataset. The statistics are calculated separately for each mini-batch as the model processes the data during training. This is what allows batch normalization to adapt to the statistics of the current batch and helps in stabilizing and accelerating training. However, the authors mentioned that their mean and variance are calculated based on the whole dataset for batch normalization, which does not make sense. As per my understanding, the moving average mean and variance are computed by accumulating the mean and variance from each batch using an exponential moving average formula (\alpha*\mu_{moving} + (1-\alpha)*\mu_{i}). The moving average mean and variance are not the mean and variance of the whole dataset. The lack of clarity on batch normalization in the paper raises concerns about its overall quality.

Author response:
**[W3]** Yes, the moving average mean and variance are not the mean and variance of the whole dataset. They can be viewed as weighted mean, instead. As the adaptation process goes on, the mean and variance should be more and more precise, so later batches should have higher weight.

Reviewer comment (Evaluation):
I may have a misunderstanding regarding the evaluation settings; however, based on my interpretation of the paper, it appears that the proposed method does not demonstrate significant improvements over the baseline approaches in terms of target domain classification accuracies. I kindly request the authors to provide a more detailed explanation or consider revising the experiment section for improved clarity on this matter.

Author response:
**[W4]** CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. For example, as reviewer nYkW metioned, comparing with EdgeMix in table1, in terms of the accuracy drop, our method is clearly at least 3x smaller than EdgeMix. Combined with other modules, our method lowers about 2x accuracy drop on source domain with neglectable decrease in adaptation accuracy.

Reviewer comment (Novelty):
The proposed work appears to be closely aligned with federated learning. I am curious about the authors' motivation for framing this work as a sequence of source-free domain adaptation. To me, the workflow seems to have stronger connections to federated learning rather than source-free domain adaptation. Thus, the literature on federated learning should be given.

Author response:
**[Minors]** The core idea of our paper is that CoSDA achieves "continual SFDA", while in federated setting, the predictions from multiple clients are gathered and averaged without the concept of "continual". Thanks for pointing out our typos.

Reviewer comment (Writing):
Typos: 3.1 “to consist with” -> “to be consistent with”

Author response:
**[Minors]** The core idea of our paper is that CoSDA achieves "continual SFDA", while in federated setting, the predictions from multiple clients are gathered and averaged without the concept of "continual". Thanks for pointing out our typos.

Reviewer comment (Writing):
In the second paragraph of the introduction, the authors mentioned that “SFDA also allows for spatio-temporal separation of the adaptation process since the model training on source domain is independent of the knowledge transfer on target domain”. What does the spatio-temporal separation refer to?

Author response:
**[Q1]** During each adaptation process, we have only unlabeled data from current domain. Neither data nor weights from previous domains are stored. Therefore, each adaptation process can be performed at different time and location, which we refer to as spatio-temporal separation.

Reviewer comment (Theory):
In Equation 1, is h_{\psi}(\tilde{x}) referring to the logits or the softmax probabilities? If it represents the logits, it raises a question regarding the use of KL-divergence between two distributions: \tilde{p}, which is a softmax probability vector with each element in the range [0,1], and h_{\psi}(\tilde{x}), which has elements ranging from negative infinity to positive infinity.

Author response:
**[Q2]** $h_{\psi}(\tilde{x})$ refers to softmax probabilitie.

Reviewer comment (Evaluation):
I do not quite understand the in-sequence evaluation settings. From the result tables, the authors listed, most baselines perform better without the proposed methods. Then, how could the readers evaluate the effectiveness of the proposed methods?

Author response:
**[Q3]** CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. Ideally, adaptation accuracy should be as high as possible, while accuracy drop as small as possible.

Reviewer comment (Novelty):
The proposed method lacks novelty in terms of the framework design. For example, the teacher-student architecture has been proposed from the self-supervised learning framework, e.g., MoCo. The mix-up is another already invented technique to augment both the input space and the label space. The consistent loss has been utilized in those above mentioned methods already. Meanwhile, the mutual information maximization is a widely applied regularization technique during representation learning. Combining all, I think the novelty of the design is hard to justify.

Author response:
**[W1]** The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy.

Reviewer comment (Theory):
The paper claims continual SFDA, where from the method design, there is no specific module designed to deal with the model catastrophic forgetting issue, except accepting the teacher model’s hard label to measure the KL divergence from the student prediction on the mixed up sample. The teacher model is leveraging exponential averaging, but the student model is not in any manner distilled from the teacher model. The model weights forgetting is not addressed in any technical design.

Author response:
**[W2]** The dual-speed optimization strategy is the module designed for continual SFDA. The student model captures short-term features and update to adapt to target domains, while the teacher model filters out long-term domain-invariant features.

Reviewer comment (Evaluation):
Across the compared methods, the proposed CoSDA does not show advantageous results over the other methods. For example, in Table 1, CosDA is not compellingly better than EdgeMix, where actually many of the DA protocols EdgeMix shows better results. In Table 2, CosDA even combined with some other modules, such as NRC or AaD, is not better than AaD. This suggests that the proposed way does not achieve as advantageous performance as the state-of-the-art methods.

Author response:
**[W3]** CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. In your example, in terms of the accuracy drop, our method is clearly at least 3x smaller than EdgeMix. Combined with other modules, our method lowers about 2x accuracy drop on source domain with neglectable decrease in adaptation accuracy.