Paper: ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (ICLR-2024)

Reviewer comment (Reproducibility):
The authors mentioned that there is not any non-linear layer in ViDA. It is not clear what they mean by that. There is no activation? How are they implementing this network? The authors do mention that there is a direct linear projection of ViDA; however, I would like to understand the relationship of the layers within the model.

Author response:
**Q1. The implementation of ViDA**

Thank you for your insightful comments. As illustrated in Figure 2 of the submission, there are three sub-branches. The single linear layer in the middle branch is derived from the original network, while the right and left branches with bottleneck structures, representing the high-rank ViDA and low-rank ViDA, respectively.
In both ViDA branches, two linear layers (fully connected layers) handle the transformation of feature dimensionality, **and there are no activation layers, such as ReLU and Leaky ReLU**.  For example, in the high-rank branch, we first employ the up-projection layer with parameters $W^h_{up} \in R^{d \times d_{h}}$, which elevates the feature dimensionality from $B \times  N \times  d$ to $B \times  N \times  d_{h}$, where the middle dimension of the high-rank feature satisfies $d_{h} \geq d$ (e.g., $d_{h} = 128$). Next, we use the down-projection layer with parameters $W^h_{down} \in \mathbb{R}^{d_h \times d}$ to restore the feature dimensionality from $B \times N \times d_h$ to $B \times N \times d$, where B is the batch size and N is the sequence length.

For the relationship between ViDAs and the original network, with identical inputs and no non-linear operations, the three branches exhibit a linear relationship. For example, let Wv and Wo denote the parameter matrices for ViDA and the linear layer of the original model, respectively. Since they share the same input x, their outputs can be expressed as Y1 = Wv ⋅ x and Y2 = Wo ⋅ x, where ⋅ denotes matrix multiplication. Combining these outputs yields: Y = Y1 + Y2 = (Wv + Wo) ⋅ x. It is evident that this combination remains linear property, involving only matrix multiplication and addition operations without introducing non-linear transformations.

Reviewer comment (Experiments):
How did the authors decide parameter 𝜭 in equation 5? Was it decided empirically?

Author response:
**Q2. The parameter 𝜭 in equation 5**

𝜭 is an empirical parameter employed to regulate the adjustment of $\lambda_{h}$ and $\lambda_{l}$ within the Homeostatic Knowledge Allotment (HKA) strategy. Utilizing statistical measures, we have computed an average uncertainty value of approximately 0.2. Consequently, we configure 𝜭 to be 0.2 to enhance the discrimination between samples exhibiting substantial distribution shifts and those with minor distribution shifts. We will add this detailed explanation in the revised version.

Reviewer comment (Reproducibility):
In what sense there is not any non-linear layer in ViDA? Is there no activation? How are you implementing this network? Is there a direct linear projection of ViDA? It would be important to understand the relationship of the layers within the model.

Author response:
**Q1. The implementation of ViDA**

Thank you for your insightful comments. As illustrated in Figure 2 of the submission, there are three sub-branches. The single linear layer in the middle branch is derived from the original network, while the right and left branches with bottleneck structures, representing the high-rank ViDA and low-rank ViDA, respectively.
In both ViDA branches, two linear layers (fully connected layers) handle the transformation of feature dimensionality, **and there are no activation layers, such as ReLU and Leaky ReLU**.  For example, in the high-rank branch, we first employ the up-projection layer with parameters $W^h_{up} \in R^{d \times d_{h}}$, which elevates the feature dimensionality from $B \times  N \times  d$ to $B \times  N \times  d_{h}$, where the middle dimension of the high-rank feature satisfies $d_{h} \geq d$ (e.g., $d_{h} = 128$). Next, we use the down-projection layer with parameters $W^h_{down} \in \mathbb{R}^{d_h \times d}$ to restore the feature dimensionality from $B \times N \times d_h$ to $B \times N \times d$, where B is the batch size and N is the sequence length.

For the relationship between ViDAs and the original network, with identical inputs and no non-linear operations, the three branches exhibit a linear relationship. For example, let Wv and Wo denote the parameter matrices for ViDA and the linear layer of the original model, respectively. Since they share the same input x, their outputs can be expressed as Y1 = Wv ⋅ x and Y2 = Wo ⋅ x, where ⋅ denotes matrix multiplication. Combining these outputs yields: Y = Y1 + Y2 = (Wv + Wo) ⋅ x. It is evident that this combination remains linear property, involving only matrix multiplication and addition operations without introducing non-linear transformations.

Reviewer comment (Reproducibility):
How was the parameter 𝜭 in equation 5 set?

Author response:
**Q2. The parameter 𝜭 in equation 5**

𝜭 is an empirical parameter employed to regulate the adjustment of $\lambda_{h}$ and $\lambda_{l}$ within the Homeostatic Knowledge Allotment (HKA) strategy. Utilizing statistical measures, we have computed an average uncertainty value of approximately 0.2. Consequently, we configure 𝜭 to be 0.2 to enhance the discrimination between samples exhibiting substantial distribution shifts and those with minor distribution shifts. We will add this detailed explanation in the revised version.

Reviewer comment (Presentation):
I suggest incorporating details such as the ablation study on homeostatic knowledge allotment from the appendix to the main paper.

Author response:
**Q3. incorporating the details from the appendix to the main paper.**

Thank you for your valuable comments. We will move the ablation study on HKA from the appendix to the main paper. We have made modifications in the **updated PDF paper, specifically in Table 6 and Section 4.6, both marked in blue font**.

Reviewer comment (Theory):
There is no theoretical justification or an intuitive explanation for why the low-rank adapter acquires domain-agnostic information, whereas the high-rank adapter collects domain-specific knowledge other than experimental observation.

Author response:
**Q1. More intuitions of the low-rank adapter and high-rank adapter**

Thank you for the constructive advice. To further validate the distinct domain representations of low-rank and high-rank ViDA , we conduct an additional experiment to invert the scale factors within the Homeostatic Knowledge Allotment (HKA) strategy. Specifically, for samples exhibiting high uncertainty, we reduce $\lambda_{h}$ while increasing $\lambda_{l}$. As shown in the Table below and Table 12 of the Appendix, building upon the ablation study of the submission, we integrate the **Inversed HKA** approach into Ex4, which already incorporates both low-rank and high-rank ViDAs. This adaptation yields an average error rate of 46.3 on ImageNet-C, marking a slight 0.7% and 2.9% error rate increase compared to Ex4 and Ex5, respectively. This experiment underscores how the proposed HKA strategy promotes the different domain representations between low-rank and high-rank VIDA models. 

|   | Contributions |   Average error rate |
| --- | --- | --- |
| Ex4 | ViDAh + ViDAl | 45.6 |
| Ex5 | ViDAh + ViDAl + HKA | 43.4 |
| **Inversed HKA** | ViDAh + ViDAl + Inversed HKA | **46.3** |

To provide more straightforward verification, we extend our analysis by incorporating the qualitative analysis of Class Activation Mapping (CAM) in Figure 4 of the submission. Besides, we supplement and visualize the CAM of our proposed ViDAs and Inversed HKA in **Figure 4 of the updated PDF paper**. Meanwhile, we incorporate relevant explanations, marked in blue font. The low-rank ViDA is inclined to put more weight on the foreground sample while tending to disregard background noise shifts. This indicates that the low-rank ViDA attends to locations with more general and task-relevant information. The high-rank ViDA allocates more attention to locations characterized by substantial domain shift, encompassing the entirety of the input images. This behavior aligns with the high-rank branch's tendency to fit global information and predominantly extract domain-specific knowledge from the target domain data. Our proposed ViDAs exhibit higher response values on foreground samples, showcasing improved feature representation in target domains. Conversely, the Inversed HKA displays inferior feature attention, concentrating solely on specific foreground samples and failing to encompass complete semantic information. These visualizations, combined with previous results, jointly validate the intuition that the high-rank adapter focuses on domain-specific knowledge, while the low-rank adapter emphasizes domain-shared knowledge.

Reviewer comment (Experiments):
What about the performance of the proposed approach CIFAR100-to-CIFAR100C method for ResNeXt-29 architecture?

Author response:
**Q2. CIFAR100-to-CIFAR100C experiment with ResNeXt-29 architecture**

Thank you for your comprehensive feedback. The following table presents the results of our CIFAR100-to-CIFAR100C CTTA experiment with ResNeXt-29 as the backbone. Our approach substantially reduces the error rate by 15.7% compared to the source model and achieves competitive performance compared with other CTTA methods. We have supplemented the table and incorporated relevant explanations in **Appendix Section C.5, marked in blue font**.
Combining Table 1 and Table 13 in the submission, we have validated the effectiveness of our method on three classification CTTA benchmarks using the same CNN backbones as CoTTA (Wang et al., 2022). 

| Method(ResNeXt-29) |   Source | BN adapt  | Tent  | CoTTA | Ours |
| --- | --- | --- | --- | --- | --- |
| Average error rate |   46.4 | 35.4 | 60.9 | 32.5 | 30.7(+15.7%) |

Reviewer comment (Reproducibility):
Using the source data, retrain the "model added with low/high-rank adapters" for a few stages. Without access to the source domain data, it is impossible to employ off-the-shelf pre-trained models.

Author response:
**Q3. Pre-train on source data**

Pre-training the low-rank and high-rank ViDAs using source data is an unnecessary step and does not compromise the effectiveness of our approach. ViDAs can demonstrate comparable CTTA performance when they have a relatively stable initial parameter. As illustrated in the following table, we conducted an additional experiment on the Cityscape-to-ACDC scenario. ViDAs with random initial parameters and ViDAs with parameters pre-trained on ImageNet achieved 60.5 and 61.4 mIoU in target domains, respectively, exhibiting notable improvements compared to previous methods.
To better showcase the practicality, we have included the aforementioned experiments with other parameter initialization techniques in **Appendix C.4, along with corresponding analyses highlighted in blue font**.

|  | Adapter pretrain | Fog | Night | Rain | Snow | Mean (IoU) |
| --- | --- | --- | --- | --- | --- | --- |
| Source [58] | - | 69.1 | 40.3 | 59.7 | 57.8 | 56.7 |
| CoTTA [57] | - | 70.9 | 41.2 | 62.4 | 59.7 | 58.6 |
| Ours | Source | **71.6** | 43.2 | **66.0** | 63.4 | 61.1 |
| Ours | Random initial | **71.6** | 43.6 | 64.9 | 61.9 | 60.5 |
| Ours | ImageNet | **71.6** | **44.3** | **66.0** | **63.5** | **61.4** |

Reviewer comment (Theory):
There is no theoretical justification, nor even an intuitive explanation, for why the low-rank adapter acquires domain-agnostic information whereas the high-rank adapter collects domain-specific knowledge other than experimental observation.

Author response:
**Q1. More intuitions of the low-rank adapter and high-rank adapter**

Thank you for the constructive advice. To further validate the distinct domain representations of low-rank and high-rank ViDA , we conduct an additional experiment to invert the scale factors within the Homeostatic Knowledge Allotment (HKA) strategy. Specifically, for samples exhibiting high uncertainty, we reduce $\lambda_{h}$ while increasing $\lambda_{l}$. As shown in the Table below and Table 12 of the Appendix, building upon the ablation study of the submission, we integrate the **Inversed HKA** approach into Ex4, which already incorporates both low-rank and high-rank ViDAs. This adaptation yields an average error rate of 46.3 on ImageNet-C, marking a slight 0.7% and 2.9% error rate increase compared to Ex4 and Ex5, respectively. This experiment underscores how the proposed HKA strategy promotes the different domain representations between low-rank and high-rank VIDA models. 

|   | Contributions |   Average error rate |
| --- | --- | --- |
| Ex4 | ViDAh + ViDAl | 45.6 |
| Ex5 | ViDAh + ViDAl + HKA | 43.4 |
| **Inversed HKA** | ViDAh + ViDAl + Inversed HKA | **46.3** |

To provide more straightforward verification, we extend our analysis by incorporating the qualitative analysis of Class Activation Mapping (CAM) in Figure 4 of the submission. Besides, we supplement and visualize the CAM of our proposed ViDAs and Inversed HKA in **Figure 4 of the updated PDF paper**. Meanwhile, we incorporate relevant explanations, marked in blue font. The low-rank ViDA is inclined to put more weight on the foreground sample while tending to disregard background noise shifts. This indicates that the low-rank ViDA attends to locations with more general and task-relevant information. The high-rank ViDA allocates more attention to locations characterized by substantial domain shift, encompassing the entirety of the input images. This behavior aligns with the high-rank branch's tendency to fit global information and predominantly extract domain-specific knowledge from the target domain data. Our proposed ViDAs exhibit higher response values on foreground samples, showcasing improved feature representation in target domains. Conversely, the Inversed HKA displays inferior feature attention, concentrating solely on specific foreground samples and failing to encompass complete semantic information. These visualizations, combined with previous results, jointly validate the intuition that the high-rank adapter focuses on domain-specific knowledge, while the low-rank adapter emphasizes domain-shared knowledge.

Reviewer comment (Experiments):
What about the performance of the proposed approach CIFAR100-to-CIFAR100C method for ResNeXt-29 architecture?

Author response:
**Q2. CIFAR100-to-CIFAR100C experiment with ResNeXt-29 architecture**

Thank you for your comprehensive feedback. The following table presents the results of our CIFAR100-to-CIFAR100C CTTA experiment with ResNeXt-29 as the backbone. Our approach substantially reduces the error rate by 15.7% compared to the source model and achieves competitive performance compared with other CTTA methods. We have supplemented the table and incorporated relevant explanations in **Appendix Section C.5, marked in blue font**.
Combining Table 1 and Table 13 in the submission, we have validated the effectiveness of our method on three classification CTTA benchmarks using the same CNN backbones as CoTTA (Wang et al., 2022). 

| Method(ResNeXt-29) |   Source | BN adapt  | Tent  | CoTTA | Ours |
| --- | --- | --- | --- | --- | --- |
| Average error rate |   46.4 | 35.4 | 60.9 | 32.5 | 30.7(+15.7%) |

Reviewer comment (Experiments):
Using the source data, retrain the "model added with low/high-rank adapters" for a few steps. Without access to the source domain data, it is impossible to employ off-the-shelf pre-trained models.

Author response:
**Q3. Pre-train on source data**

Pre-training the low-rank and high-rank ViDAs using source data is an unnecessary step and does not compromise the effectiveness of our approach. ViDAs can demonstrate comparable CTTA performance when they have a relatively stable initial parameter. As illustrated in the following table, we conducted an additional experiment on the Cityscape-to-ACDC scenario. ViDAs with random initial parameters and ViDAs with parameters pre-trained on ImageNet achieved 60.5 and 61.4 mIoU in target domains, respectively, exhibiting notable improvements compared to previous methods.
To better showcase the practicality, we have included the aforementioned experiments with other parameter initialization techniques in **Appendix C.4, along with corresponding analyses highlighted in blue font**.

|  | Adapter pretrain | Fog | Night | Rain | Snow | Mean (IoU) |
| --- | --- | --- | --- | --- | --- | --- |
| Source [58] | - | 69.1 | 40.3 | 59.7 | 57.8 | 56.7 |
| CoTTA [57] | - | 70.9 | 41.2 | 62.4 | 59.7 | 58.6 |
| Ours | Source | **71.6** | 43.2 | **66.0** | 63.4 | 61.1 |
| Ours | Random initial | **71.6** | 43.6 | 64.9 | 61.9 | 60.5 |
| Ours | ImageNet | **71.6** | **44.3** | **66.0** | **63.5** | **61.4** |

Reviewer comment (Novelty):
My main concern is the novelty. Separating the extracted features into domain-invariant and domain-specific components has been extensively explored in previous domain adaptation and domain generalization methods. It lacks clear pieces of evidence to show why the proposed method is preferable in the context of test-time adaptation.

Author response:
**Q1. Clear pieces of evidence**

Thank you for your patient response. First, we would like to emphasize the novelty and unique design of our method for the Continual Test-Time Adaptation (CTTA) task. The CTTA task differs from previous domain adaptation (DA) and domain generalization (DG) tasks in that it requires simultaneously addressing both error accumulation and catastrophic forgetting problems while striking a balance between them. Classical DA methods mainly focus on aligning features between the source domain and target domain to extract domain-invariant knowledge. Meanwhile, classical DG methods utilize meta-learning or feature alignment schemes to absorb domain-invariant information from source data (Zhou et al., 2021; Li et al., 2017). However, for CTTA tasks, practical constraints prohibit access to source domain data. This significantly increases the difficulty of extracting various domain knowledge. To this end, we propose the Homeostatic low-rank and high-rank adapters to efficiently extract and manage both domain-specific and domain-shared knowledge, addressing the specific error accumulation and catastrophic forgetting problems in CTTA. Meanwhile, the CTTA task places a strong emphasis on adaptation efficiency since each sample is only observed once. Therefore, our proposed method enables the efficient extraction of different domain knowledge using efficient adapter parameters, without compromising the model plasticity. This stands in contrast to absorbing knowledge through cumbersome model parameters. To further address the CTTA challenges and improve its efficiency, we introduce the Homeostatic Knowledge Allotment (HKA) strategy based on the distribution of each sample. This strategy adaptively combines diverse domain knowledge from each ViDA, making the most effective use of every opportunity to encounter each target sample.

To comprehensively showcase the effectiveness of the proposed method in the context of CTTA, we systematically validate each contribution step by step. Initially, we conduct an ImageNet-to-ImageNet-C CTTA experiment utilizing a combination of two high-rank adapters or two low-rank adapters, as elaborated in Appendix Table 7. To ensure fairness, these experiments were conducted without implementing the HKA strategy and with an equal number of adapters as our proposed ViDAs. Remarkably, the low-rank adapters (Ex2) consistently demonstrated lower long-term error rates compared to both the source model and the two high-rank adapters. This can be attributed to the inclination of the low-rank ViDAs to learn general information and alleviate catastrophic forgetting in the CTTA problem. On the other hand, the performance of the high-rank adapters initially surpassed our approach (Ex4) in the early stages, covering the first few target domains. However, a noticeable degradation in performance became evident in later target domains. This observation highlights a crucial finding: while high-rank ViDAs enhance domain-specific knowledge acquisition and efficient cross-domain learning during the initial phases of CTTA, they simultaneously exacerbate catastrophic forgetting throughout the entire adaptation process. 

Then, we validate the effectiveness of the HKA strategy in the CTTA task. We conduct an additional experiment to invert the scale factors within the HKA strategy. Specifically, for samples exhibiting high uncertainty, we reduce $\lambda_{h}$ of the high-rank adapter while increasing $\lambda_{l}$ of the low-rank adapter, employing a strategy opposite to that in the submission. As shown in the following table, building upon the ablation study (Table 6) of the main paper, we integrate the Inversed HKA approach into Ex4, which already incorporates both low-rank and high-rank ViDAs. This adaptation yielded an average error rate of 46.3 on ImageNet-C, marking a 2.9% error rate increase compared to Ex5. This experiment highlights how the proposed HKA strategy promotes different domain representations between low-rank and high-rank VIDAs and can more efficiently address the CTTA task.
|   | Contributions |   Average error rate |
| --- | --- | --- |
| Ex4 | ViDAh + ViDAl | 45.6 |
| Ex5 | ViDAh + ViDAl + HKA | 43.4 |
| **Inversed HKA** | ViDAh + ViDAl + Inversed HKA | **46.3** |

Reviewer comment (Evaluation):
The experimental results, as illustrated in Table 1, indicate that the improvements achieved by the proposed method in comparison to previous approaches, such as CoTTA, are marginal.

Author response:
**Q2. The experiment results of Table 1**

First, as shown in Table 1 of the submission, our approach achieves an 11.4% improvement in classification accuracy compared to CoTTA when ViT-base is used as the backbone. These results indicate that our approach is more effective when ViT is employed as the backbone, leveraging its enhanced global context awareness. ViT, with its powerful self-attention mechanism, can adaptively comprehend the domain-specific and domain-shared knowledge extracted by our low-rank and high-rank ViDA.

Second, utilizing ResNet50 as the backbone, our method outperforms the CoTTA approach by 1.5% in classification accuracy. Although the performance enhancement may not be as remarkable as observed with ViT as the backbone, other methods applied to ResNet50 also demonstrate marginal improvements. Meanwhile, our method is also a memory-efficient CTTA approach. In contrast to CoTTA, the updated ViDA parameters can be reduced to 7.6% of the updated CoTTA parameters, significantly reducing computational costs in the continual adaptation process.

Reviewer comment (Experiments):
The section of the paper discussing large models, as well as the associated experiments, appear incomplete and somewhat abrupt.

Author response:
**Q3. Large model experiments**

Thank you for your comprehensive comments. Examination of Tables 2 and 3 in the submission reveals a significant increase in the average error rate, from 28.2% to 39.3%, with the pre-trained encoder parameters of SAM. This is attributed to SAM, being a pixel-level foundational model, encountering limitations when transferred to image-level classification tasks.
To support this observation, we conducted segmentation CTTA using SAM pre-trained parameters on the Cityscapes-to-ACDC scenario. It's worth noting that Segformer (Xie et al., 2021), utilized in the primary experiments (Table 4 of the submission), lacks positional encoding. Therefore, we adopt the SETR model (Zheng et al., 2021) as our new baseline, incorporating SAM's pre-trained parameters. As shown in the table below or Table 9 of the Appendix, our approach with SAM pre-trained parameters outperforms other methods in the ACDC target domains. This affirms our hypothesis: SAM, functioning as a pixel-level foundational model, excels in capturing fine-grained feature representations in dense CTTA tasks. We plan to further refine experiments with larger models in the revised version.

|  | Pretrained | Fog | Night | Rain | Snow | Mean (IoU) |
| --- | --- | --- | --- | --- | --- | --- |
| Source | Source model | 	72.6|	43.1|	63.0|	64.3|	60.8|
| Source | SAM | 74.8|	44.1|	66.7|	66.6|	63.0 |
| Cotta | SAM | 75.4|	45.9|	67.3|	68.7|	64.3|
| Ours | SAM | **76.5** | **47.2** | **68.1** | **70.7** | **65.6** |

Reviewer comment (Evaluation):
Observing the t-SNE plots in the low-rank branch (Figure 1), it appears that the nighttime representation in the ACDC dataset is not aligning as a domain-shared characteristic. I think nighttime data is not clustering in the low-rank space, which may indicate unique domain characteristics. Quantitative analysis with clustering metrics could clarify this.

Author response:
**Q1. Quantitative analysis of domain distribution distances within the ACDC dataset**

Thank you for the constructive advice. We conduct a quantitative analysis of the domain distribution distances within the ACDC dataset when comparing low-rank and high-rank representations. Specifically, following the k-means clustering metrics (MacQueen, 1967), we first calculate the Euclidean distance clustering center in every target domain by considering all samples within each domain.Therefore, in the ACDC dataset, we obtain four domain clustering centers and use these centers to represent each target domain distribution. Using the same calculation process, we compute the distances for the source model, low-rank branch, and high-rank branch, utilizing features from the third transformer block.The quantitatively normalized results are presented in the table below, showing in the order of continual adaptation (Fog → Night → Rain → Snow).

| **Distance** | **Source model** | **Low-rank branch** | **High-rank branch** |
| --- | --- | --- | --- |
| **Fog-Night** | **0.73** | **0.29** | **0.98** |
| **Night-Rain** | **0.62** | **0.23** | **0.65** |
| **Rain-Snow** | **0.51** | **0.10** | **0.80** |
| **Snow-Fog** | **0.38** | **0.13** | **0.47** |
| **Mean** | **0.56** | **0.19** | **0.73** |

Based on the experimental results, we observe that the Low-rank adapter significantly reduces the domain distribution distance in the ACDC dataset. Especially in the Fog-Night and Night-Rain scenarios, the distance decreased from 0.73 to 0.29 and from 0.62 to 0.23, respectively. Compared to the Source model, the Low-rank adapter noticeably aligns the feature distribution between different domains, showing a relatively domain-shared representation. In the case of the high-rank adapter, there is a larger distance between domains. In conjunction with the lower intra-class divergence presented in Figure 3(b) of the submission, it is evident that the high-rank adapter effectively facilitates domain-specific knowledge extraction in each target domain.

Reviewer comment (Evaluation):
In Figure 3(a), the inter-domain divergence does not seem to exhibit significant differences between high-rank and low-rank features across conditions c1 to c9. Does this imply that the effectiveness of the rank-based approach is sensitive to the type of corruption present in the data? Understanding this could inform the robustness of the proposed method across a wider array of scenarios.

Author response:
**Q2. The factors influencing the effectiveness of the rank-based approach.**

In Continual Test-Time Adaptation (CTTA), two main factors influence the effectiveness of low-rank and high-rank: **1)** different target domain distributions (types of corruption), and **2)** the length of continual adaptation. To explore the impact of corruption types, as demonstrated in Appendix Table 7, we conduct an ImageNet-to-ImageNet-C CTTA experiment using either high-rank adapters or low-rank adapters. To ensure fairness, these experiments were carried out without implementing the Homeostatic Knowledge Allotment (HKA) strategy and employing an equal number of adapters as our proposed ViDAs. Across all corruption types, both the low-rank adapters (Ex2), high-rank adapter (Ex3), and our ViDAs (Ex4) outperformed the baseline source model's classification accuracy (Ex1), confirming the robustness and effectiveness of our contributions. However, for various target domains, the classification capability and feature representation of low-rank and high-rank adapters indeed differ. For instance, there is a significant difference among them regarding major target corruptions such as brightness, snow, frost, fog, and contrast, whereas the distinction is less pronounced for a few target corruptions like pixelate and Jpeg.

On the other hand, the impact on the effectiveness of the rank-based approach also arises from the length of continual adaptation. As illustrated in Appendix Figure 5 (b), we conducted a 10-round CTTA experiment on ImageNet-to-ImageNet-C, simulating a long-term adaptation scenario by repeating 10 rounds of 15 corruption sequences in ImageNet-C. Remarkably, the high-rank ViDA achieves competitive results compared to other methods during the initial 1 to 3 rounds, demonstrating the capacity of high-rank features to efficiently learn target domain-specific knowledge. However, an increase in error rates becomes evident during the later rounds (rounds 5 to 10). These results validate the potential for encountering catastrophic forgetting when exclusively focusing on domain-specific knowledge. In contrast, the performance of the low-rank ViDA remains consistently robust throughout the continual adaptation process, verifying it concentrates more on extracting domain-shared knowledge and effectively preventing the catastrophic forgetting problem. This highlights the distinct effectiveness of the two types of adapters at different continual adaptation stages. 

Finally, as shown in the table below, we extend the inter-domain divergence in Figure 3 of the submission. In the first round, the average inter-domain divergence for high-rank adapters is 0.34, whereas for low-rank adapters, it is 0.30. Moving to the second round, the high-rank adapter and low-rank adapter achieve average inter-domain divergences of 0.37 and 0.29, respectively. In summary, with the increase in the length of continual adaptation, the gap in inter-domain divergence between high-rank and low-rank features also widens.

| **Inter-Domain Div (mean)** | **Low-rank adapter** | **High-rank adapter** |
| --- | --- | --- |
| **1 Round** | **0.30** | **0.34** |
| **2 Round** | **0.29** | **0.37** |
| **3 Round** | **0.30** | **0.42** |

In conclusion, the absence of notable inter-domain divergence among conditions c1 to c9 can be ascribed to the combined influence of the corruption types and their early stage in continual adaptation.

Reviewer comment (Experiments):
Regarding Figure 1, it would be insightful to include the t-SNE results for each transformer block. Such detailed visualizations would help to understand feature spaces across the different layers of the network.

Author response:
**Q3. The t-SNE results for each transformer block**

We supplement and visualize the t-SNE results for each transformer block in **Appendix Figure 5 (a) of the updated PDF paper.** Meanwhile, we incorporate corresponding explanations, marked in blue font. The visualization revealed that the qualitative results obtained from different layers (i.e., transformer blocks 1, 2, and 4) of the Segformer-B5 model exhibit similar distribution representations. In detail, the low-rank ViDA effectively reduces the distribution distance across different target domains, indicating its focus on extracting task-relevant knowledge. On the other hand, the high-rank ViDA exhibits notable distribution discrepancies among the various target domains, indicating its focus on extracting domain-specific knowledge.

Reviewer comment (Evaluation):
Since t-SNE visualizations represent relative rather than absolute distances, I am curious about the quantitative differences in domain distribution distances within the ACDC dataset when comparing low-rank to high-rank representations. Quantifying this difference could bolster the demonstration of your method's efficacy in managing domain shifts.

Author response:
**Q1. Quantitative analysis of domain distribution distances within the ACDC dataset**

Thank you for the constructive advice. We conduct a quantitative analysis of the domain distribution distances within the ACDC dataset when comparing low-rank and high-rank representations. Specifically, following the k-means clustering metrics (MacQueen, 1967), we first calculate the Euclidean distance clustering center in every target domain by considering all samples within each domain.Therefore, in the ACDC dataset, we obtain four domain clustering centers and use these centers to represent each target domain distribution. Using the same calculation process, we compute the distances for the source model, low-rank branch, and high-rank branch, utilizing features from the third transformer block.The quantitatively normalized results are presented in the table below, showing in the order of continual adaptation (Fog → Night → Rain → Snow).

| **Distance** | **Source model** | **Low-rank branch** | **High-rank branch** |
| --- | --- | --- | --- |
| **Fog-Night** | **0.73** | **0.29** | **0.98** |
| **Night-Rain** | **0.62** | **0.23** | **0.65** |
| **Rain-Snow** | **0.51** | **0.10** | **0.80** |
| **Snow-Fog** | **0.38** | **0.13** | **0.47** |
| **Mean** | **0.56** | **0.19** | **0.73** |

Based on the experimental results, we observe that the Low-rank adapter significantly reduces the domain distribution distance in the ACDC dataset. Especially in the Fog-Night and Night-Rain scenarios, the distance decreased from 0.73 to 0.29 and from 0.62 to 0.23, respectively. Compared to the Source model, the Low-rank adapter noticeably aligns the feature distribution between different domains, showing a relatively domain-shared representation. In the case of the high-rank adapter, there is a larger distance between domains. In conjunction with the lower intra-class divergence presented in Figure 3(b) of the submission, it is evident that the high-rank adapter effectively facilitates domain-specific knowledge extraction in each target domain.