Paper: Test-time Adaptation against Multi-modal Reliability Bias (ICLR-2024)

Reviewer comment (Reproducibility):
The paper lacks some experimental details, such as clarifying the terms “Statical” and “Dynamic” in Table 1 and explaining the distinctions between the proposed Self-Adaptive Attention and traditional self-attention. It is unclear what these terms mean and how they affect the performance of different methods. The paper should provide more definitions and discussions on these terms. Moreover, the paper should elaborate on how the Self-Adaptive Attention differs from the conventional self-attention in terms of design, implementation, and advantages.

Author response:
**A1.1:** We apologize for any confusion regarding these terms. Allow us to elaborate:

**"Statical" and "Dynamic"**. "Statical" denotes the state where the layer normalization (LN) and attention-based fusion (AF) layers remain frozen during the test-time phase. Conversely, "Dynamic" implies the updating of parameters within all normalization layers rooted in the source model as most existing TTA methods do in the test-time phase. 

**Self-Adaptive Attention v.s. traditional self-attention**. The traditional self-attention mechanism usually keeps the parameters inherited from the source model and performs information integration across modalities. Apparently, the distribution shift between training and test-time data might hinder the similarity estimation between tokens. As a result, reliable fusion on biased modalities cannot be guaranteed. Instead, we hope that the model could focus more on the unbiased modalities and avoid the interoperation from the bias. To this end, we propose repurposing the cross-modal attention-based fusion layers in a self-adaptive way. As depicted in Figure 2 of the manuscript, our default approach in the RFRA framework involves updating only the parameters of the last Transformer layer (referred to as the AF layer) in a self-adaptive manner. We dub this paradigm as self-adaptive attention-based fusion, abbreviated as "SAF". SAF essentially repurposes the standard AF operation by modulating the parameters within the attention layer with the guidance of the proposed objective function.

Reviewer comment (Experiments):
The confidence threshold $gamma$ work as the important parameter in the confidence-aware loss. However, the influence of the confidence threshold $\gamma$ on TTA performance is not explored, and it would be beneficial to understand its role more explicitly. The paper could conduct more experiments and analysis to show how different values of $\gamma$ affect the accuracy and robustness of TTA. The paper could provide some insights on how to choose an appropriate value of $\gamma$.

Author response:
**A2:** Thanks for your comment. In response to your concern, we investigate the influence of the only hyper-parameter (\textit{i.e.,} threshold $\gamma$ in Eq. 6) in our approach. To this end, we vary $\gamma$ in the range of $[0.1, 0.2, 0.3, e^{-1}, 0.4, 0.5]$ and perform corresponding experiments on the Kinetics50-C benchmark with fog and traffic corruptions. Results are depicted in Fig. 10 within the revised manuscript. For your convenience, we attach the corresponding numerical results in the following table.

| Threshold ($\gamma$ in Eq. 6) | 0.1  | 0.2  | 0.3  | $e^{-1}$ | 0.4  | 0.5  |
| :---------------------------- | :--: | :--: | :--: | :------: | :--: | :--: |
| Video-Fog                     | 54.2 |  56  | 56.8 |   57.4   | 57.5 | 57.7 |
| Audio-Traffic                 | 69.2 | 69.1 | 69.1 |   69.0   | 69.3 | 69.0 |

The results illustrate the stability of RFRA across varying threshold values of $\gamma$.

Reviewer comment (Reproducibility):
How does the attention value in Figure 4 calculated? Some explanation is needed. The paper should provide more details on how to compute the attention value for each modality pair in Figure 4.

Author response:
**A3:** Thanks for your comment. In Figure 4, the attention values between modalities, denoted as "Attention X-Y" (X, Y $\in$ {A, V}), are calculated using Equation 4 from the manuscript:

$$\mathbf{A}=\operatorname{Softmax}\left(\frac{\mathbf{Q} \mathbf{K}^T}{\sqrt{d}}\right)$$.

For instance, "Attention X-X" represents the self-attention, computed by setting both the query ($\mathbf{Q}$) and key ($\mathbf{K}$) as the tokens of modality X. Meanwhile, "Attention X-Y" (X, Y $\in$ {A, V}) corresponds to the configuration where the query and key tokens come from modality X and modality Y, respectively.

Reviewer comment (Experiments):
An outlying problem, how does the audio information used in autonomous vehicle? As most scenarios of autonomous vehicle might use the visual sensor and could your method used in such case?

Author response:
**A4:** Thanks for your comment. It's crucial to note that the multi-modal reliability bias revealed in this study includes but not limited to autonomous vehicles and can manifest across various modalities in diverse scenarios. While the majority of autonomous vehicles primarily incorporate sensors like cameras, radars, and lidars, the potential utilization of microphones and audio data in autonomous driving has received increasing attention. Interestingly, audio information could significantly complement the capabilities of the autonomous system. For instance, as highlighted in [A], audio input aids in emergency vehicle recognition. This underlines the broader applicability of the multi-modal reliability bias challenge and its potential implications for the wider community.

Reviewer comment (Writing):
There is a typographical error in Eq. (4); “A” should be bold. This is a minor mistake that can be easily corrected.

Author response:
**A5:** We appreciate your feedback on the notation typos. In the updated version, we have carefully revised the typos.

[A] Ivan Kharitonov. "Survey on Acoustic Sensors in Self-Driving Cars." 2023, https://hackernoon.com/survey-on-acoustic-sensors-in-self-driving-cars](https://hackernoon.com/survey-on-acoustic-sensors-in-self-driving-cars)

Reviewer comment (Experiments):
The paper contains extensive analysis and experiments, but some settings require further clarification. For instance, it's not entirely clear what "Attention A-V" means in Figure 4 and how the results demonstrate the method's robustness. In the analysis of Figure 5, claims are made about the importance of maintenance between audio and video modalities, but the contrast with clean results is not evident. Additional clarification or supplementary results are needed to support these claims.

Author response:
> *Q2.1: The paper contains extensive analysis and experiments, but some settings require further clarification. For instance, it's not entirely clear **what "Attention A-V" means** in Figure 4 and **how the results demonstrate the method's robustness**.* 

**A2.1: ** **What "Attention A-V" means.** "Attention A-V" denotes the cross-attention from the audio modality to the video modality. Specifically, in Figure 4, the attention values between modalities, denoted as "Attention X-Y" (X, Y $\in$ {A, V}), are derived using Equation 4 from the manuscript:

$$\mathbf{A}=\operatorname{Softmax}\left(\frac{\mathbf{Q} \mathbf{K}^T}{\sqrt{d}}\right)$$.

For instance, "Attention X-X" represents the self-attention, computed by setting both the query ($\mathbf{Q}$) and key ($\mathbf{K}$) as the tokens of modality X. Meanwhile, "Attention X-Y" (X, Y $\in$ {A, V}) corresponds to the configuration where the query and key tokens correspond to modality X and modality Y, respectively.

**How the results demonstrate the robustness.** Larger attention values indicate a greater focus on a particular modality during cross-modal fusion. In Figure 4, "Tent" displays slightly better robust fusion effects compared to "AF," potentially due to the narrowed domain gap achieved by repurposing the LN. In contrast, our method demonstrates significant improvements in reliability estimation (attention values) for both video and audio bias situations across varying severities. This highlights the necessity of the self-adaptive attention-based fusion paradigm for multi-modal TTA, as it enhances the model's capability to adapt to reliability bias scenarios.

> *Q2.2: In the analysis of Figure 5, claims are made about the importance of maintenance between audio and video modalities, but **the contrast with clean results is not evident**. Additional clarification or supplementary results are needed to support these claims.* 

Thanks for your comment. We have supplemented more visualization results on different baselines for comparisons. Specifically, Fig. 11 in the Appendix of the manuscript shows the attention matrixes of the source model with vanilla attention-based fusion (AF), and the model adapted by Tent with dynamic LN.

Reviewer comment (Writing):
There are a few typos and vague statements in the paper, such as "trafﬁc noise in the audio modality" in the caption of Figure 1, "information bias" on Page 2, and inconsistent notations like "(Stat. LN) & AF" or "Stat. (LN & AF)" in Tables 1-3. These should be corrected for clarity and consistency.

Author response:
> *Q3: There are a few **typos and vague statements** in the paper, such as "trafﬁc noise in the audio modality" in the caption of Figure 1, "information bias" on Page 2, and inconsistent notations like "(Stat. LN) & AF" or "Stat. (LN & AF)" in Tables 1-3. These should be corrected for clarity and consistency.* 

We appreciate your feedback on the typos. In the updated version, we have carefully revised the typos.

Reviewer comment (Reproducibility):
While the paper is well-written, it lacks essential details about CAV-MAE, such as the number of attention layers used for fusion. As many evaluations rely on the CAV-MAE framework, providing this information is crucial for the readers and reviewers to fully grasp the methodology. Additionally, visualizing the corruption types on both video and audio modalities would enhance the paper's clarity and help readers better understand the benchmarks.

Author response:
> *Q1.1: While the paper is well-written, it **lacks essential details about CAV-MAE**, such as the number of attention layers used for fusion. As many evaluations rely on the CAV-MAE framework, providing this information is crucial for the readers and reviewers to fully grasp the methodology.* 

**A1.1**: We apologize for the missing details on the backbone that confuses the reviewer. In the revised manuscript, we have supplemented the details of CAV-MAE ([A]) and more implementation details of our RFRA in Section C. For your convenience, we attach the added statement as follows. 

In the implementation, we use the CAV-MAE model as the backbone. CAV-MAE adopts an encoder-decoder-like architecture that is pre-trained on large-scale video data with both the contrastive learning and mask image modeling paradigms. The CAV-MAE encoder consists of 11 Transformer layers dedicated to each modality for the modality-speciﬁc feature extraction, alongside one Transformer layer for cross-modal fusion. The input to the CAV-MAE encoder involves 10-second video clips containing both video and corresponding audio data. For the video stream, CAVMAE samples 10 frames within each video clip and randomly selects one frame feeding into the visual Transformer encoder. For the audio stream, each 10-second audio waveform is converted into one spectrogram and then inputted to the audio Transformer encoder.

During the ﬁne-tuning phase, we maintain the visual and audio encoders of the pre-trained model and add one randomly initialized classiﬁcation head to them. The ﬁne-tuned model is regarded as the source model and denoted as “Source (Stat. (LN & AF))”. To investigate the robustness of different fusion manners, we design another variant of the source model that utilizes 12 Transformer layers for feature extraction and performs late fusion between the classiﬁcation logit of each modality. The corresponding model variant is denoted as “Source ((Stat. LN) & LF)”. During the test-time adaption phase, unless otherwise speciﬁed, all baselines update the parameters of all normalization layers rooted in the source model, i.e., referred to as “Dyn. LN”. In contrast, as depicted in Fig. 2, our default approach in the RFRA framework involves updating only the parameters of the last Transformer layer to ensure reliable fusion, denoted as “SAF”.

> *Q1.2: Additionally, **visualizing the corruption types on both video and audio modalities** would enhance the paper's clarity and help readers better understand the benchmarks.*

**A1.2**: Thanks for your valuable suggestion. In this work, to comprehensively evaluate modality bias, we introduce different distribution shifts on the video and audio modalities for the test sets of VGGSound ([B]) and Kinetics ([C]) datasets. For the video corruptions, we follow [D] to apply 15 kinds of corruptions into the video, and each corruption has 5 kinds of severity levels for extensive validations. Specifically, the corruptions on video modality include ''Gaussian Noise", ''Shot Noise", ''Impulse Noise", ''Defocus Blur", ''Glass Blur", ''Motion Blur", ''Zoom Blur", ''Snow", ''Frost", ''Fog", ''Brightness", ''Elastic", ''Pixelate", ''Contrast", and ''JPEG". Similar to the video modality, we add 6 kinds of common audio noise (https://freesound.org) with 5 kinds of severity levels captured in the wild. Specifically, the corruptions on audio modality include ''Gaussian Noise", ''Paris Traffic Noise", ''Crowd Noise", ''Rainy Noise", ''Thunder Noise" and ''Windy Noise". 

In response to your valuable suggestion, we've incorporated visualizations of corruption types in both video and audio modalities within the updated manuscript. Fig. 6 showcases the visualization results of various visual corruption types observed within the constructed Kinetics-C benchmark. Additionally, Fig. 7 presents the Mel spectrogram visualizations, highlighting the raw audio and the corresponding audio corruption types. For a comprehensive understanding, we encourage you to refer to the updated manuscript for an in-depth exploration of these visualizations.

Reviewer comment (Experiments):
The paper introduces the modulation of Q, K, and V parameters to address reliability bias. However, it's not clear why the authors chose to update all parameters of Q, K, and V simultaneously. Explaining this choice and discussing the possibility of updating only the MLP parameters of Q and K would provide valuable insights. Furthermore, while the novel attention layer repurposing is effective for reliability bias, the paper should address whether this approach comes at the cost of efficiency for test-time adaptation.

Author response:
> *Q2.1: The paper introduces the modulation of Q, K, and V parameters to address reliability bias. However, it's not clear why the authors chose to update all parameters of Q, K, and V simultaneously. **Explaining this choice and discussing the possibility of updating only the MLP parameters of Q and K** would provide valuable insights.* 

Thanks for your constructive comment. In our default approach, we update $W_{{\Theta}^h}$ and $B_{{\Theta}^h}$ $(h\in {q,k,v})$ within the last Transformer layer of the source model to ensure reliable fusion. 

In response to your valuable comment, we conduct additional experiments to explore the impact of different repurposing schemes. To this end, we design three variants: one that updates only the query and key projection layers, another that updates only the value projection layers, and a third that updates the final classification head. Results are summarized in Table 8 within the revised manuscript. For your convenience, we attach the corresponding results in the following table.

| Corruption    | Source |  QK  |  V   | MLP  | QKV (ours) |
| :------------ | :----: | :--: | :--: | :--: | :--------: |
| Video-Fog     |  46.7  | 51.7 | 53.6 | 49.1 |    57.4    |
| Audio-Traffic |  65.5  | 68.8 | 67.2 | 66.7 |    69.0    |

Results in the following table illustrate that the default setting, updating the query, key, and value projection layers simultaneously, exhibits significant performance superiority. Modulating the classification head demonstrates minimal effectiveness (e.g., from $46.7$ to $49.1$). Conversely, the attention modulation scheme achieves adaptive fusion between discrepant modalities, mitigating the multi-modal reliability bias problem (e.g., from $46.7$ to $51.7$). Moreover, modulation on the query, key, and value projection layers introduces additional parameters for reliable fusion, resulting in further improvements in robustness (e.g., from $46.7$ to $57.4$).

> *Q2.2: Furthermore, while the novel attention layer repurposing is effective for reliability bias, the paper should address **whether this approach comes** **at the cost of efficiency for test-time adaptation**.*

Thanks for your valuable suggestion. Different from most TTA methods that update the parameters of normalization layers, our RFRA repurposes the last Transformer layer of the CAV-MAE model as elaborated in Section 2 of the manuscript. 

In response to your insightful suggestion, we conduct additional experiments comparing the efficiency of the two paradigms. To this end, we choose the attention-fusion-based CAV-MAE model as the source model (\textit{i.e.}, source (Stat. (LN \& AF))), and conduct experiments on the VGGSound-C benchmark. We measure both the size of learnable parameters and the GPU time during the test-time adaptation phase. Results are summarized in Table 7 within the revised manuscript. For your convenience, we attach the corresponding results in the following table.

| Method         | \#params (M) | GPU time (14,046 pairs) |
| :------------- | :----------: | :---------------------: |
| Tent (Dyn. LN) |     0.2      |      209.5 seconds      |
| EATA (Dyn. LN) |     0.2      |      207.6 seconds      |
| SAR (Dyn. LN)  |     0.2      |      286.1 seconds      |
| RFRA (SAF)     |     1.8      |      134.1 seconds      |

The results underscore that RFRA achieves adaptation more efficiently, primarily due to its module repurposing approach. While the normalization layer updating scheme requires fewer parameters, it necessitates more time for propagation.

Reviewer comment (Experiments):
The paper introduces a crucial hyper-parameter, the confidence threshold ($\gamma$) in Equation 6. While this threshold is fixed in all experiments, it's essential to include ablation studies to explore the sensitivity of the proposed method to variations in this hyper-parameter. A more comprehensive analysis would provide a deeper understanding of the method's robustness.

Author response:
> *Q3: The paper introduces a crucial hyper-parameter, the confidence threshold ($\gamma$) in Equation 6. While this threshold is fixed in all experiments, it's essential to include ablation studies to **explore the sensitivity of the proposed method to variations in this hyper-parameter**. A more comprehensive analysis would provide a deeper understanding of the method's robustness.*

Thanks for your comment. In response to your concern, we investigate the influence of the only hyper-parameter (\textit{i.e.,} threshold $\gamma$ in Eq. 6) in our approach. To this end, we vary $\gamma$ in the range of $[0.1, 0.2, 0.3, e^{-1}, 0.4, 0.5]$ and perform corresponding experiments on the Kinetics50-C benchmark with fog and traffic corruptions. Results are depicted in Fig. 10 within the revised manuscript. For your convenience, we attach the corresponding numerical results in the following table.

| Threshold ($\gamma$ in Eq. 6) | 0.1  | 0.2  | 0.3  | $e^{-1}$ | 0.4  | 0.5  |
| :---------------------------- | :--: | :--: | :--: | :------: | :--: | :--: |
| Video-Fog                     | 54.2 |  56  | 56.8 |   57.4   | 57.5 | 57.7 |
| Audio-Traffic                 | 69.2 | 69.1 | 69.1 |   69.0   | 69.3 | 69.0 |

The results illustrates the stability of RFRA across varying threshold values of $\gamma$.

Reviewer comment (Presentation):
Some of the figures, such as Figure 1c and Figure 3b, suffer from low image clarity. Improving the quality of these figures would enhance the paper's visual presentation and make the findings more accessible to readers.

Author response:
> *Q4: Minor: Some of the figures, such as Figure 1c and Figure 3b, suffer from **low image clarity**. Improving the quality of these figures would enhance the paper's visual presentation and make the findings more accessible to readers.*

We appreciate your feedback. In the revised manuscript, we have enhanced the quality of these figures to improve their visual presentation.

Reviewer comment (Experiments):
The authors have conducted extensive evaluations on two newly constructed benchmarks regarding the most challenged setting (severity 5 in this paper), and the results indeed verify the effectiveness of handling the cross-modal reliability bias challenge. Even so, I think some challenging TTA settings that are orthogonal to the cross-modal reliability bias might help to improve the practicalness and impact of this work. First, in the practical multi-modal scenarios, the severity of distribution shifts might dynamically vary. It would make the work more practical if the authors could additionally investigate the robustness of the proposed method under the setting MIXED SEVERITIES. Furthermore, it is also common that the corruption types continually vary in the wild, resulting in the demand for continual test-time adaption (CTTA). The proposed approach would be more solid and universal if the proposed method could work in the MIXED DISTRIBUTION SHIFTS setting (i.e., continual TTA setting).

Author response:
> *Q1.1: I think some challenging TTA settings that are orthogonal to the cross-modal reliability bias might help to improve the practicalness and impact of this work. First, in the practical multi-modal scenarios, the severity of distribution shifts might dynamically vary. It would make the work more practical if the authors could **additionally investigate the robustness of the proposed method under the setting MIXED SEVERITIES**.* 

**A1.1**: In this paper, we reveal the multi-modal reliability challenge for the TTA community and propose a reliable fusion and robust adaption (RFRA) approach to tackle the issue. Following the widely-used TTA evaluation protocol ([A, B, C, D]), we introduce different corruption types to either the video or audio modality. In the main paper, we reported the performance under different corruption types with a severity level of 5. 

In response to your insightful suggestion, we conduct additional experiments on the Kinetics50-C benchmark, investigating the robustness of the proposed method under the setting of MIXED SEVERITY. To this end, we create test pairs for each corruption type by blending severity levels 1 through 5, resulting in 5N test pairs, where N represents the original size of the test data. After that, we shuffle the obtained test pairs and randomly choose N pairs for each corruption type. To verify the effectiveness of RFRA under the MIXED SEVERITY setting, we choose the typical TTA method Tent ([A]) and the SOTA TTA method SAR ([E]) as baselines for comparisons. The results are depicted in Fig. 8 within the revised manuscript. For your convenience, we attach the corresponding numerical result (regarding accuracy) in the following tables. 

| Kinetics50-C (Video Corruption, Mixed Severity) |  Gauss.  |   Shot   |  Impul.  |  Defoc.  |  Glass   |   Mot.   |   Zoom   |   Snow   | Frost    | Fog      | Brit.    | Digital  | Contr.   | Pix.     | JPEG     | AVG      |
| :---------------------------------------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Tent  ([A])                                     |   58.4   |   58.3   |   57.0   |   73.4   |   71.3   |   76.0   |   71.7   |   67.1   | 68.3     | 63.7     | **79.5** | 69.4     | 74.9     | 76.5     | 72.9     | 69.2     |
| SAR ([E])                                       |   58.6   |   58.2   |   56.8   |   73.1   |   70.1   |   75.6   |   71.1   |   66.4   | 67.1     | 63.8     | 79.4     | 69.2     | 74.5     | 76.5     | 72.3     | 68.8     |
| RFRA (Ours)                                     | **59.4** | **59.3** | **57.9** | **73.5** | **72.1** | **76.2** | **72.6** | **68.2** | **70.3** | **68.7** | 79.4     | **69.7** | **75.1** | **76.7** | **73.7** | **70.2** |

| Kinetics50-C (Audio Corruption, Mixed Severity) |  Gauss.  |  Traff.  |  Crowd.  |   Rain   |  Thund.  |   Wind   |   AVG    |
| :---------------------------------------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: |
| Tent  ([A])                                     |   76.1   |   68.9   |   70.3   |   70.4   |   68.0   | **72.5** |   71.0   |
| SAR ([E])                                       |   76.1   |   66.3   |   69.1   |   69.1   |   68.5   |   72.2   |   70.2   |
| RFRA (Ours)                                     | **76.5** | **71.2** | **71.2** | **72.2** | **72.6** | **72.5** | **72.7** |

The results indicate the effectiveness of RFRA in addressing cross-modal reliability bias across various corruption types exhibiting mixed severity levels.

Reviewer comment (Reproducibility):
Some details are missing. It is not clear how many layers of the attention module are used and repurposed during test time adaption in the proposed approach. And how many parameters do these self-attention layers account for? It is encouraged to supplement comparisons with the TTA baselines regarding the number of modulated parameters and the adaption time, which would make the comparisons more comprehensive.

Author response:
>  *Q2.1: **Some details are missing**. It is not clear how many layers of the attention module are used and repurposed during test time adaption in the proposed approach. And how many parameters do these self-attention layers account for?* 

**A2.1**: We apologize for the initial oversight in providing comprehensive details. In the revised manuscript, we have supplemented the details of CAV-MAE ([H]) and implementation details of our RFRA. For your convenience, we attach the added statement as follows. 

In the implementation, we use the CAV-MAE model as the backbone. CAV-MAE adopts an encoder-decoder-like architecture that is pre-trained on large-scale video data with both the contrastive learning and mask image modeling paradigms. The CAV-MAE encoder consists of 11 Transformer layers dedicated to each modality for the modality-speciﬁc feature extraction, alongside one Transformer layer for cross-modal fusion. The input to the CAV-MAE encoder involves 10-second video clips containing both video and corresponding audio data. For the video stream, CAVMAE samples 10 frames within each video clip and randomly selects one frame feeding into the visual Transformer encoder. For the audio stream, each 10-second audio waveform is converted into one spectrogram and then inputted to the audio Transformer encoder.

During the ﬁne-tuning phase, we maintain the visual and audio encoders of the pre-trained model and add one randomly initialized classiﬁcation head to them. The ﬁne-tuned model is regarded as the source model and denoted as “Source (Stat. (LN & AF))”. To investigate the robustness of different fusion manners, we design another variant of the source model that utilizes 12 Transformer layers for feature extraction and performs late fusion between the classiﬁcation logit of each modality. The corresponding model variant is denoted as “Source ((Stat. LN) & LF)”. During the test-time adaption phase, unless otherwise speciﬁed, all baselines update the parameters of all normalization layers rooted in the source model, i.e., referred to as “Dyn. LN”. In contrast, as depicted in Fig. 2, our default approach in the RFRA framework involves updating only the parameters of the last Transformer layer to ensure reliable fusion, denoted as “SAF”.

> *Q2.2: It is encouraged to supplement **comparisons with the TTA baselines regarding the number of modulated parameters and the adaption time**, which would make the comparisons more comprehensive.*

**A2.2:** Thanks for your valuable suggestion. Different from most TTA methods that update the parameters of normalization layers, our RFRA repurposes the last Transformer layer of the CAV-MAE model as elaborated in Section 2 of the manuscript. In response to your insightful suggestion, we conduct additional experiments comparing the efficiency of the two paradigms. To this end, we choose the attention-fusion-based CAV-MAE model as the source model (\textit{i.e.}, source (Stat. (LN \& AF))), and conduct experiments on the VGGSound-C benchmark. We measure both the size of learnable parameters and the GPU time during the test-time adaptation phase. Results in the following table highlight that our RFRA accomplishes adaptation in less time. The efficiency of RFRA can be attributed to its module repurposing approach. Although the normalization layer updating scheme occupies fewer parameters, it demands more time for propagation.

| Method         | \#params (M) | GPU time (14,046 pairs) |
| :------------- | :----------: | :---------------------: |
| Tent (Dyn. LN) |     0.2      |      209.5 seconds      |
| EATA (Dyn. LN) |     0.2      |      207.6 seconds      |
| SAR (Dyn. LN)  |     0.2      |      286.1 seconds      |
| RFRA (SAF)     |     1.8      |      134.1 seconds      |

Reviewer comment (Presentation):
The results of the default setting can be added to the ablation tables for clear clarification. In the current form, the readers need to compare the main table (1,2,3) and the ablation tables for contrast.

Author response:
> *Q3: The results of **the default setting can be added** to the ablation tables for clear clariﬁcation. In the current form, the readers need to compare the main table (1,2,3) and the ablation tables for contrast.*

**A3**: We apologize for the confusion arising from the initial presentation. To address this concern, we've made necessary adjustments in the revised manuscript by incorporating the default setting's results, distinctly highlighted in pink, within the ablation tables (Tables 4, 5, and 6).

Reviewer comment (Experiments):
My questions are mainly in the efficiency comparisons between the proposed SAF module and the TTA baselines, and some clarification on statements. Moreover, I wonder about the effect of the proposed method on the settings of MIXED SEVERITIES and MIXED DISTRIBUTION SHIFTS. Certainly, this is optional during the rebuttal time because the settings are out of the scope of the paper, but I think the results would strengthen this work.

Author response:
> Q1.2: Furthermore, it is also common that the corruption types continually vary in the wild, resulting in the demand for continual test-time adaption (CTTA). The proposed approach would be more solid and universal if the proposed method could work **in the MIXED DISTRIBUTION SHIFTS setting (i.e., continual TTA setting)**.

**A1.2**: Following the setting of continual TTA ([F, G]), we conduct additional experiments on the Kinetics50-C benchmark under the MIXED DISTRIBUTION SHIFTS setting as the reviewer suggested. In this setting, both baseline methods (Tent and SAR) alongside our RFRA continually adapt to evolving corruption types, and the averaged performance across all corruption types is reported. To ensure comprehensive evaluations, we vary the severity levels from 1 to 5. The results are summarized in Fig. 9 within the updated manuscript.
For your convenience, we attach the corresponding numerical result (regarding accuracy) in the following tables.

| Kinetics50-C (Video Corruption, Mixed Distribution Shifts) | Severity 1 | Severity 2 | Severity 3 | Severity 4 | Severity 5 | AVG      |
| ---------------------------------------------------------- | :--------: | :--------: | :--------: | :--------: | :--------: | -------- |
| Tent  ([A])                                                |     76     |    72.1    |    67.1    |    39.5    |    38.3    | 58.6     |
| SAR ([E])                                                  |    76.1    |    71.8    |  **69.0**  |    64.3    |    59.6    | 68.2     |
| RFRA (Ours)                                                |  **76.8**  |  **72.5**  |  **69.0**  |  **65.0**  |  **61.7**  | **69.0** |

| Kinetics50-C (Audio Corruption, Mixed Distribution Shifts) | Severity 1 | Severity 2 | Severity 3 | Severity 4 | Severity 5 |   AVG    |
| -------------------------------------------------- | :--------: | :--------: | :--------: | :--------: | :--------: | :------: |
| Tent  ([A])                                        |    71.2    |    70.8    |    70.0    |    69.4    |    68.5    |   70.0   |
| SAR ([E])                                          |    71.0    |    70.6    |    70.0    |    69.8    |    69.3    |   70.1   |
| RFRA (Ours)                                        |  **72.2**  |  **71.6**  |  **71.0**  |  **70.4**  |  **69.7**  | **71.0** |

Notably, the performance of the vanilla TTA method (Tent) significantly degrades, especially under high-level mixed distribution shifts (e.g., 38.3 at Severity 5). In contrast, our RFRA showcases relative robustness against these shifts across different severity levels. It's worth highlighting that while our RFRA was primarily designed to address the challenge of multi-modal reliability bias, it demonstrates consistent performance superiority compared to SAR, an approach explicitly tailored for handling mixed distribution shifts in the continual TTA setting. This observation underscores the adaptability and resilience of RFRA beyond its initial design scope.