Paper: Advancing Test-Time Adaptation for Acoustic Foundation Models in Open-World Shifts (ICLR-2024)

Reviewer comment (Writing):
Writing quality could be very improved (perhaps doing in their next version), including issues with citations code/format usages and grammar.

Author response:
**A1**: We appreciate your feedback regarding the writing quality. We have carefully revised the citation formats and typos in the new version of our paper.

Reviewer comment (Novelty):
The literature review on frozen acoustic model-based adaptation lacks depth. The current review primarily applies existing algorithms from the ML community, and the proposed method inadvertently overlaps with reprogramming/prompting due to an inadequate literature survey. For example, additional losses-based adapters/input-only training have also been proposed in [4].

Author response:
**A2**: Thanks for highlighting these references. We have incorporated a discussion of them within the related work section in the revised version.  

**A3**: In summary, the key difference is that our proposed method performs **unsupervised** adaptation-based entropy minimization while reprogramming and jointly reprogramming plus adapter perform **supervised** optimization with access to annotated target data pairs. Furthermore, while reprogramming-based methods focus on training additional small parameter sets for data input parts or adapters within the model, our method focuses on the feature modulation on norm layers without additional parameter introduction.

Reviewer comment (Theory):
The theoretical connection between test-time adaptation efficacy and frozen pre-trained models is unclear. A well-known recent insight involves latent space alignment via Wasserstein measurements, yet the related discussion is absent here.

Author response:
We sincerely appreciate the further feedback and suggestions. We make the following promises in the final version:

- add preliminary results for bias-only tuning and make the connection between our work with existing parameter-efficient techniques
- add a discussion on related theoretical findings of the large pre-trained AMs

Furthermore, we will carefully consider the usage of “foundation models” again, and the change of colors for Figure 2.

Reviewer comment (Novelty):
The related speech processing references are missing, including frozen model adaptation via Bayesian adaptation in speech, and frozen model adaptation with trainable inputs reprogramming/prompting.

Author response:
**A2**: Thanks for highlighting these references. We have incorporated a discussion of them within the related work section in the revised version.

Reviewer comment (Experiments):
Ablation experiments are missing on similar parameter-efficient baselines work on frozen pre-trained adaptation diagrams (e.g., input reprogramming, adapters, prompting), making it difficult to justify the novelty of applying trainable layer norms.

Author response:
**A5**: We hope to kindly clarify that parameter efficiency and applying trainable layer norms are not our main contributions even though our method is indeed parameter-efficient. Rather, we investigate TTA within the audio modality and discover that non-silent frames with high entropy contain vital semantic content that cannot be dropped. This motivates us to develop a learning-based adaptation approach to address it. 

Besides, there is a key difference in the experimental setting. That is, the mentioned parameter-efficient baselines are trained using supervised signals, while our method performs unsupervised optimization.

Reviewer comment (Experiments):
In terms of frozen pre-trained models, only Wav2Vec 2.0 Large has been evaluated. The term "foundation model" is often used for models with over 1 billion or even 100 billion parameters to showcase emergent abilities, which is not the case here.

Author response:
**A4**: Thanks for the good suggestion. We choose the Wav2vec2 models due to their popularity in the community and we acknowledge the importance of evaluating diverse acoustic models. Hence, we conduct additional experiments using, Hubert-Base, Hubert-Large, WavLM-Base, and WavLM-Large to compare the adaptation performance under different model sizes, and training data sources. The experimental results are discussed in Appendix A.5 in the new version. 

Furthermore, we believe that there hasn’t been a clear definition for the "acoustic foundation models" in terms of model sizes. This perspective -- "The term "foundation model" is often used for models with over 1 billion or even 100 billion parameters" -- is often discussed in the context of large language models. We refer to "acoustic foundation models" as those large acoustic models that serve fundamental roles and can be extended to various downstream speech tasks.

Reviewer comment (Novelty):
One of the paper's goals is heuristic-free test-time adaptation. As for me, the authors mainly move existing heuristics into loss functions, which makes it a learnable approach and gets rid of hyper-parameters that need to be manually set (at least when ignoring optimization hyperparameters such as learning rate, etc.). But the heuristics remain, don't they?

Author response:
**A1**: Thanks for this good question. We acknowledge that the heuristics still remain. Our "heuristic-free" refers to removing the filter-based method for unreliable data with high entropy. Our original intention is to make this heuristic-based method into a learning-based method. We will clarify our description and revise it in the new version.

Reviewer comment (Writing):
Eq.(3) & Eq.(4): Are these frame-level (what the text says) or sequence-level (what the equations look like) quantities?

Author response:
**A4**: We are sorry for the confusion. Eq.(3) and Eq.(4) should be frame-level quantities. We have revised Equations (3) and (4), as well as the cardinalities in Equation (5) in the updated version of our paper. Thanks for pointing out these issues.

Reviewer comment (Writing):
Eq.(4): Please fix subscript - subscript S with i or sum over i? Utterance-level E?

Author response:
**A4**: We are sorry for the confusion. Eq.(3) and Eq.(4) should be frame-level quantities. We have revised Equations (3) and (4), as well as the cardinalities in Equation (5) in the updated version of our paper. Thanks for pointing out these issues.

Reviewer comment (Theory):
Eq.(5): Cardinalities of z’ don’t match.

Author response:
**A4**: We are sorry for the confusion. Eq.(3) and Eq.(4) should be frame-level quantities. We have revised Equations (3) and (4), as well as the cardinalities in Equation (5) in the updated version of our paper. Thanks for pointing out these issues.

Reviewer comment (Experiments):
The text says "The classification of frames as silent or non-silent was determined based on pseudo labels derived from model predictions." For Figure 3, is the classification done once on the initial, non-adapted model or repeatedly on the respective adapted model?

Author response:
**A5**: The classification is done repeatedly on the respective adapted model.

Reviewer comment (Experiments):
How are the hyper-parameters (e.g., loss coefficients) selected?

Author response:
**A6**: Hyper-parameters are selected using standard search techniques. The implementation details of hyper-parameters can be found in Appendix A.2.

Reviewer comment (Presentation):
Something is off with the inline citations.

Author response:
**A9**: Thanks for pointing out these issues. We have revised them in the new version.

Reviewer comment (Writing):
Typo: "model training _phrase_"

Author response:
**A9**: Thanks for pointing out these issues. We have revised them in the new version.

Reviewer comment (Theory):
Why do we need the auxiliary loss in Eq.(5) to improve the consistency of speech signals? Could we get rid of this heuristic by using a proper sequence-level loss (e.g., sequence-level instead of frame-level entropy in Eq.(2))?

Author response:
**A3**: Thanks for the great suggestion.  We agree that adopting the sequence-level loss, such as sequence discriminative training loss might further improve the TTA performance. We consider exploring this as a future work.

Reviewer comment (Evaluation):
Summary of Table 2, for example: 'Ours' outperforms 'SUTA' 0.2 - 1.1% absolute, but WER for lower SNRs is still a multiple of WER on "clean" data (as much as 82.2 vs 17.5). This makes me wonder what the lowest achievable WER (e.g., human WER) is, in particular on the heavily corrupted data sets? This information might be useful to understand the empirical results:  

(i) If bound by information loss of corruption: The proposed approach may be optimal and the small gains highly significant. But the task may be not the best to evaluate the proposed method.  

(ii) If not bound: What is missing to considerably close the WER gap?

Author response:
**A10**: This is a good question. We agree that the human WER bound can indeed assist in understanding the experimental results. However, it is difficult for us to obtain such numbers as it requires conducting human subjective studies.  Besides, heavily corrupted speech data may not be common cases in the real world. Our primary aim in this experimental setting is to evaluate the trend of model robustness under varying noise levels, and our results demonstrate that our proposed method outperforms all baselines in this regard.

Reviewer comment (Novelty):
The contributions seems minimal, as the core idea of TTA (Niu et al., 2023 and other related works cited in the paper) are already available in the literature.

Author response:
We recognize the existence of TTA-related works in the literature. However, our work focuses on acoustic modeling, compared to the prevalent focus on vision tasks in previous studies. Specifically, our key contributions include: 
- Identifying that noisy speech frames with high entropy often contain non-silent segments crucial for semantic understanding.
- Introducing a learning-based adaptation approach, complemented by short-term consistency regularization, to augment the adaptation performance.

Reviewer comment (Experiments):
In comparison to different vocabulary sizes, such as employing a BPE model with a larger vocabulary, how does this work perform? Would the entropy for non-silent frames significantly increase under such circumstances?

Author response:
Thanks for the question. Our proposed method is generalizable to models with large vocabulary sizes. Theoretically, the maximum entropy for non-silent frames is expected to increase due to the larger number of classes. Practically, this might also depend on the test input and models. We conduct an additional experiment using the Conformer-CTC model with BPE tokenization, observing an increase in entropy for non-silent frames from 59.4\% to 70.0\%, as illustrated in Table 2.
  
Table 2: Entropy Distribution at Step 0 for models with different vocabulary sizes.

| Vocab Size | Wav2vec2 Base | Conformer-CTC | 
| ---    | ---   | ---    | 
| n-sil-h | 0.594 | 0.700 | 
| n-sil-l | 0.406 | 0.300 |  
| sil-h   | 0.362 | 0.497 |  
| sil-l   | 0.638 | 0.503 |

Reviewer comment (Experiments):
Could this approach be adapted to function with a model architecture similar to RNN-Transducer? What specific modifications would be necessary to enable its compatibility with such an architecture?

Author response:
We think that our proposed approach is adaptable to architectures like RNN-Transducer. The primary modification would involve basing the prediction probability and pseudo labels on the decoder outputs.

Reviewer comment (Evaluation):
Could you provide further clarification on the decoding strategies outlined in Section 5.4 of your paper? Specifically, post-adaptation, how do different inference strategies, such as beam search or greedy decoding, impact the inference process, and what are the key differences in their outcomes?

Author response:
The performance differences between the two decoding strategies are detailed in Table 4, Sec.5.3.2. Beam search incorporates an additional language model in the decoding process, while greedy search does not. This language model, with its inherent text distribution prior, can help eliminate impossible text combinations during decoding. Moreover, the inclusion of a language model provides a mechanism to address text-domain shifts.

Reviewer comment (Experiments):
In the ablation study the impact of the two terms in equation 5 is explained. I think the conclusion about the impact of components depends on the setups. The importance of the two components could be different in the noise shift than the accent shift. what do you think?

Author response:
Thanks for this question. We agree that the importance of two components may indeed be different in the noise shift and accent shift because these shifts come from distinct sources. Therefore, we conduct additional ablation studies for them respectively. Specifically, we conduct experiments using the dataset with level-3 Gaussian noises in LS-C for noise shift and speech of speaker BWC with Mandarin as L1 in L2-Arctic for accent shift. All experiments are carried out using the Wav2vec2 Base model. The results are detailed in Table 1: 

Table 1: Ablation study of core components in the noise shift and accent shift

| Method | Noise Shift | Accent Shift | 
| ---    | ---   | ---    | 
| Ours     | **24.0** | **23.0** | 
| w/o STCR | 25.1 | 23.4 | 
| w/o CEA  | 35.9 | 26.9 | 
| Source | 39.5 | 28.5 |
 
These experimental findings indicate a pronounced efficacy of our method in mitigating noise shifts as opposed to accent shifts. We conjecture the reason could be that the shift caused by Gaussian noises for each frame is consistent while other shifts such as accent shift could be different within frames.

Reviewer comment (Theory):
Do you consider language as a shift? could a foundation model be adapted to a new language?

Author response:
Thanks for the interesting question. In our current experimental setting, we mainly address acoustic shifts including Gaussian and environmental noises, accents, and sung speech. We consider the text-domain shifts within the same language, reflecting variations in the linguistic content or context of the speech data. Adapting the model to a new language is an important research problem in multilingual acoustic models. We consider it as future work.

Reviewer comment (Reproducibility):
what is a practical way to get alpha in equation 5?

Author response:
Alpha represents the weight for short-term consistency regularization and is a hyperparameter in our experiments. We employ standard grid search techniques to search the value ranging from 0.1 to 1 with 0.1 an interval. The details can be found in Appendix A.2.