Advancing Test-Time Adaptation in Wild Acoustic Test Settings (2024)

Abstract: Acoustic foundation models, fine-tuned for Automatic Speech Recognition (ASR), suffer from performance degradation in wild acoustic test settings when deployed in real-world scenarios. Stabilizing online Test-Time Adaptation (TTA) under these conditions remains an open and unexplored question. Existing wild vision TTA methods often fail to handle speech data effectively due to the unique characteristics of high-entropy speech frames, which are unreliably filtered out even when containing crucial semantic content. Furthermore, unlike static vision data, speech signals follow short-term consistency, requiring specialized adaptation strategies. In this work, we propose a novel wild acoustic TTA method tailored for ASR fine-tuned acoustic foundation models. Our method, Confidence-Enhanced Adaptation, performs frame-level adaptation using a confidence-aware weight scheme to avoid filtering out essential information in high-entropy frames. Additionally, we apply consistency regularization during test-time optimization to leverage the inherent short-term consistency of speech signals. Our experiments on both synthetic and real-world datasets demonstrate that our approach outperforms existing baselines under various wild acoustic test settings, including Gaussian noise, environmental sounds, accent variations, and sung speech.

## introduction deep learning-based acoustic models have exhibited remarkable performance in scenarios where the training and test sets adhere to the independent and identically distributed (i.i.d) assumption. however, real-world applications frequently involve domain shifts between training and test sets, such as noise variations due to environmental sounds (reddy et al., 2019), and timbre variations due to accent or pronunciation changes @xcite . while recent acoustic foundation models, such as wav2vec2 @xcite , fine-tuned on automatic speech recognition (asr) achieve excellent performances, they exhibit notable performance degradation when confronted with the testtime speech in the wild, as depicted in figure 1 . consequently, there exists an emergent demand to adapt these acoustic foundation models in wild acoustic test settings when deployed in the real world. prior methods for mitigating domain shifts require access to domain-specific source data under the unsupervised domain adaptation setting @xcite , limiting the application to online scenarios where speech data come from the wild world with mixed distribution shifts. test-time adaptation (tta) emerges as a critical paradigm for addressing distribution shifts at inference time, enabling online updates of models on test data in a source-free way. recent work, suta @xcite , presents a pilot study on tta for asr models by applying entropy minimization to speech frame adaptation, demonstrating impressive performance on single-utterance tta. however, suta focuses on mild test settings, e.g., testing on speech with synthetic and real noises. in the dynamic wild world, acoustic foundation models may face arbitrary test speech data with severe distribution shifts, such as sung speech. as such, stabilizing online tta under wild acoustic test settings remains an open and unexplored question. recent work, sar @xcite , proposes an efficient optimization scheme for stabling online tta in the wild vision test settings. however, direct adoption of sar to speech data is challenging because sar characterizes high-entropy noisy speech samples as unreliable and potentially harmful for model adaptation and proposes to filter them out for stabling under wild vision test settings. in this work, we empirically identify a substantial proportion of noisy frames within nonsilent speech segments under wild acoustic test settings. we observe that these frames contain vital semantic information crucial for accurate recognition and merely discarding these noisy frames may adversely affect model performance. consequently, rather than excluding these noisy non-silent frames, we propose confidence enhanced adaptation (cea), which performs framelevel adaptation using a confidence-aware weight scheme. cea prioritizes uncertain frames and encourages models to focus more on these uncertain frames by 'denoising' their intermediate representations. additionally, we emphasize that frames within a short speech segment are temporally coherent, largely due to the consistent nature of phonemic content within such windows, thus proposing short-term consistency regularization to stabilize wild acoustic tta. this contrasts with image samples in a batch, which are frequently treated as independent entities. we conduct a wide range of experiments for asr fine-tuned acoustic foundation models on both synthetic and real-world datasets, systematically assessing the model's robustness against gaussian noises, environmental sounds, accents of second language (l2) learners, and singing (a.k.a. sung speech). the experimental results demonstrate the effectiveness of our method under wild acoustic test settings. in summary, our contributions are summarized as follows: ## preliminary we center our focus on the fully test-time adaptation framework, characterized by episodic model adaptation, where the model is reset after processing each utterance. we denote the asr fine-tuned acoustic foundation model as f θ (y|x). we investigate the popular acoustic foundation models such as wav2vec2 @xcite , hubert @xcite , wavlm @xcite , which can be typically decomposed into two constituent components: a feature extractor g ϕ (z|x), parameterized by ϕ, and a transformer encoder h θ (y|z), parameterized by θ. this decomposition is expressed as: ## method in this section, we first analyze the common source of domain shifts in the wild acoustic test settings, and then provide our findings and methods for addressing the wild acoustic shifts. the overview of our method is presented in figure 2 . ## experiments in this section, we undertake an evaluation of the robustness of asr fine-tuned acoustic foundation models under wild acoustic test settings. we discuss the robustness against synthetic noises including gaussian noises and real-world environmental sounds in section 5.2, real-world data shifts including l2 accents and singing voice (sung speech) in section 5.3, and decoding strategy pertaining to language models in section 5.4. we provide more evaluation results using various acoustic models in appendix b.4. ## analysis 6.1 generalization on different asr models we examine the robustness of ctc-based acoustic foundation models in our main experiments and appendix b.4. to verify the efficacy of our method on other end-to-end asr models such as conformer and transducer, we conducted experiments on conformer-ctc @xcite and conformer-transducer @xcite as per @xcite . for consistent setting and fair comparison, we experimented with dsing-test and reported the results in ## conclusions in this paper, we study the test-time adaptation of asr fine-tuned acoustic foundation models under wild acoustic test settings. by investigating the role of high-entropy noisy frames within non-silent speech segments, we introduce confidence enhanced adaptation with a confidenceaware weight optimization scheme to prioritize these noisy frames for efficient adaptation via denoising their intermediate representations rather than discarding them. moreover, our emphasis on short-term stability of speech signals leads us to apply consistency regularization, yielding further improvement for stable online tta. our experimental findings suggest a consistent improvement for different types of acoustic shifts and different degrees of corruption on synthetic and real-world datasets, demonstrating the efficacy of our approach under wild acoustic test settings.