Continual Test-time Adaptation for End-to-end Speech Recognition on Noisy Speech (2024)

Abstract: Deep Learning-based end-to-end Automatic Speech Recognition (ASR) has made significant strides but still struggles with performance on out-of-domain samples due to domain shifts in real-world scenarios. Test-Time Adaptation (TTA) methods address this issue by adapting models using test samples at inference time. However, current ASR TTA methods have largely focused on non-continual TTA, which limits cross-sample knowledge learning compared to continual TTA. In this work, we first propose a Fast-slow TTA framework for ASR that leverages the advantage of continual and non-continual TTA. Following this framework, we introduce Dynamic SUTA (DSUTA), an entropy-minimization-based continual TTA method for ASR. To enhance DSUTA’s robustness for time-varying data, we design a dynamic reset strategy to automatically detect domain shifts and reset the model, making it more effective at handling multi-domain data. Our method demonstrates superior performance on various noisy ASR datasets, outperforming both non-continual and continual TTA baselines while maintaining robustness to domain changes without requiring domain boundary information.

## introduction deep learning-based end-to-end automatic speech recognition (asr) has made remarkable progress in recent years, achieving low recognition error rates for in-domain samples. however, domain shifts frequently occur in real-world scenarios. although recent large-scale asr models exhibit some generalization to out-of-domain test samples, their performance on out-of-domain samples still lags behind the in-domain performance. test-time adaptation (tta) is an attractive method to address domain shift issues during inference time. tta adapts the model using only single or batched test samples without needing the source training data at testing time. specifically, the source model is adapted via unsupervised objectives like entropy minimization (em) @xcite or pseudo-labeling (pl) @xcite in inference time. tta methods can be characterized into two categories: 1) non-continual tta methods adapt the source model for each test utterance and reset to the original model for subsequent samples @xcite , and 2) continual tta (ctta) continuously adapts the model for target domains, leveraging knowledge learned across samples to improve performance @xcite . tta methods initially strive in the field of computer vision @xcite @xcite . in speech recognition, recent studies have tailored tta methods with embased optimization @xcite @xcite , proposing new training objectives and demonstrating effectiveness across datasets. however, existing asr tta methods only focus on non-continual tta, constraining the model to learn knowledge across samples. there is limited research on ctta for end-to-end asr. recently, awmc @xcite proposed a pseudo-labeling ctta method for asr on a single test domain. however, as shown in previous work @xcite , pseudo-labeling is not as effective as em-based methods, and its ability on long multi-domain testing data is unknown. in this work, we propose a general fast-slow tta framework that leverages the advantages of both continual and non-continual tta. based on this framework, we introduce an em-based ctta method named dynamic suta (dsuta) for asr. furthermore, to enhance the robustness of dsuta on time-varying domain data, we propose a dynamic reset strategy to automatically detect domain shifts and determine when to reset the model to the original source model. this strategy improves fast-slow tta over long sequences of multidomain data streams. we demonstrate the effectiveness of our method on single-domain and multi-domain time-varying asr benchmarks under different acoustic conditions, simulating real-world changing environments. our method outperforms the strong singleutterance baseline suta @xcite and the ctta baseline awmc @xcite , showing robustness to domain changes even without knowing the domain boundaries. our contributions can be summarized as follows: ## methodology section 3.1 describes the proposed fast-slow tta framework. following this framework, section 3.2 extends suta into dynamic suta. to handle multi-domain scenarios better, we propose a dynamic reset strategy in section 3.3. ## conclusion in this work, we advance the non-continual test-time adaptation (tta) method for asr into a continual learning framework using a novel approach to stabilize adaptation and improve performance. specifically, we introduce dynamic suta (dsuta), a fast-slow method that combines noncontinual and continual tta, demonstrating significant improvements on single-domain test data. additionally, we propose a statistical dynamic reset strategy to enhance robustness and performance on time-varying test data streams. experimental results indicate that our proposed method outperforms the non-continual suta baseline and previous continual tta methods using pseudo labeling.