wav2vec- S : Adapting Pre-trained Speech Models for Streaming (2024)

Abstract: Pre-trained speech models, such as wav2vec 2.0, have significantly advanced speech-related tasks, including speech recognition and translation. However, their applicability in streaming scenarios is limited because these models are trained on complete utterances, leading to a mismatch with incremental streaming inputs. This paper identifies three critical design aspects within the architecture of wav2vec 2.0 and proposes a novel model, wav2vec-S, which incorporates simple modifications to ensure consistent speech representations during both training and inference phases for streaming speech inputs. Furthermore, we demonstrate that wav2vec-S models can be efficiently adapted from pre-trained wav2vec 2.0 models through continued pre-training and effectively finetuned to meet various latency requirements in downstream applications. Experiments on speech recognition and translation tasks show that wav2vec-S outperforms strong baseline models and achieves a superior balance between quality and latency.

## introduction the advent of pre-trained speech models, such as wav2vec @xcite , hubert @xcite , wavlm @xcite , and data2vec @xcite , have catalyzed significant progress in speech-related tasks, including automatic speech recognition (asr) @xcite and speech translation (st) @xcite @xcite . these models learn powerful speech representations from large amounts of unlabeled speech data through self-supervised learning techniques, leading to significant performance improvements. despite the advancements, their potential has not yet been fully harnessed for streaming scenarios. these models, trained and optimized for processing complete utterances, yield sub-optimal performance when applied to partial speech inputs in streaming scenarios. previous efforts to bridge this gap have often utilized knowledge distillation techniques @xcite @xcite , with the goal of transferring knowledge from offline pre-trained models to streaming student models. however, these strategies require taskspecific distillation, and generally involve complex, multi-stage training processes @xcite or the need to balance multiple distillation objectives @xcite . this limits their applicability as universal pre-trained models. moreover, these methods do not fully exploit the potential of large-scale unlabeled data for streaming speech tasks. in this paper, we focus on wav2vec 2.0 @xcite and identify three critical design aspects of its architecture that hinder its applicability to streaming tasks. accordingly, we propose a novel model, wav2vec-s, which incorporates simple modifications to accommodate streaming requirements. specifically, we substitute layer normalization for group normalization in the feature extractor, shift from convolution-based relative position encoding to absolute position encoding, and implement a block-wise unidirectional selfattention mechanism, all designed to eliminate or restrict dependency on future context, as required for maintaining consistent representations during training and inference for streaming inputs. we demonstrate that instead of training from scratch, wav2vec-s can be efficiently adapted from pretrained wav2vec 2.0 models through continued pretraining on unlabeled speech using the same selfsupervised training objectives. in addition, we in-troduce mechanisms in both the pre-training stage and the fine-tuning stage to allow the model to meet various latency requirements in downstream tasks. in summary, our contributions are threefold: ## preliminary wav2vec 2.0 represents a significant advancement in speech processing and has been widely utilized as a backbone for various speech-related tasks @xcite @xcite @xcite . our proposed model is based on wav2vec 2.0 with essential modifications to support streaming applications. we first briefly review its model architecture and training objectives. ## our proposal: wav2vec-s we next describe how to adapt wav2vec 2.0 to wav2vec-s for streaming, focusing on three key aspects: architecture, pre-training, and fine-tuning. ## conclusion in this paper, we propose a streaming pre-trained speech model, wav2vec-s, which is effectively adapted from the pretrained wav2vec 2.0 model for streaming tasks through simple architectural modifications and continued pretraining. experiments on both streaming asr and st tasks show the superiority of wav2vec-s in terms of performance, pre-training and inference efficiency.