S peech UT : Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022)

Abstract: The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods. In this paper, we propose a unified-modal speech-unit-text pre-training model, SpeechUT, to connect the representations of a speech encoder and a text decoder with a shared unit encoder. Leveraging hidden-unit as an interface to align speech and text, we can decompose the speech-to-text model into a speech-to-unit model and a unit-to-text model, which can be jointly pre-trained with unpaired speech and text data respectively. Our proposed SpeechUT is fine-tuned and evaluated on automatic speech recognition (ASR) and speech translation (ST) tasks. Experimental results show that SpeechUT gets substantial improvements over strong baselines, and achieves state-of-the-art performance on both the LibriSpeech ASR and MuST-C ST tasks. To better understand the proposed SpeechUT, detailed analyses are conducted. The code and pre-trained models are available at https://aka.ms/SpeechUT.

## introduction self-supervised pre-training with large-scale unlabeled data obtains remarkable progress on various downstream tasks @xcite @xcite @xcite . specifically, pre-trained models, such as bert @xcite and gpt @xcite , have extensively promoted the development of natural language processing @xcite . researchers also develop many pre-trained speech models utilizing a mass of unlabeled audio data, e.g., wav2vec @xcite and hubert @xcite . although text and speech are two different modalities, they have a natural relationship because they * work done during internship at microsoft research asia. † corresponding author (lozhou@microsoft.com). ## related work the proposed speechut is built upon the transformer encoder-decoder model @xcite and relates to discrete speech representation learning and joint speech-text pre-training. we discuss these topics in the following. discrete speech representation learning discretizing continuous speech signals for speech representation learning has drawn substantial attention. vq-wav2vec @xcite and wav2vec 2.0 @xcite attempt at discretizing speech signals into quantized units from a learnable codebook @xcite . pbert @xcite instead uses phonemes as the discrete targets in a semi-supervised setting. semface @xcite proposes to use language-independent vector quantized units as the semantic interface of encoder pre-training and decoder pre-training. inspired by the masked language model in bert @xcite , hubert @xcite first introduces the masked speech prediction of hidden units to pre-train a universal speech model. particularly, the hidden units can be clustered from log mel-filterbank features or the hidden states of the previous pre-trained model. recently, some studies explore leveraging the discrete hidden units to build speech-to-speech translation systems @xcite , which first convert source speech into target units, then generate the target waveform from predicted units. however, our goal in this paper is to jointly pre-train speech and text with the hidden units as the intermediate bridge. joint speech-text pre-training single-modal pre-trained models have achieved remarkable results in both natural language processing and spoken language processing, such as bert @xcite , unilm @xcite , xlnet @xcite , wav2vec 2.0 @xcite , hubert @xcite , and wavlm @xcite . thanks to the rapid development of these single-modal pre-training works, researchers begin to pre-train a cross-modal model with both speech and text data @xcite @xcite @xcite @xcite . one category of these works focuses on pre-training a unified encoder model for spoken language understanding @xcite @xcite . in parallel to our work, speechlm @xcite leverages two kinds of tokenizers to tokenize speech and text, and aims at unifying speech and text modalities into the same semantic space within one encoder model. when fine-tuning an encoder-decoder model, a randomly initialized decoder needs to be superimposed on the encoder for speech-to-text tasks @xcite . besides, maestro @xcite utilizes paired speech-text data to learn speech-text alignment through a modality-matching algorithm in rnn-t framework. our proposed speechut model is most related to encoder-decoder pre-trained models like speecht5 @xcite and stpt @xcite , in which speech and text are directly connected by a shared encoder. unlike them, speechut leverages hidden units @xcite as the bridge between the speech encoder and the text decoder, decoupling the conventional model into two pre-trained speech-to-unit and unit-to-text models. ## speechut figure 2 shows the overall framework of speechut, which leverages the unit representation as the bridge between speech and text. in this section, we will introduce the model architecture, pre-training, and fine-tuning methods. ## conclusion in this paper, we propose speechut, a unifiedmodal speech-unit-text pre-training model, which bridges the modality gap between speech and text representation with hidden units. by pre-training with the speech-to-unit task, masked unit modeling task, and unit-to-text task, speechut significantly outperforms strong baselines as well as previous works and achieves state-of-the-art performance on downstream speech recognition and speech translation tasks. in the future, we are interested in removing the dependence on a small amount of paired asr data before pre-training, and extending speechut to a multilingual model.