S peech T 5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022)

Abstract: Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.

## introduction starting with elmo @xcite and bert @xcite , substantial work has shown that pre-trained models can significantly improve in various natural language processing (nlp) tasks figure 1 : an illustration of the speecht5 framework, which treats spoken language processing tasks as a speech/text to speech/text format, including automatic speech recognition (asr), speech translation (st), speech identification (sid), text to speech (tts), voice conversion (vc), and speech enhancement (se). @xcite @xcite @xcite . following the pre-training techniques in nlp, self-supervised speech representation learning has also been investigated and shown promising results, benefiting from richly learned representations @xcite @xcite @xcite @xcite , such as wav2vec 2.0 @xcite and hubert @xcite . however, previous speech pre-training work suffers from two problems: (1) most of them learn the speech representation with only unlabeled speech data but ignore the importance of textual data to spoken language tasks (e.g., automatic speech recognition) which require the modality transformation; (2) most of these models solely rely on a pre-trained speech encoder for various downstream tasks, leaving the decoder not pre-trained for the sequence-to-sequence generation tasks. how to design a unified encoder-decoder model that can take advantage of both unlabeled speech and text data to improve various spoken language processing tasks is not well explored. inspired by the t5 method @xcite , we attempt to formulate each spoken language processing task as a speech/text to speech/text problem via an encoder-decoder framework, which enables us to use the same pre-trained model with bimodal data across diverse tasks, as shown in figure 1 . to achieve this, we propose a unifiedmodal pre-training framework, speecht5, containing an encoder-decoder backbone network and modal-specific pre/post-nets. with the pre-nets, the input speech/text is embedded in a shared space, and the encoder-decoder backbone network models the sequence-to-sequence conversion, from which the model-specific post-nets generate the speech/text output. particularly, speecht5 is mainly pre-trained with a denoising sequence-tosequence method by leveraging large-scale unlabeled text and speech corpus. to align the textual and acoustic information into a unified semantic space, the proposed speecht5 model ( 1 ) maps text and speech representations into a shared vector quantization space, and (2) randomly mixes up the quantized latent representations and the contextual states, which can better guide the quantizer to learn the cross-modal features. we fine-tune speecht5 on a wide variety of downstream spoken language processing tasks, including automatic speech recognition (asr), textto-speech (tts), speech translation (st), voice conversion (vc), speech enhancement (se), and speaker identification (sid). massive experiments show that the proposed speecht5 model achieves a significant improvement on these spoken language processing tasks compared with the state-of-theart baselines. specifically, the proposed speecht5 outperforms wav2vec 2.0 @xcite and hubert @xcite with the base model on the asr task and also performs better than the state-of-the-art voice transformer network @xcite on the vc task. besides, speecht5 is significantly superior to speechnet @xcite and pre-trained models from superb @xcite and achieves the stateof-the-art performance (i.e., 96.49%) on the sid task. we further provide an empirical comparison of the pre-training tasks and modules, and the ablation study demonstrates the effectiveness of the proposed joint speech-text pre-training method. the contributions of this paper are summarized as follows. ## speecht5 in this section, we propose speecht5, a unifiedmodal framework for learning joint contextual representations for speech and text data via a shared encoder-decoder structure. ## related work large-scale pre-training models such as bert @xcite , t5 @xcite , wav2vec 2.0 @xcite , and hubert @xcite have drawn much attention in the nlp and speech communities, due to its strong capabil-ity of generalization and efficient usage of largescale data @xcite @xcite @xcite @xcite @xcite . however, the research mentioned above effects gear towards single-modal learning, hence they can only be used in either text or speech modeling. although some speech-language pre-training work @xcite @xcite attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-specific layers for different tasks and do not pre-train a decoder for generation tasks such as speech synthesis or text generation. besides, a series of research work begins to investigate joint text and speech training @xcite @xcite @xcite ), but they are mainly designed for speech to text tasks. the proposed speecht5 method is most related to t5 @xcite . the core idea of the t5 model, a unified framework for a variety of text-based language problems, is to treat every text processing problem as a "text-to-text" problem. speecht5 is also related to speech chain @xcite , which leverages the asr model and tts model to build a closed-loop machine speech chain to train models on the concatenation of both labeled and unlabeled data, and speechnet @xcite , which designs a universal modularized model to perform multiple speech processing tasks with multi-task learning. the differences from the above models are that (1) speecht5 is a shared cross-modal encoder-decoder framework, whose input and output are speech or text through multiple pre/post-nets; (2) speecht5 attempts to pre-train and improve the universal model with large-scale unlabeled text and speech data. another related work is superb @xcite , a benchmark to examine the capability of pre-trained models such as hubert @xcite . superb focuses on investigating a simple framework to learn superb tasks with a frozen and shared pre-trained encoder and lightweight prediction modules fine-tuned for each task. in contrast, the goal of speecht5 is to learn all spoken language processing tasks by fine-tuning a unifiedmodal encoder-decoder model, which is pre-trained on unlabeled speech and text corpus. ## conclusion in this paper, we have proposed speecht5 as a pretrained encoder-decoder model for various spoken language processing tasks. we convert all spoken language processing tasks into a speech/text to speech/text format and propose a novel joint pretraining method to utilize cross-modal information by leveraging the unlabeled speech and text data. the proposed unified encoder-decoder model can support generation tasks such as speech translation and voice conversion. massive experiments show that speecht5 significantly outperforms all baselines in several spoken language processing tasks. in the future, we are going to pre-train the speecht5 with a larger model and more unlabeled data. we are also interested in extending the proposed speecht5 framework to address multilingual spoken language processing tasks for future work.