Motivated by the success of T5 ( Text-To-Text Transfer Transformer ) in pre-trained natural language processing models , we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech / text representation learning . The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific ( speech / text ) pre / post-nets . After preprocessing the input speech / text through the pre-nets , the shared encoder-decoder network models the sequence-to-sequence transformation , and then the post-nets generate the output in the speech / text modality based on the output of the decoder . Leveraging large-scale unlabeled speech and text data , we pre-train SpeechT5 to learn a unified-modal representation , hoping to improve the modeling capability for both speech and text . To align the textual and speech information into this unified semantic space , we propose a cross-modal vector quantization approach that randomly mixes up speech / text states with latent units as the interface between encoder and decoder . Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks , including automatic speech recognition , speech synthesis , speech translation , voice conversion , speech enhancement , and speaker identification . We release our code and model at https://github.com / microsoft/ SpeechT5 . Starting with ELMo ( Peters et al . , 2018 ) and BERT ( Devlin et al . , 2019 ) , substantial work has shown that pre-trained models can significantly improve in various natural language processing ( NLP ) tasks Figure 1 : An illustration of the SpeechT5 framework , which treats spoken language processing tasks as a speech / text to speech / text format , including automatic speech recognition ( ASR ) , speech translation ( ST ) , speech identification ( SID ) , text to speech ( TTS ) , voice conversion ( VC ) , and speech enhancement ( SE ) . ( Radford et al . , 2019 ; CONNEAU and Lample , 2019 ; Yang et al . , 2019 ; Dong et al . , 2019 ; Lewis et al . , 2020 ) . Following the pre-training techniques in NLP , self-supervised speech representation learning has also been investigated and shown promising results , benefiting from richly learned representations ( Chung and Glass , 2018 ; Chuang et al . , 2020 ; Song et al . , 2019 ; Baevski et al . , 2020 ; Wang et al . , 2021 ; Hsu et al . , 2021 ; Chung et al . , 2021a ) , such as wav2vec 2.0 ( Baevski et al . , 2020 ) and HuBERT ( Hsu et al . , 2021 ) . However , previous speech pre-training work suffers from two problems : ( 1 ) most of them learn the speech representation with only unlabeled speech data but ignore the importance of textual data to spoken language tasks ( e.g. , automatic speech recognition ) which require the modality transformation ; ( 2 ) most of these models solely rely on a pre-trained speech encoder for various downstream tasks , leaving the decoder not pre-trained for the sequence-to-sequence generation tasks . How to design a unified encoder-decoder model that can take advantage of both unlabeled speech and text data to improve various spoken language processing tasks is not well explored . Inspired by the T5 method ( Raffel et al . , 2020 ) , we attempt to formulate each spoken language processing task as a speech / text to speech / text problem via an encoder-decoder framework , which enables us to use the same pre-trained model with bimodal data across diverse tasks , as shown in Figure 1 . To achieve this , we propose a unifiedmodal pre-training framework , SpeechT5 , containing an encoder-decoder backbone network and modal-specific pre / post-nets . With the pre-nets , the input speech / text is embedded in a shared space , and the encoder-decoder backbone network models the sequence-to-sequence conversion , from which the model-specific post-nets generate the speech / text output . Particularly , SpeechT5 is mainly pre-trained with a denoising sequence-tosequence method by leveraging large-scale unlabeled text and speech corpus . To align the textual and acoustic information into a unified semantic space , the proposed SpeechT5 model ( 1 ) maps text and speech representations into a shared vector quantization space , and ( 2 ) randomly mixes up the quantized latent representations and the contextual states , which can better guide the quantizer to learn the cross-modal features . We fine-tune SpeechT5 on a wide variety of downstream spoken language processing tasks , including automatic speech recognition ( ASR ) , textto-speech ( TTS ) , speech translation ( ST ) , voice conversion ( VC ) , speech enhancement ( SE ) , and speaker identification ( SID ) . Massive experiments show that the proposed SpeechT5 model achieves a significant improvement on these spoken language processing tasks compared with the state-of-theart baselines . Specifically , the proposed SpeechT5 outperforms wav2vec 2.0 ( Baevski et al . , 2020 ) and HuBERT ( Hsu et al . , 2021 ) with the BASE model on the ASR task and also performs better than the state-of-the-art voice Transformer network ( Huang et al . , 2021 ) on the VC task . Besides , SpeechT5 is significantly superior to SpeechNet ( Chen et al . , 2021b ) and pre-trained models from SUPERB ( Yang et al . , 2021 ) and achieves the stateof-the-art performance ( i.e. , 96.49 % ) on the SID task . We further provide an empirical comparison of the pre-training tasks and modules , and the ablation study demonstrates the effectiveness of the proposed joint speech-text pre-training method . The contributions of this paper are summarized as follows . • To the best of our knowledge , this is the first work to investigate a unified encoder-decoder framework for various spoken language processing tasks . • We propose a cross-modal vector quantization approach , which learns the implicit alignment between acoustic and textual representation with large-scale unlabeled speech and text data . • Extensive experiments on spoken language processing tasks demonstrate the effectiveness and superiority of the proposed SpeechT5 model . In this paper , we have proposed SpeechT5 as a pretrained encoder-decoder model for various spoken language processing tasks . We convert all spoken language processing tasks into a speech / text to speech / text format and propose a novel joint pretraining method to utilize cross-modal information by leveraging the unlabeled speech and text data . The proposed unified encoder-decoder model can support generation tasks such as speech translation and voice conversion . Massive experiments show that SpeechT5 significantly outperforms all baselines in several spoken language processing tasks . In the future , we are going to pre-train the SpeechT5 with a larger model and more unlabeled data . We are also interested in extending the proposed SpeechT5 framework to address multilingual spoken language processing tasks for future work .

Challenge: Existing speech pre-training methods ignore the importance of textual data and solely depend on encoders leaving the decoder out of pre-training for generation tasks.
Approach: They propose a unified-modal encoder-decoder framework with shared and modal-specific networks for self-supervised speech and text representation learning by using unlabeled text and speech corpus.
Outcome: The fine-tuned proposed model is evaluated on a variety of spoken processing tasks and outperforms state-of-the-art models on voice conversion speaker identification tasks.