Unified Speech-Text Pre-training for Speech Translation and Recognition (2022)

Abstract: In this work, we describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method utilizes multi-task learning to integrate four self-supervised and supervised subtasks for cross modality learning. A self-supervised speech subtask, which leverages unlabelled speech data, and a (self-)supervised text to text subtask, which makes use of abundant text training data, take up the majority of the pre-training time. Two auxiliary supervised speech tasks are included to unify speech and text modeling space. Detailed analysis reveals learning interference among subtasks. In order to alleviate the subtask interference, two pre-training configurations are proposed for speech translation and speech recognition respectively. Our experiments show the proposed method can effectively fuse speech and text information into one model. It achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task.

## introduction pre-training can learn universal feature representations from a large training corpus and is beneficial for downstream tasks with limited amounts of training data @xcite @xcite . with the advancement of computational power and self-supervised pre-training approaches, large volumes of unlabeled data may now be used in pre-training. methods, such as bert @xcite , bart @xcite and wav2vec2.0 @xcite , have emerged as the backbone of many speech and natural language processing tasks. the aforementioned pre-training methods focus on learning feature representation either from text or speech. many speech applications combine information learnt from both speech and text corpora to achieve state of the art results. in speech processing, transcribed speech training data is generally very scarce for many languages. it is difficult to build robust linguistic knowledge representation solely based on labeled speech training data. @xcite ; @xcite propose to generate synthetic data from text to augment speech training corpus. @xcite demonstrate that models initialized with pre-trained wav2vec2.0 and mbart @xcite modules are competitive for the multilingual speech to text translation task. @xcite propose to concatenate the acoustic model and bert model for speech q&a. @xcite align speech utterance representation to the corresponding text sentence representation, in which both representations are generated from unsupervised pre-trained models, for speech understanding. in this study, we are interested in pre-training for speech to text tasks using the attention based encoder-decoder (aed) framework. in particular, we seek to answer the question whether the integration of data from different modalities is beneficial for representation learning. to answer this question, we propose speech and text joint pre-training (stpt), a multi-task learning framework to combine different modalities, i.e., speech and text, in the pre-training stage. a self-supervised speech subtask and a (self-)supervised text to text subtask dominate the pre-training computation to leverage large amounts of unlabelled speech data and abundant text training corpus. two auxiliary supervised speech subtasks are used to unify different modalities in the same modeling space. the proposed method fuses information from the text and speech training corpus into a single model, and it effectively improves the performance of down-stream tasks, such as speech to text translation (st) and automatic speech recognition (asr). our contributions are summarized as follows: pre-training: self-supervised pre-training is usually optimized with two different criteria: contrastive loss @xcite @xcite and masked prediction loss @xcite . contrastive loss focuses on distinguishing the positive samples from the negative ones given the reference sample and it has achieved great success for speech recognition @xcite . masked prediction loss has been first studied for natural language processing tasks @xcite with subsequent application to speech processing @xcite . @xcite combine contrastive loss and masked prediction loss, which shows good performance for the downstream asr task. the optimization of our self-supervised speech task is more related to the masked prediction loss. instead of predicting the hard discretized label for the masked frames, which is error prone, we use kl divergence to minimize the distribution difference between the same feature frames with and without masking. please refer to subsection 3.2 for more details. self-training (or iterative pseudo labelling): self-training is another widely used approach to take advantage of unlabelled speech data to improve the asr performance @xcite @xcite @xcite @xcite . a seed model, which usually is trained with a small amount of supervised speech training data, is employed to generate pseudo labels for the unlabelled speech data. the speech data with pseudo labels is augmented into the training dataset to build another model, which is expected to outperform the seed model due to more training data exposure. similar to self-training, we also use small amounts of supervised data to unify the speech and text modeling space. however, the self-supervised speech training in this work avoids making hard predictions and uses kl divergence to maximize the mutual information between masked span and observed feature frames. multi-task learning: due to data scarcity, multitask learning is widely adopted to leverage parallel text training data for st @xcite @xcite . those methods primarily use supervised speech data sets during multi-task learning, whereas our method can leverage large amounts of unlabeled speech data during the pre-training stage, which has the potential to improve performance even further. a concurrent work from @xcite also proposes to jointly pre-train speech and text for asr and text to speech application, which is fully unsupervised. our method focuses on taking advantage of the supervised speech data, which could be the same data used for fine-tuning, to improve the joint speech text pre-training. our results demonstrate the efficacy of supervised speech data in pretraining. another concurrent work is from @xcite , which focuses on speech encoder pre-training using both speech and text data. our method emphasizes the encoder-decoder framework and training both encoder and decoder in the pre-training stage. the choice of the t2t subtask depends on the downstream task. for asr, the t2t subtask is a denoising autoencoder task (bart) @xcite while st utilizes a text based neural machine translation task. the ssl subtask is a selfsupervised speech learning task to leverage large amounts of unlabelled speech data optimized by the masked prediction loss. the last two supervised speech tasks (s2p and s2t) unify two modalities, i.e., speech and text, into one modeling space. in this study, we find that the subtasks for the asr pre-training are complementary, while subtask interference is observed in the st pre-training at some encoder layers. we propose two different configurations: fully shared encoder (fse) (figure 1(a) ) for the asr pre-training, and partially shared encoder (pse) (figure 1(b) ) for the st pre-training. the fse configuration aims to encourage information sharing between different subtasks while the pse configuration tries to minimize the information sharing between encoder only subtasks, i.e., ssl and s2p, and sequence to sequence aed tasks, i.e., subtask t2t and s2t. more subtask interference analysis is presented in subsection 5.2. we describe the details of each subtask in the following subsections. ## experimental setting in the pre-training, we first train modules with the t2t subtask until they are converged. it helps to stabilize the training and achieve a better result. then the entire model is jointly optimized with all subtasks mentioned in section 3. finally, the pre-trained model is fine-tuned on the downstream tasks. in the fine-tuning stage, we keep optimizing the model with the t2t and s2t subtasks. two encoder-only subtasks (ssl and s2p) are dropped, since the model has learnt good speech representation from the unlabeled speech data in pre-training. two downstream tasks, asr and st, are examined. the asr system is evaluated on four librispeech @xcite evaluation sets: dev-clean, dev-other, test-clean and test-other. wer is reported in the experiments. st models are evaluated on two translation directions: english-spanish (en-es) and english-french (en-fr). case-sensitive detokenized sacrebleu @xcite is reported on the tst-common testset from @xcite . for both asr and st pre-training, 60k hours of unlabelled english speech data from librilight @xcite ) is used to build the selfsupervised speech task if not specifically mentioned. we employ the same labelled data for the supervised learning in pre-training and fine-tuning, i.e., librispeech training data for asr and must-c for st. for asr pre-training, the lib-rispeech language model (lm) training dataset is used to build the monolingual bart model. for st pre-training, we take the parallel training corpus from wmt. more details about the training data could be found in appendix a. ## conclusion in this work, we present a method to jointly pretrain speech and text in one model for speech translation and recognition under the aed framework. config. librispeech (wer ↓) must-c (bleu ↑) dev clean dev other en-es en-fr stpt 2.0 4.4 33.1 39.7 -t2t pt 2.4 5.0 31.9 39.2 -aed task 2.9 5.6 31.3 38.0 -joint pt 2.8 6.4 30.6 35.4