Curriculum Pre-training for End-to-End Speech Translation (2020)

Abstract: End-to-end speech translation poses a heavy burden on the encoder because it has to transcribe, understand, and learn cross-lingual semantics simultaneously. To obtain a powerful encoder, traditional methods pre-train it on ASR data to capture speech features. However, we argue that pre-training the encoder only through simple speech recognition is not enough, and high-level linguistic knowledge should be considered. Inspired by this, we propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks.

## introduction speech-to-text translation (st) is essential to breaking the language barrier for communication. it aims to translate a segment of source language speech to the target language text. to perform this task, prior works either employ a cascaded method, where an automatic speech recognition (asr) model and a machine translation (mt) model are chained together, or an end-to-end approach, where a single model converts the source language audio sequence to the target language text sequence directly @xcite . due to the alleviation of error propagation and lower latency, the end-to-end st model has been a hot topic in recent years. however, large paired data of source audios and target sentences are required to train such a model, which is not easy to satisfy for most language pairs. to address this issue, previous works resort to pre-training technique @xcite , where they leverage the available asr and mt data to pre-train an asr model and an mt model respectively, and then initialize the st model with the asr encoder and the mt decoder. this strategy can bring faster convergence and better results. the end-to-end st encoder has three essential roles: transcribe the speech, extract the syntactic and semantic knowledge of the source sentence and then map it to a semantic space, based on which the decoder can generate the correct target sentence. these pose a heavy burden to the encoder, which can be alleviated by pre-training. however, we argue that the current pre-training method restricts the power of pre-trained representations. the encoder pre-trained on the asr task mainly focuses on transcription, which learns the alignment between the acoustic feature with phonemes or words. it cannot capture linguistic knowledge or understand the semantics, which is essential for translation. in order to teach the model to understand the sentence and incorporate the required knowledge, extra courses should be taken before learning translation. motivated by this, we propose a curriculum pre-training method for end-to-end st. as shown in figure 1 , we first teach the model transcription through asr task. after that, we design two tasks, named frame-based masked language model (fmlm) task and frame-based bilingual lexicon translation (fblt) task, to enable the encoder to understand the meaning of a sentence and map words in different languages. finally, we fine-tune the model on st data to obtain the translation ability. for the fmlm task, we mask several segments of the input speech feature, each of which corresponds to a complete word. then we let the encoder predict the masked word. this task aims to force the encoder to recognize the content of the utterance and understand the inner meaning of the sentence. in fblt, for each speech segment that aligns with a complete word, whether or not it is masked, we ask the encoder to predict the corresponding target word. in this task, we give the model more explicit and strong cross-lingual training signals. thus, the encoder has the ability to perform simple word translation, and the burden on the st decoder is largely reduced. besides, we adopt a hierarchical manner where different layers are guided to perform different tasks (first 8 layers for asr and fmlm pre-training, and another 4 layers for fblt pre-training). this is mainly because the three pre-training tasks have different requirements for language understanding and different output spaces. the hierarchical pre-training method can make the division of labor more clear and separate the incorporation of source semantic knowledge and cross-lingual alignments. we conduct experiments on the librispeech en-fr and iwslt18 en-de speech translation tasks, demonstrating the effectiveness of our pre-training method. the contributions of our paper are as follows: (1) we propose a novel curriculum pretraining method with three courses: transcription, understanding and mapping, forcing the encoder to have the ability to generate necessary features for the decoder. (2) we propose two new tasks to learn linguistic features, fmlm and fblt, which explicitly teach the encoder to do source language understanding and target language meaning mapping. (3) experiments show that both the proposed courses are helpful for speech translation, and our proposed curriculum pre-training leads to significant improvements. ## conclusion and future work this paper investigates the end-to-end method for st. we propose a curriculum pre-training method, consisting of an elementary course with an as-r loss, and two advanced courses with a framebased masked language model loss and a bilingual lexicon translation loss, in order to teach the model syntactic and semantic knowledge in the pre-training stage. empirical studies have demonstrated that our model significantly outperforms baselines. in the future, we will explore how to leverage unlabeled speech data and large bilingual text data to further improve the performance. besides, we expect the idea of curriculum pre-training can be adopted on other nlp tasks.