End-to-End Simultaneous Speech Translation with Pretraining and Distillation: Huawei N oah’s System for A uto S im T ran S 2022 (2022)

Abstract: This paper describes the system submitted to AutoSimTrans 2022 from Huawei Noah’s Ark Lab, which won the first place in the audio input track of the Chinese-English translation task. Our system is based on RealTranS, an end-to-end simultaneous speech translation model. We enhance the model with pretraining, by initializing the acoustic encoder with ASR encoder, and the semantic encoder and decoder with NMT encoder and decoder, respectively. To relieve the data scarcity, we further construct pseudo training corpus as a kind of knowledge distillation with ASR data and the pretrained NMT model. Meanwhile, we also apply several techniques to improve the robustness and domain generalizability, including punctuation removal, token-level knowledge distillation and multi-domain finetuning. Experiments show that our system significantly outperforms the baselines at all latency and also verify the effectiveness of our proposed methods.

## introduction simultaneous speech translation (st) task @xcite aims to translate speech into the corresponding text in another language while reading the source speech. prior works mainly focus on the cascaded solution, i.e., first recognize the speech with a streaming asr model and then translate into the target language with simultaneous nmt @xcite model. such cascaded systems can leverage off-the-shelf asr and nmt systems, which have large-scale data for training. recently, end-to-end simultaneous st models are also proposed @xcite and have shown promising improvements towards cascaded models when experimented on the same amount of data, especially in low latency requirement. end-to-end models are believed to have the advantages of lower latency, smaller model size and less error propagation @xcite , but suffer from data scarcity. a well-trained end-toend model typically needs a large amount of training data. to alleviate the data scarcity problem, pretraining @xcite and data augmentation @xcite are two main techniques. we examine the effectiveness of the two techniques for improving end-to-end models in this work. specifically, our end-to-end st model follows realtrans @xcite , an encoder-decoder model and the encoder is decoupled into acoustic encoder and semantic encoder. the acoustic encoder is used to extract acoustic features which has a similar function as the asr encoder. therefore we initialize it with a pretrained asr encoder. the semantic encoder is required to learn semantic knowledge, which benefits the translation task, so we initialize it with a pretrained nmt encoder. the decoder is also initialized with a pretrained nmt decoder to produce target text decoding. for data augmentation, we construct pseudo st corpus based on asr data and the pretrained nmt model. the ground-truth transcription is translated into target language texts, and so speech-transcriptiontranslation triplets for st training are built. this is also known as sequence-level knowledge distillation @xcite . generally, the nmt data can also be augmented with a tts model to generate pseudo speech. however, the data quality highly depends on the tts performance and it is hard for tts to produce voices similar to those in real scenarios. thus we do not utilize this method and leave it to the future work. another popular technique for audio data augmentation is specaugment @xcite , which randomly masks a block of consecutive time steps and/or mel frequency channels of the input speech features during training. it is a simple and lowimplementation cost method and has been shown effective in avoiding overfitting and improving robustness. we apply it to all audio-related model training. the training procedure for our st model mainly contains three steps: asr and nmt pretraining, large-scale training on the constructed pseudo data, and finetuning on the in-domain data. during training, we remove all the punctuation in source text in audio-related training (i.e., excluding nmt pretraining) to relieve the learning burden and improve recognition quality. to enhance the final performance after finetuning, we also utilize token-level knowledge distillation from the full-sentence nmt model and multi-domain finetuning trick. our model is used to participate in the audio input track of the autosimtrans 2022 chinese-english translation task. in this track, an in-domain st data called bstc @xcite (contains about 70 hours of audios) is provided, which is very limited. therefore, assisted with extra asr and nmt data, we use the aforementioned techniques to achieve remarkable improvement at all latency requirement, which results in winning the first place of the track. we also conduct more experiments to examine the effectiveness of our used techniques. the experiments show that all of our used methods contribute to the improvement of the final model. ## model description we build our model based on realtrans @xcite , an end-to-end simultaneous speech translation model with its encoder decoupled into acoustic encoder and semantic encoder (see figure 1 ). with a ctc module guiding the acoustic encoder to produce acoustic-level features, the decoupling relieves the burden of the st encoder and makes the two separate modules focus on different knowledge, which benefits the model training. realtrans leverages the unidirectional conv-transformer @xcite as the acoustic encoder for gradual downsampling, and weightedshrinking for bridging the modality gap between speech and text. with weighted-shrinking, long speech features are shrunk to similar lengths as their corresponding transcription, which makes the input of the semantic encoder more similar to the input of nmt encoder. in this way, the difficulty of knowledge transferring when we initialize the semantic encoder with nmt encoder becomes smaller. apart from the semantic encoder, we also initialize the acoustic encoder with pretrained asr encoder, and the decoder with pretrained nmt decoder, which has been shown very useful in boosting the performance @xcite . for simultaneous policy, we use the wait-kstride-n policy @xcite , which has shown promising improvement over the conventional wait-k policy @xcite . ## training procedure our model training consists of three steps: asr and nmt pretraining, large-scale training on the constructed pseudo data, and finetuning on the indomain data. each step may contain different techniques to enhance model performance and we will describe them in-detailed as follows. ## conclusion in this work, we describe the details of our submitted system to autosimtrans 2022, which won the first place in chinese-english audio input track. our model is based on the end-to-end simultaneous speech translation model realtrans and follows three-step training procedure, including asr and nmt pretraining, large-scale training on the pseudo data and finetuning on the in-domain data. our experiments proves the superiority of our model and training procedure and also examines the effectiveness of different techniques like punctuation removal and multi-domain finetuning.