Multilingual Speech Translation with Unified Transformer: Huawei N oah’s Ark Lab at IWSLT 2021 (2021)

Abstract: This paper describes the system submitted to the IWSLT 2021 Multilingual Speech Translation (MultiST) task from Huawei Noah’s Ark Lab. We use a unified transformer architecture for our MultiST model, so that the data from different modalities (i.e., speech and text) and different tasks (i.e., Speech Recognition, Machine Translation, and Speech Translation) can be exploited to enhance the model’s ability. Specifically, speech and text inputs are firstly fed to different feature extractors to extract acoustic and textual features, respectively. Then, these features are processed by a shared encoder–decoder architecture. We apply several training techniques to improve the performance, including multi-task learning, task-level curriculum learning, data augmentation, etc. Our final system achieves significantly better results than bilingual baselines on supervised language pairs and yields reasonable results on zero-shot language pairs.

## introduction multilingual speech translation (multist) aims to develop a single system that can directly translate speech in different languages into text in many other languages. due to data scarcity of speech translation (st), multilingual and multimodal models are promising as they enable knowledge transferred from other languages and related tasks like automatic speech recognition (asr) or neural machine translation (nmt). they also allow zero-shot translation in the settings of no direct parallel data. the iwslt 2021 multist task is held for evaluating the performance under the circumstances. this paper describes our system for the task. we build a unified model for both speech-and text-related tasks, so that the knowledge from different modalities (speech and text) and different tasks (in this work, the tasks include st, asr, and nmt) can be learned together to enhance st. specifically, our model consists of three parts -feature extractor, semantic encoder and decoder. for all the tasks, the semantic encoder and the decoder will be shared to learn unified representations. it follows the transformer @xcite encoderdecoder framework to learn modality-independent features and output text representations. we use the conv-transformer @xcite as feature extractor for speech input, and the word embedding for text input. the extracted features are then fed to the semantic encoder regardless of the input modality. however, it is difficult for such a unified model to directly digest knowledge from diverse tasks. therefore, we apply task-level curriculum learning for our model. we presume the st task is more difficult than the other two tasks (asr and nmt), as it not only requires acoustic modeling to extract speech representations, but also needs alignment knowledge between different languages for translation. to this end, our training is divided into three steps -asr and nmt pre-training, joint multitask learning, and st fine-tuning. what's more, to alleviate the data scarcity problem, we also apply ctc multi-task learning @xcite , data augmentation techniques including specaugment @xcite and time stretch @xcite , and knowledge distillation @xcite , etc. we conduct experiments in the constrained setting, i.e., only the multilingual tedx @xcite dataset is used for training. it contains speech and transcripts from four languages (spanish, french, portuguese, and italian), and some of them are translated into english and/or other languages of the four mentioned above. several language pairs for st are provided without parallel training corpus and evaluated as zero-shot translation. the experimental results show that our unified model can achieve competitive results on both supervised settings and zero-shot settings. ## model architecture the architecture of our unified model, which is based on transformer @xcite , is shown in figure 1 . the nmt part (both input and output is text) follows the basic transformer setting, i.e. 6 layers for both the encoder and the decoder, each with 8 attention heads, 512 hidden dimensions, and 2048 hidden units in feed-forward layers. for the speech input, we replace the word embedding layer with the conv-transformer @xcite encoder as acoustic modeling to extract audio features, and the rest are shared. the conv-transformer encoder gradually downsamples the speech input with interleaved convolution and transformer layers. we downsample the speech input 8× times with three conv-transformer blocks, each contains three convolution layers (the stride number is 2 in the second convolution layer, and 1 in other layers) and two transformer layers. the conv-transformer is set following @xcite and also consistent with the shared parts (in terms of hidden dimensions, etc). then, the output is fed into the shared semantic encoder and decoder to produce text representations. for language encoding, we apply language projection @xcite to learn language-specific information. it replaces the language embedding in conventional multilingual models with a projec-tion matrix before the positional embedding layer. with language ids and input modality, our unified model can recognize the task that needs to be completed. for example, our model will perform asr with speech input and the same language input and output ids. ## techniques our model is trained in an end-to-end manner with all available data, including the asr data (speech and transcript) and the st data (speech, transcript and translation). from the st data, we also extract the speech-transcript pairs as asr data, and the transcript-translation pairs as nmt data. we apply task-level curriculum learning to train our model. at the same time, data augmentation, knowledge distillation, and model ensemble are used to further improve the performance. we describe the techniques in details in the rest of this section. ## conclusions we present our system submitted to iwslt 2021 for multilingual speech translation task. in our system, we build a unified transformer-based model to learn the knowledge from different kinds of data. we introduce a task-level curriculum learning procedure to enable our unified model to be trained efficiently. our results show the efficacy of our unified model to perform multilingual speech translation in both supervised settings and zero-shot settings. moreover, the results demonstrate the potential of incorporating multilingual and even multi-modal data into one powerful unified model.