ZJU ’s IWSLT 2021 Speech Translation System (2021)

Abstract: In this paper, we describe Zhejiang University’s submission to the IWSLT2021 Multilingual Speech Translation Task. This task focuses on speech translation (ST) research across many non-English source languages. Participants can decide whether to work on constrained systems or unconstrained systems which can using external data. We create both cascaded and end-to-end speech translation constrained systems, using the provided data only. In the cascaded approach, we combine Conformer-based automatic speech recognition (ASR) with the Transformer-based neural machine translation (NMT). Our end-to-end direct speech translation systems use ASR pretrained encoder and multi-task decoders. The submitted systems are ensembled by different cascaded models.

## introduction in this paper, we introduce our submission to the iwslt2021 multilingual speech translation task. this task focuses on speech translation (st) research across many non-english source languages. multilingual models enable transfer from related tasks, which is particularly important for lowresource languages; however, parallel data between two otherwise high-resource languages can often be rare, making multilingual translation and zeroshot translation important for many resource settings. the task provides data for two conditions @xcite : supervised, and zero-shot, including speech and transcripts for four languages (spanish, french, portuguese, italian) and translations in a subset of five languages (english, spanish, french, portuguese, italian). at evaluation time, using the provided speech in the four source languages, participants submit the generated translations in both english and spanish. in the cascaded approach, we use a conformer @xcite ) model for asr for every language. for the mt component, we use a unified transformer model for all language pairs. as previous works @xcite , we use both the clean and noisy speech transcripts, back translation data, and the mask noisy trick. for the end-to-end direct speech translation, we also created a transformer-based model. to obtain the best possible translation quality, we apply data augmentation on audio files, make a multitask decoding for incorporating the asr task @xcite . we tried various experimental parameter settings and different architectures, and finally submitted an ensembled cascaded system. ## cascaded speech translation as the task provides speech and transcripts for four languages (spanish, french, portuguese, italian) and translations in a subset of five languages (english, spanish, french, portuguese, italian). zeroshot language pairs have asr data released for training but not translations. cascades of separately trained automatic speech recognition and machine translation (mt) models can leverage all of these data sources. ## end-to-end direct speech translation we used fairseq to train end-to-end transformerbased models for st, using 80-dimensional melfilterbank features with global cepstral mean and variance normalization (cmvn), specaugment @xcite , and 1-d convolutions downsampler with the pretrained transformer-based asr model. we remove all text sequences longer than 200 tokens and all speech utterances longer than 6000 frames. in order to make full use of the speech translation data of all language pairs, we adopt a joint vocabulary of 10k for all language pairs. in the beginning, we used the asr model trained with all 4 languages asr corpus to pre-train the st, but in the end, the asr model trained with just 1 language was used to pre-train the st and the latter result was better. same as the multilingual machine translation model, we prepend the source language id tag to the frame sequence after the down-sampling of 1-d cnn layers. at the same time, we also prepend the target language id tag to the target text token sequence. we augmented the data by processing the audio files with two sox's effects as potapczyk and przybysz (2020): tempo, speed. we sampled the parameters with uniform distribution within ranges presented in as in many previous works, we also introduced a second decoder with asr task, making it a multitask setup similar to @xcite . the asr and st tasks use a joint dictionary of size 10k as the baseline. the training loss can be calculated as follows: we tried setting the value of α to 0.7 and 0.5, and the result was better when it was set to 0.5. thus, the asr and st decoder are trained jointly, and convolutional layers and encoder are shared. the experiments also proved that this kind of multi-task learning is useful. all the models consist of 12 encoder layers and 6 decoder layers, including the multi-task model. for the one encoder-one decoder baseline, we just pretrain the encoder. for the multi-task model, we use the pretrained asr model to initialize the shared encoder and asr decoder. we also tried to pretrain only the shared encoder of the multi-task model. our experimental results show that pretraining the asr decoder will not improve the final effect of speech translation, but it can reduce the loss of the asr decoder and the convergence time of training. we also tried to increase the number of encoder layers from 12 to 16, and the translation performance almost did not improve, but the number of convergence epochs decreased. ## experiments in this section, we report the results for cascaded and end-to-end direct speech translation models on various data and settings. for the asr task, we tried 2 different platforms, the results as for the cascaded speech translation models, all mt models have used the mask tokens trick, the main difference is just the different adding data. for the end-to-end direct speech translation models, all the models including the pretrained asr models are trained including the sox's effects data. all the parameter settings are almost unchanged. the asr model trained with just 1 language (spanish) was used to pretrain the st. we tried only using spanish or french asr to pretrain the st model, compared with all 4 multilingual asr. using mul-tilingual asr initialization led to a decrease of nearly 2.9 bleu on es-en testset with only es asr. pretraining with one language asr is better than with all four languages, which surprised us a bit. we originally felt that the performance of the richer corpus model should be better. perhaps understanding this problem will help improve the effectiveness of the multilingual end-to-end model. our multilingual translation model and end-toend multilingual speech translation model both adopt a unified model for all language pairs, and do not apply special processing to individual language pairs.