E dinburgh’s End-to-End Multilingual Speech Translation System for IWSLT 2021 (2021)

Abstract: This paper describes Edinburgh’s submissions to the IWSLT2021 multilingual speech translation (ST) task. We aim at improving multilingual translation and zero-shot performance in the constrained setting (without using any extra training data) through methods that encourage transfer learning and larger capacity modeling with advanced neural components. We build our end-to-end multilingual ST model based on Transformer, integrating techniques including adaptive speech feature selection, language-specific modeling, multi-task learning, deep and big Transformer, sparsified linear attention and root mean square layer normalization. We adopt data augmentation using machine translation models for ST which converts the zero-shot problem into a zero-resource one. Experimental results show that these methods deliver substantial improvements, surpassing the official baseline by > 15 average BLEU and outperforming our cascading system by > 2 average BLEU. Our final submission achieves competitive performance (runner up).

## introduction although end-to-end (e2e) speech translation (st) has achieved great success in recent years, outperforming its cascading counterpart and delivering state-of-the-art performance on several benchmarks @xcite @xcite , it still suffers from the relatively low amounts of dedicated speech-to-translation parallel training data @xcite . in textbased machine translation (mt), one solution to lack of training data is to jointly perform multilingual translation with the benefit of transferring knowledge across similar languages and to lowresource directions, and even enabling zero-shot translation, i.e. direct translation between language pairs unseen in training @xcite . however, whether and how to obtain similar success in very low-resource (and practical) scenario for multilingual st with e2e models remains an open question. to address this question, we participated in the iwslt2021 multilingual speech translation task, which focuses on low-resource st language pairs in a multilingual setup. apart from supervised evaluation, the task also offers zero-shot condition with a particular emphasis where only automatic speech recognition (asr) training data is provided for some languages (without any direct st parallel data). the task is organized in two settings: constrained setting and unconstrained setting. the former restricts participants to use the given multilingual tedx data @xcite alone for experiment; while the latter allows for additional asr/st/mt/others training data. in this paper, we address the constrained one. our e2e multilingual st model takes transformer @xcite as the backbone, and follows the adaptive feature selection (afs) framework @xcite as shown in figure 1 . afs is capable of filtering out uninformative speech features contributing little to asr, effectively reducing speech redundancy and improving st performance @xcite . we adapt afs to multilingual st, and further incorporate several techniques that encourage transfer learning and larger capacity modeling, ranging from language-specific modeling, multi-task learning, deep and big transformer, sparsified linear attention (rela) @xcite to root mean square layer normalization (rmsnorm) @xcite . inspired by @xcite , we convert the zero-shot translation problem into a zero-resource one via data augmentation with multilingual mt models. ## methods in this section, we elaborate crucial ingredients in our e2e multilingual st, which individually have already been proven successful for st or (multilingual) mt. we put them together to improve multilingual st as shown in figure 1 . note all encoder/decoder modules are based on transformer @xcite . ## experimental settings in this section, we explain the used datasets, model architectures, optimization details and evaluation metrics in our experiments. all implementations are based on the zero 2 toolkit @xcite . ## submission results the iwslt2021 task prepares a held-out test set for the final evaluation. we submitted three systems: one cascading system, one e2e single model (w/ big st transformer) and one ensemble model. results are shown in ## conclusion and future work we describe edinburgh's end-to-end multilingual speech translation system for the iwslt2021 multilingual speech translation task. we observe substantial performance improvement using largercapacity modeling (deep or big modeling) and data augmentation. in spite of the scarcity of the training data, we show that e2e models benefit greatly from multilingual modeling and deliver promising results on zero-shot translation directions (even without data augmentation). our e2e multilingual st greatly surpasses its cascading counterpart. regarding future study, we argue that exploring the multilingual transfer behavior should be very practical and promising to st. this work mainly studies transfer across similar languages. how the current model generalizes to distant languages is still an open question. besides, a general trend for deep learning is to increase the model capacity via deep and/or big modeling. however, deep models for st seem to be ineffective. identifying the reason for this and proposing simple solutions would be of high interest.