NAVER LABS E urope’s Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track (2023)

Abstract: This paper presents NAVER LABS Europe’s systems for Tamasheq-French and Quechua-Spanish speech translation in the IWSLT 2023 Low-Resource track. Our work attempts to maximize translation quality in low-resource settings using multilingual parameter-efficient solutions that leverage strong pre-trained models. Our primary submission for Tamasheq outperforms the previous state of the art by 7.5 BLEU points on the IWSLT 2022 test set, and achieves 23.6 BLEU on this year’s test set, outperforming the second best participant by 7.7 points. For Quechua, we also rank first and achieve 17.7 BLEU, despite having only two hours of translation data. Finally, we show that our proposed multilingual architecture is also competitive for high-resource languages, outperforming the best unconstrained submission to the IWSLT 2021 Multilingual track, despite using much less training data and compute.

## introduction the vast majority of speech pipelines are developed for high-resource languages, a small percentage of languages that have ample amounts of annotated data available (joshi et al., 2020). however, the assessment of systems' performance based only on high-resource settings can be problematic, since it fails to reflect the real-world performance these approaches will have in diverse and smaller datasets. moreover, as around half of the world's languages are considered to be not only low-resource, but also from oral tradition (i.e., without a written form), there is an urgent need for speech technology that can operate robustly in such low-resource settings (bird, 2011). in this context, the iwslt conference 1 proposes low-resource speech translation (st) challenges that allow the speech community to realistically benchmark st approaches using diverse and representative datasets. this paper describes naver labs europe's (nle) submission to two of the language pairs from the iwslt 2023 (agarwal et al., 2023) low-resource track: tamasheq-french (taq-fr) and quechua-spanish (que-es). most successful approaches for tackling scenarios where st data is scarce perform transfer learning across languages and modalities, leveraging multilingual pre-trained models for both speech and text (anastasopoulos et al., 2022). however, due to the large number of parameters of current transformer-based (vaswani et al., 2017) approaches, training such systems is computationally expensive and not accessible to everyone. nle's submission focuses on a multilingual parameterefficient training solution that allows us to leverage strong pre-trained speech and text models to maximize performance in low-resource languages. we present new sota results for the taq-fr pair (17 hours of training data) that represent a 57% bleu increase compared to the results achieved by khurana et al. (iwslt 2022 postevaluation). @xcite this same system achieves 23.6 bleu on the iwslt 2023 test set, an improvement of 7.71 bleu compared to the second best result submitted this year. we also present sota results in the unconstrained setting for the que-es pair (2 hours of training data), while maintaining most of the performance in the taq-fr pair. in addition, to showcase the usefulness of our parameter-efficient multilingual solution we evaluate it on the high-resource setting of the iwslt 2021 multilingual task (anastasopoulos et al., 2021). we find that our approach outperforms the best iwslt 2021 submission (fair, tang et al., 2021), despite training considerably fewer parameters (-64%), and using substantially less training data and compute. this paper is organized as follows. we first describe the architecture and training settings of our multilingual st systems in section 2. we next list the resources we use in section 3. section 4 presents our results in both low and high-resource settings. lastly, we highlight the zero-shot potential of our approach in section 5 and present our concluding remarks in section 6. ## system description in this work we focus on a parameter-efficient training solution that allows us to input the features from a pre-trained speech representation model into a pre-trained multilingual mt model, producing translations from both speech and text in multilingual settings. this setting also allows us to leverage automatic speech recognition (asr; i.e. speech-to-transcript) data. the general architecture is presented in figure 1 . the architecture is considered parameter-efficient because a small portion of its parameters are trained (bottom encoder layers and small adapters layers). architecture. we initialize our models with a pre-trained multilingual mt model, which we adapt to the st task by inputting features extracted with a frozen pre-trained speech representation model. the mt model is also frozen, except for the bottom 2 or 3 encoder layers and small adapter modules (those introduced by bapna and firat (2019), with bottleneck dimension 64) added after each encoder and decoder layer. as we show in our results, the fine-tuned encoder layers are able to map the speech features into the representation space of the pre-trained mt model and the adapters can help with domain adaptation (and possibly help alleviate the length mismatch). at inference, this model can be used for mt with very little memory overhead: the convolutional layers and adapters are disabled, and the bottom encoder layers are swapped with those of the initial pre-trained model. training settings. we train on 4 v100 gpus (80gb) for up to 200 000 updates, with a maximum batch size of 4 000 source features (or 80 seconds of audio) and accumulated gradients over two batches. @xcite we sample language pairs with a temperature of 3. @xcite we validate every 5 000 updates and perform early stopping on valid bleu for the language pair(s) of interest, with a patience of 5, averaging model weights across the last 3 checkpoints. @xcite we find best results using a single convolutional layer with stride 2, which downsamples the sequence of speech features by a factor of 2. the other hyperparameters are listed in appendix section a.1. ## experiments and results all our submissions to the low-resource st task are in the unconstrained setting, due to the use of pre-trained models, and from training on data in other languages. the datasets used in each submission are listed in appendix ## zero-shot capabilities throughout this paper we have argued that one advantage of the multilingual models we propose is their potential for zero-shot translation, a setting in which a system produces translation in an unseen language pair by leveraging its existing knowledge of both languages. in section 4.3 we showed that our models are competitive with the best submission to iwslt 2021 on the three zero-shot highresource language pairs, despite the fact that these pairs were not truly zero-shot for that system. in this section, we further illustrate the zero-shot capabilities of our models by translating tamasheq speech in two settings: 1) target language seen during both mt pre-training and st adaptation (english); 2) target language only seen during mt pre-training (korean). evaluation settings. to score bleu and chrf 16 in the chosen target languages, we use a commercial translation service to translate the french side of the iwslt 2022 test set to english and korean. @xcite sacrebleu signature: nrefs:1|case:mixed| eff:no|tok:x|smooth:exp|version:2.3.1, (en: @xmath0, ko: @xmath1.996/ko-0.9.2-ko). chrf signature: nrefs:1|case:mixed| eff:yes|nc:6|nw:0|space:no|version:2.3.1 note that this is only a silver-standard made of synthetic data, and thus the evaluation will inevitably be biased. @xcite our goal is solely to assess whether our systems have some zero-shot st abilities. we evaluate our taq-fr contrastive 1 system, and variants of this system with fewer or larger adapters. we compare with a cascade baseline, in which we first perform taq-fr st, followed by fr-en or fr-ko mt using the text-to-text path from figure 1 . in this setting, the adapters are disabled during mt. results. in we observe that, in the case of english, which was seen during st adaptation, adapters can be helpful (+2 bleu over the cascade baseline). on the other hand, for korean, unseen during st adaptation, systems with adapters in the decoder (first two rows) perform worse, as they likely bring some degree of language confusion. results are even worse with larger adapters, with over 40% of output sentences being in the wrong language. in this setting, the best results are achieved with only encoder adapters or no adapters at all (-1 bleu compared to the baseline). appendix utterance id target content 2016-11-23_id_7 ref chers auditeurs, rappelez-vous que vous écoutez studio kalangou en ce moment. fr chers auditeurs, n'oubliez pas que vous êtes avec le studio kalangou. en well, listeners, don't forget that you are with studio kalangou right now. ko ## conclusion in this paper we presented our parameter-efficient multilingual systems as submissions to the iwslt 2023 low-resource task in the tamasheq-french and quechua-spanish language pairs. the architecture we propose has several advantages: it is computationally and data efficient, it allows the same model to do both speech-to-text and textto-text translation (or transcription), it maximizes knowledge transfer to improve low-resource performance, and it has good zero-shot translation capabilities. our submissions reach a new state of the art performance, winning both speech translation challenges, especially for tamasheq-french, where we outperform the previous state of the art by more than 7 bleu points. future work will include a comprehensive evaluation of the asr capabilities of our architecture, and the investigation of adapters inside the speech representation model. moreover, when the speech representation model is frozen, a more in-depth analysis of the optimal layer is needed. tamasheq-french valid bleu no conv layer 1 conv layer 2 conv layers 3 conv layers task source target hours:minutes # utterances asr french french 218:59 117,081 asr italian italian 118:39 50,895 asr portuguese portuguese 179:33 91,257 asr spanish spanish 214:15 103,076 st french english 57:39 31,207 st french spanish 42:14 21,862 st french portuguese 26:53 14,322 st portuguese english 63:13 31,868 st spanish french 9:34 4,568 st spanish english 79:37 37,168 st spanish italian 11:50 5,616 st spanish portuguese 47:01 22,012