ON - TRAC Consortium Systems for the IWSLT 2023 Dialectal and Low-resource Speech Translation Tasks (2023)

Abstract: This paper describes the ON-TRAC consortium speech translation systems developed for IWSLT 2023 evaluation campaign. Overall, we participated in three speech translation tracks featured in the low-resource and dialect speech translation shared tasks, namely; i) spoken Tamasheq to written French, ii) spoken Pashto to written French, and iii) spoken Tunisian to written English. All our primary submissions are based on the end-to-end speech-to-text neural architecture using a pretrained SAMU-XLSR model as a speech encoder and a mbart model as a decoder. The SAMU-XLSR model is built from the XLS-R 128 in order to generate language agnostic sentence-level embeddings. This building is driven by the LaBSE model trained on multilingual text dataset. This architecture allows us to improve the input speech representations and achieve significant improvements compared to conventional end-to-end speech translation systems.

## introduction iwslt is a unique opportunity that allows each year the assessment of progress made in the area of spoken language translation (slt). this assessment is made possible throughout the organisation of an evaluation campaign including various shared tasks that address specific scientific challenges of the slt domain. in addition to the well-established shared tasks, iwslt organisers introduce new tasks to address the many challenges settings related to slt area like data scarcity, multilingualism, time and computation constraints, etc. in this context, the iwslt 2023 proposes two interesting shared tasks: low-resource and dialect speech translation (st). the former aims to assess the exploitability of current translation systems in data scarcity settings. the latter focuses on the assessment of the systems' capabilities in noisy settings: different dialects are mixed in a single dataset of spontaneous speech. for the low-resource task, several language pairs were proposed this year. in this paper, we focus on tamasheq-french, tunisian arabic-english and pashto-french. this paper reports the on-trac consortium submissions for the aforementioned tasks. the on-trac consortium is composed of researchers from three academic laboratories, lium (le mans university -france), lia (avignon university -france), mit (cambridge -usa) together with three industrial partners: airbus france, elya-data and systran. our systems for the dialect task focus on both cascaded and end-to-end approaches for st. for the low-resource task, we focus on the leveraging of models based on self-supervised learning (ssl), and on the training of st models with joint automatic speech recognition (asr), machine translation (mt) and st losses. this paper is organized as follows. section 2 presents the related work. section 3 is dedicated to detail our primary systems encoder-decoder approach. the experiments with the tunisian arabic-english dataset for low-resource and dialect st tasks are presented in section 4. results for the tamasheq-french and pashto-french tracks are presented in section 5 and 6 respectively. section 7 concludes the paper and discusses future work. ## related work before the introduction of direct or end-to-end st models @xcite , the st task was approached as a cascaded problem: the speech is transcribed using an asr model, and the transcriptions are used to train a classic mt model. the limitations of this approach include the need for extensive transcriptions of the speech signal, and the error propagation between asr and mt modules. in comparison to that, end-to-end st models offer a simpler encoder-decoder architecture, removing the need for intermediate representations of the speech signal. although at first, cas-caded models were superior in performance compared to end-to-end models, results from recent iwslt campaigns illustrate how end-to-end models have been closing this gap @xcite @xcite . moreover, the joint optimization of asr, mt and st losses in end-to-end st models was shown to increase overall performance @xcite . furthermore, ssl models for speech processing are now a popular foundation blocks in speech pipelines @xcite @xcite . these models are large trainable networks with millions, or even billions @xcite , of parameters that are trained on unlabeled audio data only. the goal of training these models is providing a powerful and reusable abstraction block, which is able to process raw audio in a given language or in multilingual settings @xcite , producing a richer audio representation for the downstream tasks to train with, compared to surface features such as mfccs or filterbanks. recent work found considerable performance gains and/or state-of-the-art performance by including these blocks in their target tasks, and more importantly, the final models can be trained with a smaller amount of labeled data, increasing the accessibility of current approaches for speech processing @xcite @xcite @xcite ). 1 recent work found considerable performance gains and/or state-of-the-art performance by including these blocks in downstream tasks. most of them focused on asr @xcite @xcite @xcite , but recent speech benchmarks @xcite cover tasks such as st, spoken language understanding, emotion recognition from speech and more. ## tunisian arabic-english track in this section, we present our experiments for translating tunisian arabic to english in the context of the dialect and low-resource tasks from iwslt 2023. section 4.1 describes the data used in our experiments. results on the st task are presented in section 4.3. ## tamasheq-french experiments in this section we present our experiments for the tamasheq-french dataset in the context of the lowresource st track. ## pashto-french experiments in this section, we present our experiments for the first edition of translating pashto speech to french in the context of the low-resource st track for iwslt 2023. ## conclusion this paper presents results obtained on three tasks from the iwslt 2023 dialectal and low-resource st track, namely tunisian to english, tamasheq to french and pashto to french. given an unconstrained condition, our submission relies heavily on the semantic speech representation learning framework samu-xls-r that greatly improves results compared to the other submitted end-toend st models by leveraging multilingual data from other languages. these data can thus come from high resource languages and help to alleviate the low-resource setting difficulty. we indeed observe slightly improved results when using a samu-xls-r model trained on more languages (tamasheq to french : 15.35 bleu when using 60 languages, 16.52 bleu when using 100 languages). we believe results could be further improved by using the unlabelled data available for the tunisian to english and the tamasheq to french tasks, and by investigating other decoders in our encoder-decoder framework.