ON - TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks (2022)

Abstract: This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2022: low-resource and dialect speech translation. For the Tunisian Arabic-English dataset (low-resource and dialect tracks), we build an end-to-end model as our joint primary submission, and compare it against cascaded models that leverage a large fine-tuned wav2vec 2.0 model for ASR. Our results show that in our settings pipeline approaches are still very competitive, and that with the use of transfer learning, they can outperform end-to-end models for speech translation (ST). For the Tamasheq-French dataset (low-resource track) our primary submission leverages intermediate representations from a wav2vec 2.0 model trained on 234 hours of Tamasheq audio, while our contrastive model uses a French phonetic transcription of the Tamasheq audio as input in a Conformer speech translation architecture jointly trained on automatic speech recognition, ST and machine translation losses. Our results highlight that self-supervised models trained on smaller sets of target data are more effective to low-resource end-to-end ST fine-tuning, compared to large off-the-shelf models. Results also illustrate that even approximate phonetic transcriptions can improve ST scores.

## introduction the vast majority of speech pipelines are developed for and in high-resource languages, a small percentage of languages for which there is a large amount of annotated data freely available @xcite . however, the assessment of systems' performance only on high-resource settings can be problematic because it fails to reflect the real-world performance these approaches will have in diverse and smaller datasets. in this context, the iwslt 2022 @xcite proposes two interesting shared tasks: low-resource and dialect speech translation (st). the former aims to assess the exploitability of current translation systems in data scarcity settings. the latter focuses on the assessment of the systems capabilities in noisy settings: different dialects are mixed in a single dataset of spontaneous speech. for the low-resource task, this year's language pairs are: tamasheq-french and tunisian arabic-english. the latter is also used, in constrained conditions, for the dialect task. this paper reports the on-trac consortium submissions for the mentioned tasks. the on-trac consortium is composed of researchers from three french academic laboratories, lia (avignon university), lium (le mans university) and lig (university grenoble alpes), together with two industrial partners: airbus france and ely-data. our systems for the dialect task focus on the comparison between cascaded and end-to-end approaches for st. for the low-resource task, we focus on the leveraging of models based on selfsupervised learning (ssl), and on the training of st models with joint automatic speech recognition (asr), machine translation (mt) and st losses. this paper is organized as follows. section 2 presents the related work. the experiments with the tunisian arabic-english dataset for low-resource and dialect st tasks are presented in section 3. results for the tamasheq-french dataset for the low-resource track are presented in section 4. section 5 concludes this work. ## related work before the introduction of direct or end-to-end st models @xcite , the st task was approached as a cascaded problem: the speech is transcribed using an asr model, and the transcriptions are used to train a classic mt model. the limitations of this approach include the need for extensive transcriptions of the speech signal, and the error propagation between asr and mt modules. in comparison to that, end-to-end st models propose a simpler encoder-decoder architecture, removing the need for intermediate representations of the speech signal. although at first, cascaded models were superior in performance compared to end-to-end models, results from recent iwslt campaigns illustrate how end-to-end models have been closing this gap @xcite @xcite . moreover, the joint optimization of asr, mt and st losses in end-to-end st models was shown to increase overall performance @xcite . ssl models for speech processing are now a popular foundation blocks in speech pipelines @xcite @xcite . these models are large trainable networks with millions, or even billions @xcite , of parameters that are trained on unlabeled audio data only. the goal of training these models is providing a powerful and reusable abstraction block, which is able to process raw audio in a given language or in multilingual settings @xcite , producing a richer audio representation for the downstream tasks to train with, compared to surface features such as mfccs or filterbanks. recent work found considerable performance gains and/or state-of-the-art performance by including these blocks in their target tasks, and more importantly, the final models can be trained with a smaller amount of labeled data, increasing the accessibility of current approaches for speech processing @xcite @xcite @xcite ). foot_0 ## tunisian arabic-english experiments in this section we present our experiments for translating tunisian arabic to english in the context of the dialect and low-resource tasks from iwslt 2022. section 3.1 describes the data used in our experiments. we investigate two types of st architectures: end-to-end architectures (section 3.3), and pipeline models (section 3.2). for the latter, we include the obtained asr results. for both, results on the st tasks are presented in section 3.4. ## tamasheq-french experiments in this section we present our experiments for the tamasheq-french dataset in the context of the lowresource st track. this dataset, recently introduced in boito et al. (2022), contains 17 h of speech in the tamasheq language, which corresponds to 5,829 utterances translated to french. additional audio data was also made available through the niger-mali audio collection: 224 h in tamasheq and 417 h in geographically close languages (french from niger, fulfulde, hausa, and zarma). 5 for all this data, the speech style is radio broadcasting, and the dataset presents no transcription. our experiments are separated in two different investigation branches: we start by presenting the models proposed for the first branch: the ssl models pre-trained and/or fine-tuned for tamasheq in section 4.1, the pipeline experiments that use wav2vec 2.0 models as feature extractors in section 4.2, and our primary system, an end-to-end architecture that directly fine-tunes a wav2vec 2.0 model, in section 4.3. section 4.4 focuses on the second branch of experiments, presenting our contrastive model that is based on the joint optimization of asr, mt and st losses. this is made possible by the use of a french asr system for generating an approximated phonetic transcription of the tamasheq audio. in section 4.5, we present and discuss our results, and lastly, section 4.6 describes some less-successful experiments. ## conclusion in this paper we presented our results for two iwslt 2022 tasks: dialect and low-resource st. focusing on the tunisian arabic-english dataset (dialect and low-resource tasks), we trained an end-to-end st model as primary submission for both tasks, and contrastive cascaded models that used external data in msa for the low-resource track. our cascaded models turned out to outperform slightly our end-to-end model, which we believe might be due to the additional 820 h of data in msa that was used to pre-train our end-to-end asr model. finally, we observe a considerable variability in our asr results, hinting that the quality of this dataset might be mixed. our experiments with the tamasheq-french dataset (low-resource task) included the training and application of wav2vec 2.0 models for st as either feature extractors or speech encoders. we find the latter to be more beneficial: by fine-tuning half of a wav2vec 2.0 base model trained on the tamasheq language on the st task, we achieve our best results. between our findings regarding the use of ssl models for low-resource st, we highlight two interesting points: first, we find that fine-tuning wav2vec 2.0 models for the asr task turns out to be effective even when the fine-tuning and target languages are not the same. second, we disappointingly observe that large models perform poorly in this low-resource setting, even after fine-tuning in the target language. these last results hint that it might be more beneficial to train wav2vec 2.0 in smaller sets of unlabeled target data (or in related languages in the same speech settings) than fine-tuning massive off-the-shelf ssl models. concluding, we also investigated the generation of approximate transcriptions on tamasheq by using a french asr model. using these transcriptions to jointly constrain an end-to-end st model on asr, mt and st losses, we achieved our second best reported results. this illustrates that even automatically generated approximate transcriptions can reduce the challenge of performing st in low-resource settings.