GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks (2023)

Abstract: This paper describes the GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks. We submitted systems for five low-resource tasks and the dialectal task. In this work, we explored self-supervised pre-trained speech models and finetuned them on speech translation downstream tasks. We use the Wav2vec 2.0, XLSR-53, and Hubert as self-supervised models. Unlike Hubert, Wav2vec 2.0 and XLSR-53 achieve the best results when we remove the top three layers. Our results show that Wav2vec 2.0 and Hubert perform similarly with their relative best configuration. In addition, we found that Wav2vec 2.0 pre-trained on audio data of the same language as the source language of a speech translation model achieves better results. For the low-resource setting, the best results are achieved using either the Wav2vec 2.0 or Hubert models, while XLSR-53 achieves the best results for the dialectal transfer task. We find that XLSR-53 does not perform well for low-resource tasks. Using Wav2vec 2.0, we report close to 2 BLEU point improvements on the test set for the Tamasheq-French compared to the baseline system at the IWSLT 2022.

## introduction recently, speech-to-text translation (s2t) has received a lot of focus in the community where neural, end-to-end approaches outperform traditional statistical approaches @xcite . recent neural approaches to s2t have shown superior performance on this task @xcite . despite the success of neural approaches to s2t, data scarcity is one of the significant challenges, given that neural networks require hundreds to thousands of hours of labeled data to train a good speech translation model @xcite . this makes developing such s2t models challenging, especially for low-resource languages. the iwslt 2023 low-resource and dialectal shared tasks @xcite give the possi-bilities for researchers to find innovative ways to develop speech translation systems for languages with limited data. unlike previous years, this year noticed an addition of more low-resource languages language pairs (up to 6) in addition to a dialect language pair. this paper describes the gmu submissions to the low-resource and dialectal tasks. our systems use self-supervised pre-trained speech models to improve speech translation models' performance in general, particularly for low-resource languages. self-supervised pre-training is possible because unlabeled data (i.e., audio or text) can be obtained easier compared to labeled data. previous research has addressed using self-supervised speech models for speech translation @xcite @xcite . however, these prior work did not consider exploring the impact of different layers of these self-supervised models to maximize the performance of s2t models. in this paper, we consider three self-supervised speech models: wav2vec 2.0 @xcite , xlsr @xcite and hubert @xcite . following the discussion by @xcite , we experimented to study the impact of removing the top n layers of these models for the speech translation task. by removing the last three layers of the wav2vec 2.0 model, we achieve more than 2 bleu improvement (8.03) on the blind test set for the tamasheq-french pair compared to the best system submitted to the iwslt 2022 lowresource shared task @xcite . similarly, using a pretrained xlsr-53, we achieved a bleu score of 16.3 on the tunisian arabic-to-english language pair without using the transcripts. ## task descriptions we are concerned with developing speech translation models in low-resource and dialectal tracks. each track poses distinct challenges. the low-language code train set hours shared task irish to english @xcite ga-eng 11 low-resource marathi to hindi @xcite resource setting has limited training data, while the dialectal one lacks standard orthography and formal grammar. both shared tasks allowed the submission of models trained under constrained and unconstrained conditions. in the constrained condition, models are only trained on data provided by the organizers. in contrast, models in the unconstrained condition can be trained on any available resources, including pre-trained models. ## proposed methods our methods consist of three different architectures. the first is an end-to-end based transformer-based architecture (e2e) trained on only provided data. the second architecture, which we name e2e-asr, is the same as the first, except that we initialize the encoder with an asr encoder. the third architecture uses self-supervised speech models as an encoder and a transformer-based decoder. we used three different self-supervised models, wav2vec 2.0, xlsr-53, and hubert, and refer to these architectures as w2v-e2e, xlsr-e2e, and hubert-e2e respectively. we used the fairseq st @xcite ) framework for all our experiments and modified this framework to accommodate our new custom model architectures. ## results and analyses for both tamasheq-french and quechua-spanish, the best bleu is achieved after removing the top 3 layers. w2v-e2e 16.7 14.4 14.1 12.9 ## data quality issues the low-resource shared tasks of the iwslt 2023 consists of six tasks, each task corresponding to one language pair. as we worked on these shared tasks, we noticed issues with the data of two tasks: maltese to english and irish to english. the maltese to english data had a number of issues that made it hard to work with. for instance, the metadata of about 1001 out of 1698 samples mentioned zero or less than zero duration for audio samples (@xmath0. therefore, we were not able to align most audio data with their utterances. the irish to english data had an issue with the development set. initially, the samples in the development were also present in the training set. however, the organizer later fixed this issue by updating the development set data. however, no matter how we trained our models, we never achieved more than 1 bleu score on the updated development set. after troubleshooting our model on the training data, we were confident that we should have gotten a bleu score that was well above 1. we proceeded with submitting our system for this task. however, we are very suspicious of the high bleu score reported on the blind test, as shown in ## conclusion in this paper, we presented the gmu systems for the iwslt 2023 dialect and low-resource speech translation tasks. our approach mainly focused on using self-supervised pre-trained speech models to improve the performance of speech translation on downstream tasks. the self-supervised pre-trained speech models used in this paper are the wav2vec 2.0, xlsr-53, and hubert. we showed that the wav2vec 2.0 and the hubert model have comparable results in low resource and dialectal transfer tasks. however, the wav2vec 2.0 performs well when we remove the top three layers, while the hubert model has no such requirements. our experiments showed that the xlsr-53 model performs poorly in the low-resource setting compared to the wav2vec 2.0 and hubert models. however, in the dialectal task, the xlsr-53 model outperforms the wav2vec 2.0 and hubert models. in the future, we plan to conduct an in-depth analysis to understand the advantages and limitations of these self-supervised pre-trained speech models while fine-tuning them on downstream speech translation tasks.