QUESPA Submission for the IWSLT 2024 Dialectal and Low-resource Speech Translation Task (2024)

Abstract: This article describes the QUESPA team speech translation (ST) submissions for the Quechua to Spanish (QUE–SPA) track featured in the Evaluation Campaign of IWSLT 2024: dialectal and low-resource speech translation. Two main submission types were supported in the campaign: constrained and unconstrained. This is our second year submitting our ST systems to the IWSLT shared task and we feel that we have achieved novel performance, surpassing last year’s submissions. Again, we were able to submit six total systems of which our best (primary) constrained system consisted of an ST model based on the Fairseq S2T framework where the audio representations were created using log mel-scale filter banks as features and the translations were performed using a transformer. The system was similar to last year’s submission with slight configuration changes, allowing us to achieve slightly higher performance (2 BLEU). Contrastingly, we were able to achieve much better performance than last year on the unconstrained task using a larger pre-trained language (PLM) model for ST (without cascading) and the inclusion of parallel QUE–SPA data found on the internet. The fine-tuning of Microsoft’s SpeechT5 model in a ST setting along with the addition of new data and a data augmentation technique allowed us to achieve 19.7 BLEU. Additionally, we present the other four submissions (2 constrained and 2 unconstrained) which are part of additional efforts of hyper-parameter and configuration tuning on existent models and the inclusion of Whisper for speech recognition

## introduction speech translation (st) has historically been a difficult task due to the lack of parallel data required to train neural end-to-end systems. as such, the traditional approach to this task has been to use a cascade of distinct modules, separating st into the subtasks of automatic speech recognition (asr) and machine translation (mt). while this allows st systems to benefit from the advances in pretrained language models (plms) for asr and mt, creating usable models for low-resource languages has remained a challenge due to the lack of support for these languages in plms. findings from previous iterations of iwslt @xcite clearly show this phenomena: large-scale ensembling and multilingual supervised pre-training are required to even reach 15 bleu @xcite in low-resource pairs such as quechua-spanish. this year, the iwslt 2024 @xcite evaluation campaign for low-resource and dialect speech translation has included several language pairs for which many teams have submitted to the unconstrained task. some language pairs such as bemba-english have recorded bleu scores as low as 0.5. we feel that as second-time entries we are able to rely on previously built st systems to leverage our work on the quechua to spanish (que-spa) language pair. quechua is an indigenous language spoken by more than 8 million people in south america. it is mainly spoken in peru, ecuador, and bolivia where the official high-resource language is spanish. it is a highly inflective language based on its suffixes which agglutinate and found to be similar to other languages like finnish. it is worthwhile to note that previous work @xcite has been somewhat successful in identifying the inflectional properties of quechua such as agglutination where another high-resource language, namely finnish, can aid for translation purposes achieving nearly 20 bleu on religious-based (text-only) tasks. the average number of morphemes per word (synthesis) is about two times larger than english. english typically has around 1.5 morphemes per word and quechua has about 3 morphemes per word. there are two main region divisions of quechua known as quechua i and quechua ii. this data set consists of two main types of quechua spoken in ayacucho, peru (quechua chanka iso:quy) and cusco, peru (quechua collao iso:quz) which are both part of quechua ii and, thus, considered a "southern" languages. we label the data set with que -the iso norm for quechua ii mixtures. the quespa team this year consists of four organizers from three different institutions: northeastern university, carnegie melon university, and pompeu fabra university. a new organizer has been introduced this year who has expertise in african languages. all of the iwslt 2023 organizers have continued to work on the project; all of the previous organizers have had experience with the que-spa language pair in the past. in this article, we report the quespa consortium submission for the iwslt 2024 and once again focus on the low-resource task at hand by combining all the two dialects quechua i and ii into one. the rest of this article is organized as follows. section 2 presents the related work. the experiments for que-spa low-resource track are presented in section 3. section 4 provides results from the six submitted systems and concludes this work. ## related work in this section, we first cover the different approaches used in previous speech processing shared tasks for quechua (section 2.1). we then discuss prior work that used a similar strategy to our primary submission to the unconstrained track (section 2.2). ## quechua-spanish in this section we present our experiments for the que-spa dataset provided in the low-resource st track at iwslt 2024, identical to the dataset from iwslt 2023. as a reminder, the audio consists of contains 1 hour and 40 minutes of constrained speech along with its corresponding translations and nearly 48 hours of asr data (with transcriptions) from the siminichik @xcite corpus. as an additional constrained setting, the dataset offers the que-spa mt corpus from previous neural mt work @xcite . the audio and corresponding transcriptions along with their translations are mostly made of radio broadcasting from the mountainous region in the andes, peru. this dataset has been used in other tasks but not in its entirety @xcite . we present the six submissions for both the constrained and unconstrained as follows: ## results and discussion results are presented in for the unconstrained setting, our findings have shown that in the past year several novel plms have been created that surpass previous models. it is clear that speech translation as a task is becoming more solvable with pre-trained techniques that perform transfer learning. the combination of the microsoft speech t5 model with data augmentation as shown in figure 2 is a new approach that has not been applied to the que-spa language pair in the past and can be considered the best performing system as of this date to our knowledge. previous systems based on w2vletter @xcite performed well but did not surpass the microsoft speech t5 model in our experiments. ## conclusion and future work our submission to the iwslt 2024 @xcite evaluation campaign for low-resource and dialect speech translation has included novelties based on the most state-of-the-art techniques for asr and st. more specifically, we have been successful by changing the sizes of the models in the constrained setting and changing the type of models in the unconstrained setting. additionally, we have shown that different data augmentation techniques can be used for increased performance on both tasks. we save for future work the experimentation of data augmentation techniques which seem to be the most advantageous novelty in this year's submission. in our opinion, data augmentation can be used for benefits in both the unconstrained and constrained tasks. our plan for future iwslt tasks is to experiment with and without features like specaugment, roll addition, and more.