QUESPA Submission for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks (2023)

Abstract: This article describes the QUESPA team speech translation (ST) submissions for the Quechua to Spanish (QUE–SPA) track featured in the Evaluation Campaign of IWSLT 2023: low-resource and dialect speech translation. Two main submission types were supported in the campaign: constrained and unconstrained. We submitted six total systems of which our best (primary) constrained system consisted of an ST model based on the Fairseq S2T framework where the audio representations were created using log mel-scale filter banks as features and the translations were performed using a transformer. The best (primary) unconstrained system used a pipeline approach which combined automatic speech recognition (ASR) with machine translation (MT). The ASR transcriptions for the best unconstrained system were computed using a pre-trained XLS-R-based model along with a fine-tuned language model. Transcriptions were translated using a MT system based on a fine-tuned, pre-trained language model (PLM). The four other submissions are presented in this article (2 constrained and 2 unconstrained) for comparison because they consist of various architectures. Our results show that direct ST (ASR and MT combined together) can be more effective than a PLM in a low-resource (constrained) setting for Quechua to Spanish. On the other hand, we show that fine-tuning of any type on both the ASR and MT system is worthwhile, resulting in nearly 16 BLEU for the unconstrained task.

## introduction low-resource machine translation (lrmt) can be considered a difficult task due to the low amount of parallel data on hand. @xcite by adding the task of automatic speech recognition (asr), complexity can be even more difficult. findings from the previous year's iwslt 2022 @xcite have shown that for lowresource language pairs like tamasheq-french, it is difficult to achieve more than 5 bleu @xcite score points for the combined task of speech translation (st), even in a unconstrained setting. this year, the iwslt 2023 @xcite evaluation campaign for low-resource and dialect speech translation has included tamasheq-french along with several other language pairs. one of the newly introduced language pairs is quechua-spanish deemed que-spa by the organizers. quechua is an indigenous language spoken in the andes mountainous region in south america. it is spoken by millions of native speakers mostly from peru, ecuador and bolivia. in those regions, the high-resource language is spanish. quechua displays many unique morphological properties of which high inflection and poly-synthetic are the two most commonly known. it is worthwhile to note that previous work @xcite has been somewhat successful in identifying the inflectional properties of quechua such as agglutination where another high-resource language, namely finnish, can aid for translation purposes achieving nearly 20 bleu on religious-based (text-only) tasks. since this is the first year that que-spa has been included in the iwslt 2023 campaign, we feel that it is important to set a proper baseline. the aim of our submission was to increase the viability of the use of a quechua-spanish st system and we thus attempted several approaches that included the use of pipelines (cascade) approaches along with joint asr + mt. we report on the six system submissions as a final takeaway for this article; however, we also compare other approaches that performed worse (1 bleu or less). our team is called quespa and consists of a consortium that spans across three universities: northeastern university (usa), universitat de pompeu fabra (spain), and carnegie mellon university (usa). our objective is to help to solve the lrmt prob-lem for quechua with the intention of at some point releasing an st system to the quechua community where we have strategic partners located in areas of peru where quechua is mostly spoken. the authors of this article have participated in several other events and written literature that includes native quechua annotations for natural language processing (nlp) systems including mt and more. this article reports the quespa consortium submissions for the iwslt 2023 dialect and lowresource tasks. we focus only on the low-resource task despite the mention of two dialects quechua i and ii. our focus is on creating the optimum models we can for the constrained task and leveraging pre-trained models for the unconstrained task further described in section 3. the rest of this article is organized as follows. section 2 presents the related work. the experiments for que-spa low-resource track are presented in section 3. section 4 provides results from the six submitted systems and concludes this work. ## related work in this section, we first cover work directly related to the asr and mt tasks of que-spa done in the past. then, we introduce related work on st models in general to provide an idea of what work is current in the field. quechua to spanish mt approaches have become more abundant in the past few years. when it comes to asr->mt, or st approaches, there are few attempts officially recorded. in this section, we list previous work in chronological order to better explain the mt approaches attempted. first, rios (2015) provided an advanced linguistic quechua toolkit that used finite state transducers (fsts) to translate from spanish to quechua. her work laid the foundation for future work and helped to promote the digitization of the quechua language. after that, @xcite and @xcite introduced several new findings that included the asr corpus used in the iwslt 2023 task for both unconstrained and constrained purposes. not long after, @xcite introduced the first known attempt of a neural mt system that included several annotators along with the state-of-the-art techniques in subsegmentation such as byte-pair encoding (bpe) @xcite . their work was then extended by others @xcite more recently to achieve 23 bleu on religious-based text, the highest performing que-spa for its time. none of the approaches before chen and fazio (2021) work included the use of pre-trained language models (plms) for low-resource languages. however, the introduction of zero-shot models occurred at the low-resource machine translation workshop in 2020 @xcite and not long after in 2021 at the americas nlp workshop @xcite . the americas nlp 2021 workshop included the use of que-spa, albeit for mt only achieving scores of 5.39 bleu through the use of a multi-lingual model trained on 10 other indigenous languages. their work did not include zero-shot task approaches as introduced by @xcite where fine-tuning was performed on a pre-trained xlm-r @xcite model that achieved impressive results (40-55 bleu). more recent work @xcite did not surpass those results for mt of que-spa. to our knowledge only one competition/shared task has attempted to process que-spa for speech translation purposes -americas nlp 2022 foot_0 . however, the findings for the task have not beeen published as of the writing of this article. their competition used corpora similar to iwslt 2023 but lacks mt data as a separate (constrained) resource. they also do not introduce the concept of constrained or unconstrained tasks as was done at iwslt 2023. apart from those tasks that directly use the que-spa language pair, several mainstream techniques are currently being used as alternatives to supervised (from scratch) training. for example, one of the most common approaches for both st and mt approaches tend to use a transformer in some capacity along with a plm. one such model that uses a multi-lingual low-resource corpus called flores @xcite ) is facebook's nllb (no language left behind) approach @xcite . their approach uses self-supervised learning (ssl) from previous innovation @xcite for multi-lingual approaches that combines asr with mt in a st task alone and is made available through fairseq @xcite . in our work, our primary systems use fairseq and facebook's plms with sentence embeddings based on previous work @xcite and the m2m (multi-to-multi) model @xcite consisting of 1.2 billion parameters. this enables zero-shot cross-lingual transfer for many low-resource languages, including quechua. we provide reference to previous work that includes either a direct or end-to-end st models @xcite . more traditional approaches typically use a cascade approach which first transcribes using an asr model and then translates using a mt model. while recent work @xcite @xcite has shown that the direct st approaches are worthy, traditional approaches work well for low-resource situations too. in our system submissions, all of our systems with exception of the primary constrained used the cascade approach. in this section we present our experiments for the que-spa dataset provided in the low-resource st track at iwslt 2023. this is the first time that this dataset has been officially introduced in its current state which contains 1 hour and 40 minutes of constrained speech audio along with its corresponding translations and nearly 60 hours of asr data (with transcriptions) from the siminichik (cardenas et al., 2018) corpus. americasnlp 2022's task used a smaller part of the dataset but the data was not presented or compiled with the same offering and, as of this writing, have not published their results. this dataset aggregates the que-spa mt corpus from previous neural mt work @xcite . the audio and corresponding transcriptions along with their translations are mostly made of of radio broadcasting, similar to the work from boito et al. (2022) which contains 17 hours of speech in the tamasheq language. we present the six submissions for both the constrained and unconstrained as follows: we present the experimental settings and results for all systems starting off with constrained systems in section 3.1 and continuing with the unconstrained systems in section 3.2. we then describe the other less successful approaches in section 3.3. finally, we offer results and discussion in section 4. ## results and discussion for the unconstrained setting, our findings have shown that for both the asr and mt models, the use of a plm with fine-tuning is necessary. we were unable to create a system from scratch that would perform as well as those presented in previous tasks. the combination of a language model and the fleurs plm for asr along with the flores 101 plm for mt constitutes our best performing system overall as shown in figure 1. the language model slightly helped for the primary system by a gain of nearly 0.10 points in bleu. the other unconstrained system based on w2vletter (pratap et al., 2019) performed much better than the constrained version making it worthwhile to explore for future iterations since it doesn't require other languages. ## conclusion concluding, we have experimented with several options for both the constrained and unconstrained settings. this constitutes the first time that experiments have been put together along with the other team submissions for the quechua to spanish task. we believe that the performance achieved here can serve as baselines for more sophisticated approaches. additionally, it came to our attention that data splits provided by the organizers can be adjusted to better fit the data. there are multiple speakers in several of the audio files, we did not take advantage of this and hope to address it in the future. also for the future, we believe that more work could be done using direct st systems with fine-tuning. we did not follow that path in this work but feel it would be advantageous.