Speech-to-Speech Translation Pipelines for Conversations in Low-Resource Languages (2025)

Abstract: The popularity of automatic speech-to-speech translation for human conversations is growing, but the quality varies significantly depending on the language pair. In a context of community interpreting for low-resource languages, namely Turkish and Pashto to/from French, we collected fine-tuning and testing data, and compared systems using several automatic metrics (BLEU, COMET, and BLASER) and human assessments. The pipelines consist of automatic speech recognition, machine translation, and speech synthesis, with local models and cloud-based commercial ones. Some components have been fine-tuned on our data. We evaluated over 60 pipelines and determined the best one for each direction. We also found that the ranks of components are generally independent of the rest of the pipeline.

## introduction one of the most challenging applications of spoken language translation is real-time interpreting of human conversations. we consider the application to community interpreting, for ethnic minorities who need assistance to access services across a language barrier, e.g., for healthcare, asylum rights, or education. the case study presented here involves bhaasha, a company that provides services for community interpreting, and the data science group of heig-vd, an academic partner. due to a growing demand, the company aims to clarify whether a system for automated interpreting meets certain quality thresholds and can be offered when human interpreters are not available. while several online offers exist, these systems do not include the desired language pairs, or their quality is clearly insufficient, and privacy is not guaranteed. we present the methods and the results of a joint project aimed at determining the best speech-tospeech translation pipeline made from off-the-shelf components, cloud-based services, or fine-tuned models, for two language pairs that are in high demand, but are insufficiently supported by existing systems: french-turkish and french-pashto. the paper is organized as follows. in section 3, we present methods for collecting and annotating data representative of the intended context of use. in section 4, we outline the design of translation pipelines, whose components can be smoothly interchanged. in section 5, we evaluate all combinations of four asr, three mt, and two tts components, either local or cloud-based, also including two fine-tuned components and a speech-to-text translation one. we present evaluation scores from automatic metrics and determine the best combination of components per direction. we also show that the ranking of components is generally independent of the other modules in a pipeline. finally, we present human scores over a subset of the data, showing that accuracy, fluency and intonation of the best pipelines are considered as 'good' or 'very good'. ## state of the art methods for speech-to-text translation (e.g. for subtitling) have been the subject of many recent publications, unlike methods for speech-to-speech translation, as the speech synthesis part is difficult to train. moreover, spoken translation has been studied more often for monologues than for conversations. the three necessary components are automatic speech recognition (asr, or speech-to-text, stt), machine translation (mt), and speech synthesis (or text-to-speech, tts). research interests, however, have shifted from loosely coupled cascades of asr and mt, to tighter coupling, and finally to recent end-to-end models @xcite . for instance, an approach to multilingual speech-to-text translation through efficient transfer learning from a pretrained speech encoder (wav2vec) and text decoder (bert), is proposed by @xcite . @xcite propose a listen, understand and translate (lut) approach to train end-to-end speech-to-text translation. @xcite compare the two paradigms -cascaded vs. end-to-end -and claim that the gap between them is almost closed. however, for low-resource languages, end-to-end systems are difficult to train due to the lack of data, while cascaded systems can use components trained with simpler tasks. alternatively, massively multilingual systems such as whisper @xcite for asr + mt claim that lowresource languages are improved thanks to higherresource similar languages. for instance, cascaded approaches can take advantage of optimized low-resource mt components (atrio and popescu-belis, 2022). the iwslt 2022, 2023 and 2024 evaluation campaigns @xcite @xcite featured various shared tasks, including speech-to-text and speechto-speech translation for low-resource languages. a typical low-resource system presented at iwslt 2023 is the marathi to hindi submission by @xcite , including an end-to-end and a cascaded system. various techniques for improving low-resource speech-to-text translation, in particular with initialization from a multilingual asr system, have been proposed @xcite @xcite . large corpora exist for well-resourced language pairs, but not for the low-resource ones that we target. while datasets of recorded speech can be more easily found, datasets with transcriptions and translations are scarce or inexistent. the must-c @xcite and multilingual tedx corpora include speech and translation in english and 8 other european languages, but not turkish or pastho. covost-2 @xcite covers speech translation from several languages to/from english and includes turkish. this resource was used to test the whisper asr + mt used here. recent developments of generative ai and large language models have enabled significant progress in speech translation and synthesis, but low-resource languages are still insufficiently supported. for instance, several companies advertise multilingual speech translation systems on the web, as apps for smartphones, or as cloud-based services, mostly for for a few well-resourced languages. for instance, one of the major players, google, offers the three components individually via apis, but also bundles them into a pipeline that often appears in informal tests as one of the best translation apps for several language pairs. other commercial offers include deepl, microsoft's bing translator, itranslate, sayhi, translate now, yandex, or talking translator. many of the related apps have received reviews from their users, which provide a form of evaluation, although ratings for specific language pairs are rarely found. in our tests, we observed that these solutions are not ready for the low-resource languages studied here, nor for use in the setting of community interpreting. ## data gathering and formatting to the best of our knowledge, there are no datasets with parallel conversational speech (i.e. interpreted in both directions) for turkish-french and pashto-french (tr-fr, ps-fr). therefore, we collected new data which suits our project's needs. the central idea of our parallel dataset is to include complete dialogues in situations encountered by bhaasha's community interpreters. for each utterance, we have a reference transcript and an audio recording, in each of the three languages of the project: an excerpt is shown in ## conclusion we have produced data and assembled numerous speech-to-speech translation pipelines, for interpreting turkish ↔ french and pashto ↔ french conversations. specifically, we have produced three hours of data in settings compatible with community interpreting, and used half of it for fine-tuning two turkish ↔ french asr and mt systems, and the other half for evaluation. we scored over 60 pipelines of asr, mt and tts systems, either based on open-weight models run locally, or on commercial cloud-based services. we identified the best-performing pipeline in each direction, and found that the ranking of components was consistent, regardless of the other components of pipelines. we used four automatic evaluation metrics (wer, bleu, comet and blaser), along with pilot human evaluations. the implementation of an online system with a push-to-talk interface, along with an offline version allowing batch processing, now paves the way towards usability testing of automatic interpretation solutions, which will also need to take into consideration factors such as privacy, cost, and deployment strategy.