FFSTC 2: Extending the Fongbe to F rench Speech Translation Corpus (2025)

Abstract: This paper introduced FFSTC 2, an expanded version of the existing Fongbe-to-French speech translation corpus, addressing the critical need for resources in African dialects for speech recognition and translation tasks. We extended the dataset by adding 36 hours of transcribed audio, bringing the total to 61 hours, thereby enhancing its utility for both automatic speech recognition (ASR) and speech translation (ST) in Fongbe, a low-resource language. Using this enriched corpus, we developed both cascade and end-to-end speech translation systems. Our models employ AfriHuBERT and HuBERT147, two speech encoders specialized to African languages, and the NLLB and mBART models as decoders. We also investigate the use of the SAMU-XLSR approach to inject sentence-level semantic information to the XSLR-128 model used as an alternative speech encoder. We also introduced a novel diacritic-substitution technique for ASR, which, when combined with NLLB, enables a cascade model to achieve a BLEU score of 37.23 ompared to 39.60 obtained by the best system using original diacritics. Among the end-to-end architectures evaluated, the architectures with data augmentation and NLLB as decoder achieved the highest score respectively, SAMU-NLLB scored the BLEU score of 28.43.

## introduction the creation of high-quality audio datasets for natural language processing (nlp) tasks remains a significant challenge. current efforts to develop speech datasets have predominantly focused on widely spoken languages such as english, french, and spanish, leaving dialectal and minority languages largely under-represented. as a result, the vast majority of the world's 7,000 languages remain underserved, with only a few dozen language directions covered in existing speech translation corpora @xcite . this lack of inclusion presents a critical problem, as it perpetuates language barriers and limits the accessibility of nlp technologies for speakers of less-represented languages. in recent years, there has been increasing attention on low-resource languages, particularly those spoken in regions such as india and africa. despite being spoken by millions of people, many of these languages remain severely under-represented in terms of linguistic resources. for instance, africa is home to over 2,000 languages @xcite , yet, as highlighted by @xcite , only around 40 of these languages have been integrated into modern language technologies. this stark disparity underscores the significant under-representation of african languages in contemporary nlp research and applications. initiatives such as the mozilla common voice project foot_0 have sought to address this gap by providing a platform for collecting speech data for certain african languages. however, the impact of such efforts remains limited. this limitation stems from the platform's design, which relies on collecting data through the reading of written texts. this approach is less effective for many african languages, which are primarily oral and have limited written resources. historically, oral traditions @xcite have been widespread across the continent, which has hindered the development of writing systems and further complicated efforts to create comprehensive linguistic datasets. while resources fongbe are limited, some datasets do exist in the state of the art. for fongbe, we have the alffa (laleye et al.) dataset for speech recognition, which includes 6 hours of audio along with transcriptions, and the ffstc dataset @xcite , which contains 31 hours of fongbe speech paired with french trans-lations. also, corpora like gigast are pseudolabeled, distinguishing them from fully humanannotated datasets such as @xcite contained in alffa, which contains a few hours of data for languages like amharic, hausa, swahili, wolof. in this paper, we present a significant extension of the existing fongbe-to-french speech translation corpus @xcite . this extension not only adds new parallel data from spoken fongbe to written french but also enables the training of an automatic speech recognition (asr) model for fongbe by providing speech recordings with their corresponding transcriptions. this dataset will provide an opportunity for research community to test all new ssl models designed for african languages @xcite , especially since no existing ssl model currently includes fongbe in its training data. the data described here will be used to organize a translation task in iwslt 2025 2 . this paper provides experiments and evaluation for automatic speech recognition (asr) in fongbe and speech translation (st) from fongbe to french. these experiments aim to assess the effectiveness of leveraging pre-trained models for low-resource language processing, with a focus on a tonal african language spoken by 4 million people. ## related work and motivations unlabeled data is far more abundant than labeled data. self-supervised learning (ssl) methods have emerged as a powerful approach to leverage such unlabeled data in machine learning, enabling the creation of pre-trained models. a first notable example in the domain of speech is the xlsr-53 model @xcite , which is pre-trained on 53 languages data. studies @xcite @xcite have extensively explored the use of pre-trained speech encoders and text decoders to enhance system performance for the speech translation task. these techniques, collectively referred to as transfer learning, have demonstrated significant effectiveness in improving performance, particularly in low-resource settings. by transferring knowledge from high-resource languages to low-resource ones, transfer learning provides a robust initialization for both encoder and decoder components, thereby significantly contributing to improved translation accuracy. in 2 international workshop on spoken language translation addition to transfer learning, other approaches have been adopted to enhance performance in lowresource speech translation, such as synthesizing parallel data @xcite . however, our work specifically focuses on the transfer learning paradigm, leveraging its proven capabilities to address the challenges of low-resource language processing. despite the challenges associated with creating speech corpora for african languages, there has been a notable shift towards the inclusion of these languages in pre-trained models. this trend is evident in the progressive integration of african languages into state-of-the-art models, such as hubert147 (boito and al, 2024), which supports 16 african languages, and its variant afrihu-bert @xcite , which extends coverage to 39 languages. various strategies have been employed to use pre-trained models effectively. for instance, some studies (mbuya and anastasopoulos; zanon boito and ortega) utilize pre-trained encoders like xlsr-53 as feature extractors or encoders paired with a transformer @xcite based decoder. our experiments align with this latter, employing hubert variants as the encoder and using a pre-trained large language model (llms) transformer based as decoder. although the literature does not provide definitive guidance on selecting the most suitable pretrained speech encoder, @xcite observed that encoders trained on the same language as the source language tend to extract more relevant audio features, thereby improving overall performance. given that fongbe is an african language and no pre-trained speech model includes it at the time of writing, we hypothesize that using a pre-trained encoder trained on other close linguistically african languages would yield promising results. to test this hypothesis, we conduct training experiments using pre-trained models hubert147 and afri-hubert as encoders, combined with pre-trained multilingual decoders such as mbart @xcite and nllb @xcite . and television programs. however, fongbe tonal nature presents unique challenges, particularly in written and translated texts. in linguistics, tone refers to the use of pitch variations to distinguish meaning in spoken language @xcite . lexical tones, in particular, help differentiate words that are otherwise phonologically identical @xcite . these pitch variations, or tonal patterns, are produced by changes in the fundamental frequency of a syllable. for the written form of fongbe, mainly based on the latin alphabet with additional symbols, these tones are typically represented using diacritics, which are essential for accurately conveying tonal distinctions in written form. fongbe primarily features two main tonemes as noted by @xcite , from which all other tones are derived. in fongbe, each syllable carries a tone, and the absence of tone marks can often lead to confusion. regarding tones, there are four tones in fongbe. the low tone ( `), the high tone ( ´), the low-high tone ( ˇ) and lastly, the mid tone ( -) marked by a small horizontal line. the absence of tone is considered as the mid tone. fongbe utilizes an alphabet comprising 23 consonants and 12 vowels as shown in ## experiments and results in this section, we present the experimental framework for (1) asr system, (2) cascade st system and end-to-end st system. ## conclusion this work represents a significant advance for fongbe speech processing, for both transcription and translation to french. by extending an existing dataset to 61 hours of high-quality audio and aligned text, we offer the research community a unique and richer resource to build and evaluate speech technologies for fongbe, a tonal and underrepresented language. our detailed experiments in both cascade and end-to-end speech translation reveal several important insights that can stimulate broader research in low-resource language technologies. our best cascade system achieved a bleu score of 39.60, underscoring the power of carefully handling tonal information. in contrast, our most effective end-to-end model achieved a bleu of 28.43, especially when leveraging data augmentation and semantic alignment. the expanded fongbe corpus and our findings open several possibilities for further research. first, improvements to diacritic substitution-potentially using more granular markers that capture subtle tonal shifts could reduce asr errors while preserving key phonological cues for translation. second, personalized or speaker-adaptive speech translation models, possibly trained to handle specific dialectal variants, may substantially enhance intelligibility and translation fidelity. finally, future self-supervised or multilingual pre-training efforts will benefit from explicitly including fongbe data, leading to more robust encoder-decoder architectures for low-resource african languages. overall, this work not only delivers the largest corpus of fongbe audio currently available for speech recognition and translation, but also highlights data-collection strategies, modelling setups, and diacritic handling approaches that can be generalized to other tonal, under-represented languages.