Voices Unheard: NLP Resources and Models for Y orùbá Regional Dialects (2024)

Abstract: Yoruba—an African language with roughly 47 million speakers—encompasses a continuum with several dialects. Recent efforts to develop NLP technologies for African languages have focused on their standard dialects, resulting in disparities for dialects and varieties for which there are little to no resources or tools. We take steps towards bridging this gap by introducing a new high-quality parallel text and speech corpus; YORULECT across three domains and four regional yoruba dialects. To develop this corpus, we engaged native speakers, traveling to communities where these dialects are spoken, to collect text and speech data. Using our newly created corpus, we conducted extensive experiments on (text) machine translation, automatic speech recognition, and speech-to-text translation. Our results reveal substantial performance disparities between standard yoruba and the other dialects across all tasks. However, we also show that with dialect-adaptive finetuning, we are able to narrow this gap. We believe our dataset and experimental analysis will contribute greatly to developing NLP tools for Yoruba and its dialects, and potentially for other African languages, by improving our understanding of existing challenges and offering a high-quality dataset for further development. We will release YORULECT dataset and models publicly under an open license.

## introduction while great strides have been made in developing nlp resources for low-resource languages, the majority of these efforts have been directed towards the "standard" dialect of these languages, largely neglecting the long tail of non-standard dialects spoken by millions @xcite . dialects of a language exhibit nuanced yet distinguishable differences in lexicon, pronunciation, spelling, and syntax, mirroring regional, societal, and cultural differences @xcite . usually, a "standard" dialect is the dialect with the highest population of speakers, and sometimes the only dialect with a standard orthography @xcite . african languages are linguistically diverse @xcite ), yet severely under-resourced. most of these languages have numerous varieties, (usually regional), some of which are mostly-spoken and lack a standard orthography @xcite . developing language technologies has been incredibly challenging for african languages @xcite @xcite @xcite @xcite , partly due to the scarcity of extensive language resources required for developing systems that are robust to the variations in linguistic features @xcite . to address this problem, in this work we focus on curating dialectal resources for yorùbá, a lowresource language with 47 million native speakers around the world. yorùbá language is native to southwestern nigeria, republic of benin, and republic of togo. yorùbá encompasses a dialect continuum including several distinct regional dialects @xcite . due to yorùbá's low-resource status, the majority of published nlp work have been done on the standard yorùbá dialect @xcite @xcite @xcite @xcite @xcite @xcite @xcite . we introduce the first-ever corpus of high quality, contemporary yorùbá speech and text data parallel across four yorùbá dialects; standard yorùbá, ifè . / i f e /, ìlàje . / i l a dz e / and ìjè . bú/ i dz e b u / in three domains (religious, news, and ted talks). this newly curated benchmark, developed with native speakers, can be used in (text-to-text) machine translation (mt), automatic speech recognition (asr), speech-to-text translation (s2tt), and speech-to-speech translation (stst) tasks. we discuss in detail the data curation process, criteria for data selection, and the steps we took to ensure data quality and integrity ( §3). we first conduct extensive experiments evaluating the zero-shot performance of recent state-of-the-art models for mt, asr, and s2tt ( §4, §5). our results and analysis indicate that current models are not robust enough to handle existing variation in yorùbá dialects. given these poor results, we proceed to adapt (fine-tune) existing models on our training data across all tasks to boost overall performance. with 802 training instances in each dialect, this approach leads to an average increase of 14 and 5 bleu points for both mt and s2tt respectively, as well as a 20-point decrease in word-error-rate for asr. our work aims to motivate the community to build technology for languages alongside their dialects, especially for low-resource dialects of low-resource languages, as this will promote linguistic diversity, and ensure that technological advancements benefit all language communities. the yorùbá language is spoken natively by roughly 47 million people in nigeria 2 and in the neighboring countries of the republic of benin and togo and also côte d'ivoire, sierra leone, cuba, and brazil. in nigeria, yorùbá speakers are mainly concentrated in the southwest region, spanning states like oyo, ogun, osun, ondo, ekiti, and lagos, and 2 https://en.wikipedia.org/wiki/yoruba_language north central states like kogi, and kwara. the extensive yorùbá-speaking population and their dispersion across various regions have led to the emergence of geography-specific linguistic variations @xcite . the number of existing yorùbá dialects is estimated between twelve to twenty-six @xcite @xcite and the differences present in these dialects are evident in pronunciation, grammatical structure, and vocabulary @xcite @xcite @xcite . also categorized as a volta-niger language within the yoruboid subgroup of the niger-congo family, yorùbá is a tonal language with three basic tones: low, middle, and high @xcite , as well as two or three contour tones. 3 previous research @xcite has in-dicated that the phonetic nuances of contour tones are a major distinguishing feature among yorùbá dialects. yorùbá dialectal forms in nigeria can be classified into five regional groupings: northwest yorùbá (nwy), northeastern yorùbá (ney), central yorùbá (cy), southwest yorùbá (swy), and southeast yorùbá (sey). phonological, lexical, and grammatical differences distinguish these groupings, given the diverse levels of mutual intelligibility among the "regional" dialects within each category @xcite @xcite . in this work, our focus lies on ifè . , a dialect in the central yoruba classification, ìjè . bú, and ìlàje . dialects, which belong to the southeast yoruba classification. we display the geographical distribution of yorùbá dialects in west africa in figure 1 . comparative dialectal analysis standard yorùbá, ifè . , ìjè . bú and ìlàje . dialects exhibit both similarities and differences in their orthographic representations, morphology, and semantics. for instance, standard yorùbá dialect has fused velar fricative /g/ and labialised voiced velar /g w / into /w/ @xcite and our curated data revealed a similar pattern for ìjè . bú. in contrast, ifè . uses /g/ in certain occurrences while ìlàje . has heavily retained the /g w / and /g/ in its representations. as a result, at the word level, "àwo . n" (3p pl.) is represented similarly in standard dialect and ìjè . bú but as "igho . n" in ifè . and "àghan" in ìlàje . . besides the contrastive consonant nature, the oral and nasal vowels are also both contrastive in ifè . and ìlàje . dialects respectively. further analsyses of yorùlect reveal that the low nasalised vowel /ã/ mostly follows "gh" in ìlàje . while the back lowermid nasalised vowel /õ/ accompanies "gh" in ifè . dialect. one remarkable semantic variation is that standard yorùbá dialect uses "so . " and "wi pe" as say/talk, however for ìlàje . and ìjè . bú the morpheme mostly used is "fo . " while ifè . uses "ghii", all of which have the same semantics. ## yorùlect corpus we curated parallel text and recorded high quality speech data across ifè . , ìjè . bú, ìlàje . , and standard yorùbá dialects. our data curation process involves three main steps: (i) text curation and dialect localization; (ii) speech recording; and (iii) text and audio alignment. ## zero-shot experiments we start by evaluating the zero-shot performance of current state-of-the-art models on the test portion of yorùlect. based on the results from this initial evaluation, we then adapt the top-performing zero-shot models by finetuning on the training portion of yorùlect and report results in §5.1. mt experiments are conducted on all dialects, while asr and s2tt experiments are conducted on all expect ìjè . bú. ## human evaluation we complement automatic evaluation metrics with a human evaluation study to assess the quality of translations and transcriptions from the best models after fine-tuning for mt and asr. previous research has shown that word error rate (wer) is not nuanced, as it treats all errors in asr text-insertions, deletions, and substitutions-the same, without considering their impact on readability @xcite . 7 for asr, one native speaker per dialect rated the quality of 30 randomly sampled transcriptions from the test set produced by our best asr models after finetuning. after listening to the source speech they assess fluency (how natural and grammatically correct the transcription sounds in their dialect) and adequacy (how accurately the transcription conveys the meaning of the source speech) using a likert scale of (1-5), the higher the better. in adequacy ↑ fluency ↑ standard 3.37 3.03 ìlàje . for mt, we ask human raters to compare the quality of translations from google translate with translations after finetuning nllb, still focusing on fluency and adequacy still using a likert scale (1-5). we provide the exact phrasings of instruction in the §a.4. our results, displayed in ## analysis and discussion does edit distance explain performance gaps? in this analysis we aim to understand how dialectal similarity influences model adaptation during finetuning. ideally, we expect dialects with higher similarity to standard yorùbá to perform better. edit distance @xcite is a simple method commonly used in dialectometry to infer pronunciation differences between language dialects @xcite @xcite . in our work, we use edit distance as a proxy for similarity between standard yorùbá and the other dialects in our corpus, expecting that dialects with a higher degree of similarity (lower edit distance) will perform better. we compute the average edit distance per dialect, @xmath0, t i ), where n is the number of sentences in the test set of the dialect, s is the sentence in standard yorùbá, t is the sentence in the corresponding dialect, and d(s i , t i ) is the edit distance between s i and t i at the character-level. we present the results of this analysis in mt in joint vs. dialect-specific finetuning. dialects often exhibit rather subtle variations in text and speech. in data-constrained scenarios like ours, it is reasonable to expect that jointly finetuning on all dialects would result in better performance compared to fine-tuning on each dialect individually. in our earlier finetuning experiments detailed in §5, we explored joint training. now, we try to compare performance between joint training and individual training on mt and asr tasks. we generally see that on both tasks, joint training is beneficial. in mt, ## related work previous works that have developing technologies and resources for machine translation @xcite @xcite @xcite , automatic speech recognition @xcite @xcite and speech translation @xcite for yorùbá have largely focused on the standard yorùbá dialect. this is because, just like other african languages, standard yorùbá is also very low-resourced, and all efforts have been directed there. several works have shown that models often exhibit performance disparities between standard languages and their dialectal counterparts @xcite @xcite @xcite @xcite @xcite . arabic language has roughly 30 regional dialects. whilst majority of work has being done on modern standard arabic (msa), arabic still has the widest coverage of tasks and datasets across several of its dialects @xcite @xcite . within african languages, some works that aim to build dialect-aware models have conducted their studies on igbo @xcite , luhya @xcite , bemba @xcite and kiswahili @xcite . ## conclusion we introduce yorùlect-the first high quality parallel text and speech corpus for four yorùbá dialects sourced primarily from native speakers, to enable asr, mt and s2tt tasks for widelyspoken varieties of yorùbá. we have provided a detailed documentation of data curation process from standard text creation, to dialect localization and speech recording in communities where these dialects are spoken. extensive experiments reveal that current models are not robust to dialectal variation, and improve significantly after our dialectadaptive finetuning. overall, our data collection methodology, new resources and improved models take a step towards enhancing the quality and equity of nlp technologies for yorùbá dialects and potentially other african languages.