Kreyòl- MT : Building MT for L atin A merican, C aribbean and Colonial A frican Creole Languages (2024)

Abstract: A majority of language technologies are tailored for a small number of high-resource languages, while relatively many low-resource languages are neglected. One such group, Creole languages, have long been marginalized in academic study, though their speakers could benefit from machine translation (MT). These languages are predominantly used in much of Latin America, Africa and the Caribbean. We present the largest cumulative dataset to date for Creole language MT, including 14.5M unique Creole sentences with parallel translations—11.6M of which we release publicly, and the largest bitexts gathered to date for 41 languages—the first ever for 21. In addition, we provide MT models supporting all 41 Creole languages in 172 translation directions. Given our diverse dataset, we produce a model for Creole language MT exposed to more genre diversity then ever before, which outperforms a genre-specific Creole MT model on its own benchmark for 23 of 34 translation directions.

## motivation from northern brazil to the gulf of mexico, spanning an area including the caribbean and central american west coast, lies the creole "civilizational region" @xcite . one of its chief characteristics: a multiplicity of creole languages, born from contact of african language speakers with european languages in the colonial era. low-resource creole languages are widely spoken here and throughout the world @xcite @xcite . historic linguistic marginalization has stymied their technological advancement: few language technologies exist for these languages despite their many speakers @xcite . better mt could greatly benefit creole language speakers. many live in areas where their language is in the minority. panama and costa rica are home to communities of west indian descent who have maintained creole languages @xcite . large haitian-speaking communities live in the dominican republic @xcite , chile, mexico @xcite , brazil @xcite , and the bahamas @xcite @xcite . language is one of the first obstacles to immigrants' social integration, and many report daily reliance on mt @xcite . the lack or low accuracy of such technologies can thus contribute to social exclusion. multiple creole-speaking communities regularly fall victim to natural disasters @xcite @xcite . as the frequency of atlantic hurricanes may be accelerated by global climate change @xcite , machine translation can provide useful tools to facilitate communication during international relief efforts @xcite . yet colonial-era stigmas dismissing creole languages as broken or incomplete persist, and serve as justifications to advantage european languages at their expense @xcite . association with lower economic status and limited use in official settings then inhibit data collection and natural language processing (nlp) development for these languages @xcite . we expand on @xcite to unify community efforts in advancing creole nlp. in addition to meeting community needs, creole language mt presents avenues of exploration for low-resource nlp. many creole languages have documented linguistic relationships with highresource languages, as well as lexical and morphosyntactic proximity to each other @xcite . (see § 3.) as such, they have potential for cross-lingual transfer @xcite , a powerful technique for low-resource nlp @xcite . this potential presents an opportunity to develop technologies for many creole languages at once. but since stateof-the-art nlp methods rely on machine learning, this development is not possible without data. we present an expansive dataset for mt of creole languages, as a meaningful first step towards developing their technologies. we contribute: some prior works have approached creole language mt, like the inclusion of early haitian-english mt in the diplomat system @xcite and the 2011 workshop on statistical machine translation (wmt) @xcite . creole languages that are the focus of more recent mt research include haitian and jamaican @xcite , naija @xcite , mauritian @xcite , sranan tongo @xcite , and singlish @xcite . additional nlp research has been conducted in sentiment analysis and named entity recognition for naija @xcite @xcite ; syntactic analysis for singlish @xcite , naija @xcite , and martinican @xcite ; and inference for jamaican @xcite . these works, while valuable steps forward, have been limited in their scope of languages. the prior work most comparable to ours is the recent creole-val @xcite , which focused on building datasets for 28 creole languages. they focused on machine comprehension, relation classification, and mt. @xcite mt dataset is a significant contribution to our work. they do not publicly release their mt datasets, however. while portions of the data we collected (including part of the cre-oleval set) are not publicly releasable, we release 11.6m aligned bitext sentences and 3.4m monolingual sentences in 40 languages. the genre coverage of creoleval's solely religious and bible text data is another limitation shared by prior works, which can preclude general-purpose mt. for example, @xcite haitian model trained on primarily religious texts achieves an impressive bleu score of 68.0 on its same-genre test set, but a score of 14.7 on a wikipedia-style test set. this indicates that creole language mt models trained on specific genres may not be applicable to general domains. we provide the most diverse creole language mt data yet, both in terms of languages and genres. (see § 3.3.) other multilingual mt works include some creole languages. in addition to @xcite five included creole languages, @xcite multilingual lego-mt model supports 8 creole languages (out of 433 total). our work is a significant expansion on this, making a meaningful contribution to the broader effort of lowresource dataset curation-including the wmt'23 shared task on the topic @xcite , which our work follows by involving noisy data filtering @xcite and document alignment @xcite . ## methodology and dataset the languages we include in our study are in ## modeling experiments we now describe the experimental setup to show the utility of our datasets by training nmt systems. we train and release models on our full datasets, but we also train on only the data we publicly release, to show its stand-alone utility. we experiment with cleaning our train sets versus leaving them as-are, and in some experiments we fine-tune mbart @xcite ) rather than training from scratch. ## results and discussion figure 3 shows the performance of all models, tested for all available language pairs. we use chrf @xcite because previous studies @xcite show that it correlates better with human judgments than bleu @xcite , and because semantic metrics such as comet @xcite and bleurt 0 25 50 75 100 acf-eng acf-fra *aoa-eng bah-eng brc-eng bzj-eng bzj-fra bzk-eng cab-eng cab-fra cri-eng crs-eng crs-fra *dcr-eng djk-ara djk-ceb djk-deu djk-eng djk-fra djk-nep djk-zho *fab-eng *fng-eng fpe-eng gcf-eng gcf-fra gcr-eng gcr-fra *gpe-eng gul-eng gul-fra *gyn-eng hat-ara hat-aze hat-deu hat-eng hat-fra hat-nep hat-zho icr-eng icr-fra jam-eng jam-fra kea-eng kea-fra *kea-hat *kea-spa kri-eng kri-fra ktu-eng ktu-fra lou-eng *mart1259-eng mart1259-fra *mfe-ara *mfe-aze *mfe-deu mfe-eng mfe-fra *mue-eng *pap-ara *pap-aze *pap-deu pap-eng pap-fra *pap-nep *pap-por *pap-spa *pap-zho pcm-eng pcm-fra pov-eng *pre-eng rcf-eng sag-eng sag-fra srm-eng srm-fra srn-eng srn-fra svc-eng *tpi-deu tpi-eng tpi-fra trf-eng *wes-eng scratch-uncleaned (→) scratch-cleaned (→) mbart-cleaned (→) scratch-uncleaned (←) scratch-cleaned (←) mbart-cleaned (←) 0 25 50 75 100 acf-eng *aoa-eng bah-eng brc-eng bzj-eng bzk-eng cri-eng crs-eng *dcr-eng djk-ara djk-ceb djk-deu djk-eng djk-fra djk-nep djk-zho *fab-eng *fng-eng fpe-eng gcf-eng gcf-fra *gcr-eng gcr-fra *gpe-eng *gul-eng gyn-eng hat-ara hat-aze hat-deu hat-eng hat-fra hat-nep hat-zho icr-eng jam-eng kea-eng kea-fra *kea-hat *kea-spa *kri-eng *ktu-eng lou-eng *mart1259-eng mart1259-fra *mfe-ara *mfe-aze *mfe-deu mfe-eng mfe-fra *mue-eng *pap-ara *pap-aze *pap-deu pap-eng *pap-fra *pap-nep *pap-por *pap-spa *pap-zho pcm-eng pov-eng *pre-eng rcf-eng sag-eng srm-eng srn-eng svc-eng *tpi-deu tpi-eng trf-eng *wes-eng scratch-uncleaned (→) scratch-cleaned (→) mbart-cleaned (→) scratch-uncleaned (←) scratch-cleaned (←) mbart-cleaned (←) figure 3 : chrf scores on our newly created test sets using models trained on the all (top) and public (bottom) splits of our datasets. given x-y pair, → and ← represent the x to y and y to x translation, respectively. zero-shot pairs are marked with an '*' sign. @xcite lack support for low-resource languages. 18 because many mt researchers have a more intuitive grasp of bleu than chrf, we also include bleu scores in figure 5 of appendix c. as expected, high-resource pairs exhibited better translation quality than their lower resource counterparts, with some deviations from this trend possibly due to inevitable non-uniformity in test set genre. some pairs like trinidadian-english exhibited chrf up to 84, and some zero-shot pairs like annobonese-english scored as low as 0. despite the generally lower performance on their languages, we highlight the potential value of our dataset's smallest bitexts. as stated in § 1, creole languages' linguistic relationships open the possibility of powerful cross-lingual transfer. previous studies @xcite @xcite have shown that mt models trained on numerous languages can often be adapted to translate new languages with few examples. our results corroborate this finding; despite having only 391 parallel sentences to split for miskito coast creole (bzk), our models achieved bleu and chrf up to 43.7 and 60.4, respectively, translating into english. vincentian creole (svc) performed even better: up to 55.5 bleu and 66.9 chrf into english, despite only 321 parallel sentences total. (such scores would likely not be attainable via traditional bilingual training on such small sets @xcite .) strikingly, we perceived even higher performance on zero-shot martinican-to-english translation, with 69.9 bleu and 84.9 chrf. we hope future studies will reveal more about our dataset's potential for low-resource cross-lingual transfer, and how it interfaces with dataset genre and diversity. this is an important exploration, not only to engineer better low-resource language systems, but as a scientific inquiry with general implications for low-resource language technologies. impact of corpora cleaning: in figure 3 , regardless of the use of public or all data, models trained on cleaned data (triangles and stars) typically outperform their counterparts trained on non-cleaned data (circles and diamonds), for higher resource languages. cleaning eliminates noise and reduces variability in the especially noisy creole datasets, which helps translation quality. however, cleaning can hurt performance by reducing the already scarce data for the lower-resource, more noisy corpora (like pichi-english with 259 sentence pairs). does fine-tuning help? from figure 3 , a comparison of scratch and mbart models trained on cleaned data (squares and x's) reveals that, while models trained from scratch exhibit strong results, fine-tuning on mbart is better for both public and all data use. these fine-tuned models scored up to 8 chrf better than scratch models. scratch models converged after 500k training steps of 33k-token batches, and mbart fine-tuning needed 500k-600k steps of only 8.2k-token batches. while this implies that mbart is data efficient, mbart fine-tuning itself is slow due to training about 700% more parameters. indeed, when comparing the total training time, mbart fine-tuning needed up to a week whereas scratch training needed only up to two days on the same hardware. comparison on existing test sets: figure 4 compares our best models against results reported by creoleval @xcite on relevant language pairs from the creoleval test sets. since creoleval data is not public, we consider it apt to evaluate our best model (mbart) fine-tuned on all cleaned data. overall, our models surpass creoleval models with +6.4 average chrf @xmath0.1 average chrf @xmath1. our dataset is much larger and more domain diverse than creoleval's; hence our improved results on most language pairs show that increasing data tends to be beneficial even if it reduces the domain specificity. we note that on a few language directions (9 of 34), creoleval's model still beat ours, indicating the possibility for negative interference from a more diverse train set in some instances. in appendix c we provide some additional comparisons of bilingual versus multilingual fine-tuning and evaluations on other existing benchmarks: lego-mt @xcite and kreolmorisienmt @xcite . ## conclusion in this work, we compile the most comprehensive dataset to date for mt of creole languages in the americas. by aggregating disparate previous works and incorporating new data sources via scraping the web and pdfs, we expand mt datasets to 21 new languages and produce the largest and most genre-diverse in 20 more. we release translation models in 172 language directions, with 23/32 beating state-of-the-art benchmark performance, as well as a public dataset with 11.6m aligned bitexts and 3.4m monolingual sentences. a large multilingual bitext like ours has potential to build the best yet or first ever mt models for many languages, something we accomplished on a surface level in this work but hope future works will continue. the data present a number of other potential uses, including: (1) training language models for applications like spelling correction @xcite @xcite ; (2) availing textual data for applications like speech recognition and speech translation, which can be vital to low-literacy communities @xcite @xcite ; (3) potential to study cross-lingual transfer between creole languages and their phylogenetic relatives in yet unseen depth @xcite ; (4) research on the effects of linguistic data augmentation for small mt datasets (focusing on creole languages' unique linguistic position); (5) development of mt-assisted documentation tools for those languages that are endangered @xcite ; (6) the introduction of a common repository where any researchers in creole nlp can accumulate datasets together and advance in collaboration @xcite ; etc. we hope that this work will provide valuable translation technologies to communities that have been historically under-served and inspire community-oriented efforts to further expand work on these low-resource languages.