Findings of the First Shared Task for Creole Language Machine Translation at WMT 25 (2025)

Abstract: Efforts towards better machine translation (MT) for Creole languages have historically been isolated, due to Creole languages’ geographic and linguistic diversity. However, most speakers of Creole languages stand to benefit from improved MT for low-resource languages. To galvanize collaboration for Creole MT across the NLP community, we introduce the First Shared Task for Creole Language Machine Translation at WMT25. This Shared Task consists of two systems tracks and one data track, for which we received submissions from five participating teams. Participants experimented with a wide variety of systems development techniques. Our evaluation campaign gave rise to improvements in MT performance in several languages, and particularly large improvements in new testing genres, though some participants found that reusing subsets of pretraining data for specialized post-training did not yield significant improvements. Our campaign also yielded new test sets for Mauritian Creole and a vast expansion of public training data for two Creole languages of Latin America.

## introduction insufficient training data remains a pronounced barrier for creating natural language processing (nlp) systems that cater to lower-resourced languages. this is particularly pronounced for the task of machine translation (mt), due to the importance of aligned bitexts. for creole languages, a geographically and linguistically diverse group (see figure 1 ), the lack of training data brings some challenges common to other low-resource scenarios, but also offers unique opportunities, due to the role of contact in shaping them. many creole-speaking communities have expressed interest in having mt support for their language @xcite . however, efforts to create nlp systems for them have largely been fragmented. this is in part due to the truly expansive scope of the term "creole". by definition, a creole language exhibits linguistic influence from an amalgamation of languages, typically both high-and low-resource @xcite . naturally, the creole languages of africa (e.g. nigerian pidgin and sango) are often viewed with different historical and cultural lenses than those of the caribbean (e.g. haitian creole and papiamento) or pacific (e.g. bislama and tok pisin). however, this geographic and cultural fragmentation misses some of the notable commonalities among creole languages. many of the creole languages of africa and the caribbean have a shared linguistic history: most of the languages in both groups emerged from european colonialism and slavery in africa, and languages in both groups are strongly influenced by european and african (typically niger-congo) languages in similar ways @xcite . while pacific creole languages are not connected to africa, they are also influenced by largely the same european colonial languages (e.g., english, french, portuguese, dutch, etc.), and many show comparable and sometimes even more pronounced grammatical influence from oceanic languages @xcite . perhaps even more important than their linguistic commonalities, creole languages across the globe are subject to similar stigmas and play similar linguistic roles in relation to high-resource languages @xcite . hence, speakers of creole languages everywhere may have shared experiences in connection to their language and could benefit similarly from improved language technologies. only recent publications have addressed the data scarcity of large numbers of creole languages, a necessary hurdle to create systems that truly serve speakers. (see section 2.) these recent efforts have been ground-breaking @xcite . however, their large scope allowed only for treatment of creole languages as a conglomerate without particular attention to specific language communities and their distinct needs. to build technologies for the wide array of creole language speakers, participation from a variety of communities is needed. in an effort to foster collaboration in nlp across these communities, we hold the first shared task for creole language mt held at wmt25. 1 we detail prior publications that inspired the need for this shared task in section 2, and we overview the task organization in section 3. the shared task consists of two subtasks: a call for system submissions-itself consisting of two tracks, namely track 1 (constrained), track 2 (unconstrained)-and a call for dataset submissions. five teams participated. submissions and evaluation results are discussed in sections 4 and 5, respectively. ## related work creole languages are a product of intense linguistic contact. they usually draw most of their vocabulary from a single source language that is often referred to as the creole language's lexifier. 2 typologically, creole languages tend to be more isolat-ing than these lexifier relatives. however, attempts to define them as distinct typological class (e.g. @xcite . in light of the typological debate, some (e.g. mufwene, 2001) prefer to define creoles by their history, which is closely associated with the long-term movement of people speaking mutually unintelligible languages. today, languages recognized as creoles are spoken by over 180 million people @xcite . for some speakers, a creole language is their mother tongue, and these speakers may be monolingual; for others, a creole language can serve as a lingua franca within the broader, multilingual community. in surveying creole language speakers, @xcite find that they highly desired mt support for their languages. @xcite further highlights the opportunity for creole language technology to serve as a bridge between higher and lower-resourced communities. he emphasizes that in contexts where creoles act as a lingua franca, it may be more viable to develop language technologies for them, rather than going straight to even lower-resourced highly localized languages. despite the demand and utility of mt for creole languages, they occupy a small portion of works in the broader mt community. when they are included, works tend to focus on individual languages. for example, the 2010 earthquake in haiti prompted the rapid development of mt systems for the french-related haitian language @xcite . early work by @xcite ; @xcite on mauritian creole acknowledged this trend, though they too developed technologies for a single language. similarly, mt systems for west african pidgin @xcite are highly relevant for other english-related creole languages, like guyanese creolese @xcite , but it has been difficult to coordinate multi-continental efforts. the first effort towards a joint, large scale mt project for creoles was proposed in creoleval @xcite with the creolem2m model, covering n-way translation between english and 26 creoles. building on this work, @xcite contributed kreyòl-mt for 41 creole languages, including the first public, extensively multilingual mt dataset for creole languages, and focusing on those spoken by the african diaspora. the resulting models have set the current state-ofthe-art for creole language mt. even with these advances, efforts to build creole technologies are not strongly unified. for instance, while @xcite focus on africa and the americas, the potential relevance of these languages for mt efforts for pacific creoles (e.g., chavacano) @xcite remains an open question. perhaps most pertinently, because both creoleval and kreyòl-mt advanced developments for creole languages as a conglomerate, focus on specific languages and their distinct needs was outside the scope of efforts. in organizing this shared task we hope to foster the beginnings of broader collaboration in creole nlp. this way, members of specific creole language communities may have a place to develop their own communities' desired technologies. ## evaluation results we designed a shared evaluation process for the two system tracks. all language pairs in the received 10 https://github.com/n8rob/creolemt_wmt25 submissions were supported by kreyòl-mt, so only our shuffled kreyòl-mt public test sets were used for official evaluation. 11 for every language pair and direction, we computed the chrf++ @xcite and bleu @xcite scores of the submission using the default parameters of the sacrebleu library. we compared these results to the baseline scores of kreyol-mt-pubtrain, kreyol-mt, and cre-olem2m as stipulated in section 3.1. note that no two teams submitted a system for the same language pair, and therefore scores could only be compared with baselines. results for the constrained track are reported in chrf++ and bleu scores are reported in the primary krey-all submission corresponds to approaches with data all merged under the seychellois creole language tag (crs); con-trastive1 represents use of language-specific tags, and constrative2 indicates partial parameter freezing (with the last 4 encoders, all decoder layers and 11 this led to complications with the unconstrained systems track. because @xcite originally shuffled and split their private and public test sets independently, the kreyol-mt model trained on the private set has been contaminated with some of the segments in the public kreyòl-mt test sets we used to construct our eval sets. yet as a publicly available model, kreyol-mt was permitted for unconstrained submissions. this is what prompted our reevaluation of ehow submissions; see section 5.2.1. the shared embeddings fixed during training). the best model for @xmath0, however all four submitted systems score within one bleu point of each other. this, combined with the observation that krey-all came up with a different system ranking (one in which constrative2 performed worst of all, @xcite ) by using a different shuffle of the same test set, suggests that score differences are not significant. recall from section 4.1.2 that lm's into-french system employed curriculum learning with bleubased difficulty ranking. the system for opposite translation direction did not employ this, but used different train/dev split (70-30 instead of 90-10). in our official evaluation, both submissions under-performed the baseline, albeit by less than 2.0 chrf++. lm's own reporting a slight improvement over the baseline for @xmath1, again gives the impression that score differences are not significant. both lm and krey-all fine-tuned the kreyol-mt-pubtrain model for a single language pair. these were valuable experiments because such attempts had not been made previously. the original work of @xcite focused primarily on the kreyòl-mt dataset and did not include extensive experiments regarding how to engineer optimal mt systems from the available data. one question left open by their work was whether, if in the absence of additional data or models, significant improvements could be achieved by post-training on a select subset of training data. results from these two studies @xcite now indicate no such significant improvements, suggesting that either very different methods would be needed to improve performance, or that researchers may find more promise in developing new datasets and using external models (rather than post-training on subsets of kreyol-mt-pubtrain's own data). ## conclusions we note several noteworthy observations from the contributions of the participants: our evaluation led to some central lessons and takeaways. it was found that constrained systems, which attempted to boost mt performance for a particular language by post-training the kreyol-mt-pubtrain model on a subset of its own training data (pertaining to the language in question), did not result in significant performance improvements, even when authors searched across other training tactics and hyperparameters to maxi-mize performance. this suggests that developing new datasets and using pre-trained models may be a more promising direction. accordingly, our participants in the unconstrained systems track showed that such methods are often effective at improving results for some language pairs, and that even in cases where performance on the original test domain does not improve, new datasets and models can bring about expansion to new testing genres. takeaways and future ambitions the variety of training approaches developed and evaluated for this shared task provides valuable insights into the training of creole mt systems. new data collected during the campaign can also be incorporated into future creole mt datasets and models. in particular, this will allow us to train a new baseline model for the next iteration of the shared task. re-training the baseline model will also give us the chance to ensure that all updated versions of kreyòl-mt models online can be evaluated with the same test sets without contamination (a critical point for interpreting results). we are proud that a number of this year's shared task organizers represent various creole languagespeaking communities. our committee includes two l1 speakers of jamaican patois, one l1 speaker of martinican creole, one l2 speaker of louisiana creole, and one l2 speaker of haitian. in the future, we hope to incorporate members from a broader diversity of creole language communities, so that our efforts better serve the realistic needs of these communities.