C reole V al: Multilingual Multitask Benchmarks for Creoles (2024)

Abstract: Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research. While the genealogical ties between Creoles and a number of highly resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.

## introduction despite efforts to extend advances in natural language processing (nlp) to more languages, creoles are markedly absent from multilingual benchmarks. as such, progress towards reliable nlp for creoles remains impeded, and consequently there is a dearth of language technologies available for the hundreds of millions of people who speak creoles around the world. the omission of creoles from such benchmarks can be attributed to two key factors: modality and stigmatization. the first, modality, is a notable factor as some creoles are rarely used in writing, and thus text-based nlp is largely moot, highlighting a need for efforts in speech technology for creoles. the latter, stigmatization, is perhaps the most salient of the two, however. as the history of many creole languages is intricately interwoven with broader western imperialism, colonialism, and slavery, creole languages are often subjected to the stigmas and prejudices stemming from these historical atrocities @xcite . on the surface, social prejudices against creoles may seem extraneous in the context of nlp. however, the consequences of this stigmatization are palpable in preventing data collection for these languages. for example, it can be greatly challenging to collect data for a language without official status in a given country, even if it is the most widely used language by the populace; common sources for language data like government documentation, educational materials, and local news may not be available. moreover, even if a creole is someone's primary language, sociolinguistic barriers 1 rooted in stigma may further prevent people from using it in various contexts, making opportunities for gathering data even more sparse. lastly, even when financial resources are available to compensate crowd-workers, logistical challenges can significantly impede data collection efforts for creole languages @xcite . stigmatization of creoles is also an ongoing issue in the scientific domain, which further inhibits work in nlp. indeed, this prejudice is deeply ingrained in linguistics, manifested in the common misconception that creoles are incomplete or under-developed languages, in direct opposition to concepts like linguistic relativism and universal grammar @xcite @xcite . this othering of creoles that has occurred in linguistics has led to a research landscape where creoles are typically categorized as exceptions among languages, and thus separated from other languages. take, for example, the widely used wals database @xcite , which lists creoles as having the language family ''other''; works in nlp or computational linguistics relying on wals to sample languages from diverse range of families as a part of their methodology consequently exclude creoles from their work @xcite @xcite @xcite . 2 beyond wals, this pattern of exclusion is palpable across nlp, as demonstrated by the marked absence of creoles in works investigating multilinguality through the lens of language families @xcite @xcite . and while other resources exist to specifically cater to @xcite , the creation of separate resources to specifically accommodate creoles is emblematic of their ghettoization within scientific spaces. in this vein, though creoles are the singular focus of this work, our datasets, code, and models will allow others to easily incorporate creoles into broader variety of projects, thus helping remedy the isolation of creoles across nlp. ## respectively,bert-base-nli-mean-tokens, bert-large-nli-mean-tokens, xlm-r-bert-base-nli-mean-tokens, xlm-r-100langs-bert-base-nli-mean-tokens. in contrast to xlmr, which is pre-trained over a wider variety of domains. previous works on multilingual factuality also observe mbert outperforming xlmr @xcite . ## natural language generation of creoles unlike nlu, where the model aims to predict an accurate label, natural language generation (nlg) is arguably a more challenging task as models should generate output that is adequate as well as fluent. a lack of data-both in terms of size and domain-further complicates nlg for creole languages. in this paper, we introduce 2 new machine translation (mt) datasets for creoles. the first covers 26 creoles with text drawn from the religious domain, and the second is a small, but very high-quality, hatian creole dataset in the educational domain. we also conduct experiments and evaluate performance on a pre-existing mt dataset for mauritian creole. ## conclusion in this work, we have addressed the absence of creole languages from contemporary nlp research by introducing benchmarks and baselines for a total of 28 creole languages. we argue that this omission in previous work has hindered the progress of nlp technologies tailored to creole-speaking populations, in addition to preventing research communities from exploring the unique linguistic situations of this diverse group of languages. with the introduction of creoleval, we have made a significant step towards bridging the gap between creole languages and other low-resource languages in nlp. we hope that the public release of our datasets and trained models will serve as an invitation to further research in this relatively unexplored domain, and expect that nlp and computational linguistics research stand to gain significantly from embracing the linguistic and cultural diversity embodied in this group of languages.