On Language Models for Creoles (2021)

Abstract: Creole languages such as Nigerian Pidgin English and Haitian Creole are under-resourced and largely ignored in the NLP literature. Creoles typically result from the fusion of a foreign language with multiple local languages, and what grammatical and lexical features are transferred to the creole is a complex process. While creoles are generally stable, the prominence of some features may be much stronger with certain demographics or in some linguistic situations. This paper makes several contributions: We collect existing corpora and release models for Haitian Creole, Nigerian Pidgin English, and Singaporean Colloquial English. We evaluate these models on intrinsic and extrinsic tasks. Motivated by the above literature, we compare standard language models with distributionally robust ones and find that, somewhat surprisingly, the standard language models are superior to the distributionally robust ones. We investigate whether this is an effect of over-parameterization or relative distributional stability, and find that the difference persists in the absence of over-parameterization, and that drift is limited, confirming the relative stability of creole languages.

## introduction a creole language arises if a pidgin, 1 developed by adults for use as a second language, becomes the native and primary language of their children. although a large portion of creole languages have their roots in western european colonialism and slavery, creole languages still serve as important lingua franca in multi-ethnic and multilingual communities, and creoles are often an important part of the local identity. moreover, there are more than a 1 a pidgin is a grammatically simplified language that develops between two or more groups that do not have a language in common. both pidgins and creoles are sometimes referred to as contact languages. hundred million speakers of creole languages world wide (fig. 1 ), with similar needs for technological assistance, and yet creoles are still largely absent from nlp research @xcite . haitian creole, for example, has 9.6 million speakers as of today; nigerian pidgin english has 100 million speakers, and singaporean colloquial english (singlish) has 3.5 million speakers. this paper sets out to collect existing resources for these three languages and provides language models for them. in doing so, we wish to take the nature of creole languages into account, not necessarily assuming that our best approaches to modeling non-creole language are also best for the creole languages. the nature of creole languages has been a matter of much debate in linguistics during the last decade @xcite : some see creole languages as natural stages in language change cycles (aboh, 2015), while others see them as a distinct typological class with unique characteristics, including, for example, a very simple morphology @xcite . another feature of creoles is that they exhibit significant variation across groups of speakers @xcite . @xcite goes as far as to call creoles a continua that cannot be captured under a single grammar. tamil mandarin(我们) cantonese(拍拖) english malay eng malay hakka(店) x standard english: "hey, when we date we always eat at the coffee shop" hokkien/ dey , wǒ men paktor always makan at kopi tiam one hey , we date always eat at coffee @xmath0: example sentence in singlish featuring multilingual vocabulary, chinese-style topic prominence combined with a subordinate clause with english word order, and a final interjection representing a discourse particle; a common feature of singlish. example from https://languagelog.ldc.upenn.edu/nll/?@xmath1. bajpai et al. ( 2017): (1) john sibei hum sup one. here, according to the authors, both sentences are valid utterances in singlish, and they both mean john is so lecherous, but the first would more likely come from a speaker of chinese, and the second from a malay speaker. from this, 2 we derive the conjecture that creole language models can benefit from learned mixtures of source languages. training on mixtures of source languages has been applied to language modeling of code-switched language @xcite , and it is clear from examples such as the one in fig. 2 that creole languages, at the sentence level, share commonalities with code-switched language, with vocabularies drawn from multiple source languages. to exploit synergies with learned mixtures of source languages, and to obtain robust performance across related, but unseen distributions, we explore ways of training creole language models with distributionally robust objectives @xcite . our results below, however, show that, somewhat surprisingly, this conjecture is probably not true, at least not in a straight-forward way. contributions we combine existing datasets and present pretrained language models for the following creole languages: nigerian pidgin english, singaporean colloquial english (singlish), and haitian creole. we perform intrinsic evaluation (word prediction), as well as extrinsic evaluation (part-of-speech tagging and named entity recognition). comparing language models trained with empirical risk minimization to languages models 2 creole languages clearly differ though in the dynamics that affect their drift. for example, yakpo (2021) discuss two seemingly similar creole languages, krio (sierra leone) and pichi (equatorial guinea). both creoles have english as their lexifier, but while krio is spoken alongside english, pichi is spoken alongside spanish. the two creoles, as a consequence, exhibit a clear difference. krio has converged increasingly toward english, while pichi has neither converged toward english nor spanish. ## creoles and corpora while creole languages are spoken by hundreds of millions, and are often a lingua franca within a larger community, only a handful of resources exist for creoles presently. some challenges to collecting data resources for creole languages can be a creole's non-standardized orthography, e.g. haitian creole @xcite , or the specific contexts in which creoles are used -it may not always be used in official capacities for news, education, and official documents, even if the creoles are widely used in most other aspects of life @xcite . this of course complicates data collection. in this work, we focus on the following creoles, as they each have diverse linguistic makeup and have some existing datasets: nigerian pidgin english west africa is one of the world's most linguistically diverse places, with nigeria alone having over 400 languages @xcite . recent work to advance african nlp has led to the creation of several datasets in nigerian pidgin english @xcite @xcite @xcite @xcite , which makes it particularly well-resourced in comparison to other creole languages. nigerian pidgin english, also referred to as simply nigerian pidgin, can further be understood as a member in the larger family of west african pidgins, as many west african countries have their own unique variation of this creole, but all share influences from many of the same languages, such as igbo, hausa, and yoruba. the first sizeable nigerian pidgin dataset comes from agić and vulić (2019), who collected parallel text from several magazines written by a religious society, which have parallel translations in many languages. this dataset has been utilized in the first attempts to develop baselines for machine translation of nigerian pidgin english @xcite . furthermore, ogueji and ahia (2019) 2019 ) also introduced a code-switching corpus of news articles and on-line comments in both nigerian standard english and nigerian pidgin. in this work, they discuss some challenges of working with nigerian pidgin, such as non-standardized spelling. they also find that different topics prompt code-switching to nigerian pidgin over nigerian standard english. more task-specific nigerian pidgin datasets have been introduced for universal dependency parsing @xcite , named entity recognition @xcite , sentiment analysis @xcite , and speech recognition @xcite . singlish singaporean colloquial english, also known as singlish, has english as a source language, but also draws parts of its grammar and vocabulary from languages such as mandarin, cantonese, hakka, hokkien, malay, and tamil. presently, few publicly available datasets exist in singlish, as this creole is primarily utilized for informal conversation between people and not for official purposes. the largest relevant corpus is the national university of singapore sms corpus from @xcite , which consists of over 67,000 text messages written by singaporeans. qualitatively, we observed that this dataset is much closer to standard english, albeit with noise from outdated sms language, than the example provided in fig. 2 , but, within this data, we still observe many hallmark features of singlish such as discourse markers and vocabulary from relevant languages. @xcite have also released a webcrawler that collects posts from an popular singaporean forum about hardware, where discussion is often in singlish. they use the resulting singlish corpus as part of their work to investigate the role of inflection for nlp with non-standard forms of english. beyond plain text corpora, @xcite introduced the first singlish universal dependency dataset, which was further expanded upon in @xcite . @xcite used this dataset as a low-resource language test case for their method of pretraining mbert @xcite . finally, a few studies have been done on private datasets for sentiment analysis @xcite , and polarity detection @xcite . haitian creole haitian creole exhibits a combination of french with many west african languages (e.g. igbo, yoruba, fon, etc.). haitian creole seized the attention of the machine transla ## datasets for creole language models we experiment with training language models for creoles with a mixture of creole data, and additional data from languages influential to each creole. data splits we begin with the creole datasets noted in tab. 1, and combine them with data of other higher-resource languages that have been influential to the creole. we combine a fixed number of these examples into a mixed-language dataset, as described in tab. 2. the mixed-language dataset for each creole includes in-formation about the original language of each sentence, so that we can form language-specific groups for dro (see §5.1 for more details on dro grouping). the total number of train and development examples were determined by the number of sentences in the base (creole) dataset for a 95-5 train-development split. singlish had equal representation of each language, with 53,006 examples per language, including singlish. haitian creole also had equally represented languages, with 8,192 examples for haitian and each additional language. for the nigerian pidgin mixed-language dataset, english, portuguese, and nigerian pidgin were composed equally with 67,615 examples each, and yoruba with only 27,260 examples due to the small size of the original data. thus, we included 95% of the yoruba wmt-news 2020 dataset. language identification within creoles as we will see in §6, training the language models on the mixed-language dataset with dro fails to produce positive results. following from this, we also create a creole-only dataset, composed of only the creole examples. in order to sort the creole examples into distinct groups for dro, we label each creole example by the collection of the selected languages present in the sentences, as determined by a language identification algorithm. 4 consider the following examples from their respective creole-only datasets: singlish: "treat him makah lah" en: 88.19%, ms: 4.34%, ta: 0.04%, and zh: 0.01% nigerian pidgin: "pikin wey like to play wit wetin no dey common and sabi one particular subject reach ground" en: 87.46%, pt: 0.23%, and yo: 0.03% haitian creole: "infomation sou kestion te tranble a ak lekol" fr: 3.50%, es: 0.08%, and yo: 0.01% while the language identification algorithm is not perfect, the confidence scores for the languages still reflect the high-level trends for the creole examples, namely, that english and malay ("makan") are indeed present in the singlish sample, and also that english and portuguese ("pikin", "sabi") are present in the nigerian pidgin example. however, for the haitian creole example, we see that none of our chosen languages have very high scores from the language identification algorithm, which begs the question: were there other languages with higher confidence from the language identification algorithm? to ensure that the languages we chose are wellrepresented in the creole examples, we looked at the distribution of the identified languages across examples in our creole-only datasets in fig. 3 . from this, we observe that choosing to identify languages specifically related to the creole (i.e. the same languages we included in the mixed-language datasets) is more reliable than trusting the language identifier pick the top five languages with the highest confidence -there appears to be a bias for falsely predicting european languages, even on creole data unrelated to these languages, as well as some strange outliers, such as tagalog being the third most commonly predicted language for haitian creole sentences. also, we see that haitian creole itself was a commonly identified language, which could explain the low confidence scores for french and spanish in the example above. finally, among our specifically chosen languages for the creoles, we see that, although the source language (e.g. english or french) is most dominant, the other languages are still well distributed, with the exception of yoruba. we surmise that the densely distributed, low-confidence scores for yoruba can probably be attributed to the fact that yoruba is a lower-resourced language. ## experiments in this section, we detail our experimental setups. we make our code and models publicly available. foot_2 ## conclusion in this paper, we bring creole languages to the attention of the nlp community. we collect data and train baseline language models for three creoles, and evaluate these models across the downstream tasks of part-of-speech tagging and named entity recognition. based on previous work suggesting the instability of creole languages @xcite , we explore the impact of using more robust learning objectives for masked language modeling of creoles, but our results show that vanilla empirical risk minimiziation is superior. we show that this is not the result of overparameterization or lack of regularization, but in-stead suggest this is a result of the relative stability of creole languages. we note that it still remains possible that significant improvements could be achieved by modeling dynamics specific to creole languages, i.e., the processes that govern their development, including social factors @xcite and second language acquisition dynamics @xcite .