J am P atois NLI : A Jamaican Patois Natural Language Inference Dataset (2022)

Abstract: JamPatoisNLI provides the first dataset for natural language inference in a creole language, Jamaican Patois.Many of the most-spoken low-resource languages are creoles. These languages commonly have a lexicon derived from a major world language and a distinctive grammar reflecting the languages of the original speakers and the process of language birth by creolization. This gives them a distinctive place in exploring the effectiveness of transfer from large monolingual or multilingual pretrained models. While our work, along with previous work, shows that transfer from these models to low-resource languages that are unrelated to languages in their training set is not very effective, we would expect stronger results from transfer to creoles. Indeed, our experiments show considerably better results from few-shot learning of JamPatoisNLI than for such unrelated languages, and help us begin to understand how the unique relationship between creoles and their high-resource base languages affect cross-lingual transfer. JamPatoisNLI, which consists of naturally-occurring premises and expert-written hypotheses, is a step towards steering research into a traditionally underserved language and a useful benchmark for understanding cross-lingual NLP.

## introduction the extensive progress that has been made in nlp research in recent years has largely been constrained to around 20 of the 7000 languages spoken around the world @xcite . creole languages, which emerge as a result of contact between speakers of different vernaculars, are even further underexplored @xcite . this work contributes to addressing this gap. we present jampatoisnli, the first natural language inference dataset in jamaican patois, which is an english-based creole spoken in the caribbean. additionally, to our knowledge, no other natural language inference corpus exists for any other creole language. jamaican patois is one of over 100 creole languages spoken by millions of inhabitants of different regions across the world, including africa, the caribbean, the americas, islands in the indian ocean and the pacific ocean (including australia and the philippines) and south asia @xcite . though there has been a recent spike in interest in work on low-resource languages in the nlp community @xcite @xcite @xcite @xcite @xcite , creoles in particular are extremely under-explored in spite of the prevalence of their usage globally @xcite . working more with this class of languages is an important step in ensuring that the benefits of nlp technology are more equitably distributed globally. additionally, the class of creole languages is a uniquely interesting point of study within the space of multilingual nlp. though creoles like jamaican patois have distinct morphosyntactic features, they often share significant lexical overlap with the high-resource base languages from which they are derived. this makes it possible to study cross-lingual transfer between high-resource and low-resource languages that are distinct, but share similar lexicons. in particular, jampatoisnli provides a benchmark for nlp researchers working to understand cross-lingual transfer to languages outside the training data of large pretrained multilingual models. creole languages like jamaican patois have the unique property of being outside the pretraining data of these models, yet highly related to their base languages, which are present in the datasets used to train the models. 1 jampatoisnli was constructed using both naturally occurring and newly constructed utterances of jamaican patois rather than through translation. this mitigates the problem of skewed cross-lingual transfer results which arises when the test dataset consists of translated examples but the training dataset does not @xcite . this also enhances the ecological validity @xcite of the dataset, as it is grounded in real world usage of the language and is thus a more relevant, realistic benchmark. these two features mean that work done with the dataset will be particularly useful for moving towards developing technologies for speakers of the language. we run studies on jampatoisnli transferring from monolingual english bert, multilingual bert, monolingual english roberta and multilingual xlm-roberta, finetuned on the multi-nli dataset, in zero-shot and fewshot settings. we find that monolingual english roberta (76.50%) and multilingual xlm-roberta (75.17%) achieve similar accuracies when we use the entire few-shot jampatoisnli training dataset with 250 examples for further finetuning. we also find that the monolingual english bert model (66.17 %) and the multilingual bert model (65.33 %), achieve similar accuracies when we use the entire few-shot jampatoisnli training dataset. in our experiments, the robertabased models strongly outperform the bert-based models. additionally, we find that few-shot performance on jampatoisnli increases much faster (with respect to the number of few-shot training examples) than on languages in americasnli, which have no strong connection to a high-resource language @xcite . lastly, we run qualitative experiments which leverage the relatedness between jamaican patois and english to understand which differences between the languages boost or inhibit the effectiveness of cross-lingual transfer. we hope that jampatoisnli prompts long-term research into building nlp tools that consider the particular difficulties and opportunities of nlp for jamaican patois and creole languages in general. the input to the task is a pair of sentences: the premise and the hypothesis. the goal is to output a label -entailment, neutral or contradiction -to describe the relationship between the pair. various approaches have been used to create nli corpora. the stanford nli (snli) @xcite , multi-nli (mnli) @xcite and adversarial nli (anli) (williams et al., 2020) english datasets, esxnli spanish dataset @xcite original chinese natural language inference (ocnli) dataset @xcite and codemixed hindi-english dataset @xcite all consist of a mixture of pre-existing sentences and crowdsourced sentences. in the japanese realistic textual entailment corpus, a collection of pre-existing sentences are filtered and paired using machine learning methods then manually annotated with labels @xcite . other nli corpora have been made using translation techniques. the natural language inference in turkish (nli-tr) dataset @xcite was created using amazon translate on snli and mnli. the cross-lingual nli (xnli) corpus @xcite was created by collecting and crowd-sourcing 750 examples then hiring human translators to translate the sentences into 15 languages. extensions of this dataset to low-resource languages such as americasnli @xcite and indicxnli @xcite have been created using human and machine translation methods. however, subsequent research has found that translation-based approaches to creating datasets can introduce subtle artifacts which can lead to skewed accuracies for cross-lingual transfer methods @xcite . jampatoisnli mitigates this problem by using original rather than translated examples. in spite of the examples given above, generally, there is a relative dearth of datasets and research into methods for low-resource languages across nli and other tasks. low-resource languages can be defined as those which are 'less studied, resource scarce, less computerized, less privileged, less commonly taught or low density' @xcite . creole languages in nlp. creole languages are typically low-resource. these languages arise through the process of creolization of another class of languages called pidgins. pidgins emerge as a result of contact between two or more groups of speakers which do not have a common language. a pidgin evolves to become a creole when it becomes the native language of the children of its speakers @xcite . 2 within the nlp community, a few datasets for different tasks have been created for creoles using a variety of methods. naijasenti is a twitter humanannotated sentiment analysis dataset which is partly comprised of 14,000 tweets in nigerian-pidgin or naija, which is an english-based creole @xcite . the authors find that codeswitching between these languages and english is a common feature in the dataset. they explore language adaptive finetuning and zero-shot cross lingual transfer from multilingual pretrained models, and achieve promising results. cross-lingual choice of plausible alternatives (xcopa) @xcite ) is a multilingual dataset for causal common sense reasoning in 11 languages, one of which is haitian creole, that was created by translating english copa. the authors find that across the languages in the dataset, translation based-approaches outperform methods which employ multilingual pretraining and finetuning. a part-of-speech tagging and dependency parsing corpus for colloquial singaporean english (singlish), an english-based creole, has also been created @xcite and further expanded @xcite using the universal dependencies @xcite scheme. the dataset was created by crawling pages on online singaporean forums. other work has also explored using machine learning methods for identifying and generating creole text. @xcite use contrastive learning to finetune bart @xcite so that the model produces novel dialogue texts in naija and yaounde (both english-based creoles). soto (2020) uses a fasttext @xcite based supervised classifier to identify instances of sentences in guadeloupean creole within a multilingual dataset. the use of machine learning models on creole languages has also been investigated. @xcite find that standard language models work better than distributionally robust ones on creoles, which shows that these languages are relatively stable. @xcite show that ancestor-tocreole transfer is non-trivial. ## experiments and results across our experiments, our goals are to: the experiments that we conduct are done in the zero-shot and few-shot settings. ## discussion we see that the relatedness between jamaican patois and english strongly contributes to the effectiveness of cross-lingual transfer in both zero-shot and few-shot settings. additionally, although natural language inference is a higher order reasoning task, our models achieved relatively high accuracy on the jampatoisnli dataset by learning the task from mnli examples in english. a natural question that arises based on these results, is whether vocabulary overlap is the primary factor that led to the boost in effectiveness of transfer learning in these experiments, or whether a higher order notion of similarity is a larger factor. comparing zero-shot and few-shot accuracies for other languages that are closely related to english but do not share the same degree of vocabulary overlap as an english-based creole (such as german) might be an interesting line of future research. interestingly, though jamaican patois developed as a result of contact between speakers of english and speakers of west african languages (some of which are present in multilingual bert's and xlm-roberta's training corpus), the multilingual models were not more effective base pretrained language models than the monolingual models. another possible direction for future research might be to determine whether there are methods that allow for more effective leveraging of the multilingual characteristic of the models during finetuning for creole target languages. ## conclusion jampatoisnli is a natural language inference dataset in an english-based creole, constructed from existing and novel examples of jamaican patois. our experiments show that the language's relatedness to english significantly boosts the effectiveness of cross-lingual transfer, even for the higher order task of natural language inference in both zero-shot and few-shot settings. we hope that the creation of this dataset encourages further research in the field on methods to improve crosslingual transfer for creole target languages, and the creation of other low-resource language and creole language datasets.