Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for G uinea- B issau Creole (2025)

Abstract: We introduce a new dataset for machine translation of Guinea-Bissau Creole (Kiriol), comprising around 40 thousand parallel sentences to English and Portuguese. This dataset is made up of predominantly religious data (from the Bible and texts from the Jehovah’s Witnesses), but also a small amount of general domain data (from a dictionary). This mirrors the typical resource availability of many low resource languages. We train a number of transformer-based models to investigate how to improve domain transfer from religious data to a more general domain. We find that adding even 300 sentences from the target domain when training substantially improves the translation performance, highlighting the importance and need for data collection for low-resource languages, even on a small-scale. We additionally find that Portuguese-to-Kiriol translation models perform better on average than other source and target language pairs, and investigate how this relates to the morphological complexity of the languages involved and the degree of lexical overlap between creoles and lexifiers. Overall, we hope our work will stimulate research into Kiriol and into how machine translation might better support creole languages in general.

## introduction the proliferation of datasets and models for a wide range of the world's languages has significantly expanded the coverage of machine translation (mt) technologies in recent years. nevertheless, despite their often significant numbers of speakers, creole languages are digitally underrepresented and have minimal representation in mt technologies @xcite . this is, at least in part, because creoles have historically been subject to stigmatisation, particularly in administrative and educational settings @xcite . for many creoles, like other low-resource languages, translations of religious texts form the largest parallel digital resources available @xcite , but language technologies trained on religious data alone have been shown to have limited applicability in non-religious contexts @xcite . religious texts not only focus on theological concepts but are also rooted in specific socio-cultural contexts which may not align with those of the region where an mt model would be used @xcite . using religious texts for nlp applications and models may also pose particular harms when implemented in practice @xcite , making it especially important to evaluate models trained on religious data with a view to the downstream domain. in this work, we look at guinea-bissau creole (also known as kiriol foot_0 ), a portuguese creole spoken primarily in guinea-bissau. with approximately 350,000 native speakers and 1.5 million l2 speakers, it is used by over 90% of the population of guinea-bissau and is the de facto language of national identity @xcite . despite this, digital support for kiriol is low, with religious texts and dictionary resources being virtually the only public digital material in kiriol currently available to our knowledge. thus, our aims are twofold: to introduce a dataset for kiriol which provides resources for building mt models, and to investigate how this data can be best utilised to train translation models for general (non-religious) domains. our contributions are as follows: ## dataset here we introduce our dataset, taken from three online resources: bible.com, foot_2 monthly publications from the jehovah's witnesses (jw) website, foot_3 , and a portuguese-kiriol bilingual dictionary. 5 we extract parallel aligned bible verses from the almeida revista e corrigida (portuguese), 6 the new international version (english) 7 and the traduson antigu (kiriol). 8 we extract english, portuguese and kiriol paragraphs from online editions of jw's watchtower magazine (wt), a monthly bible study resource, and a small number of paragraphs from the jw monthly article series "how your donations are used" 9 . both of the jw resources are available in a wide range of low-resource languages, and while they are religious-themed, they use more informal language than the biblical texts and include discussion of more general topics such as finance, communication, transport, relationships, and technology. as such, we describe these as semireligious data. for both jw resources, we align the extracted paragraphs using html tags, meaning these entries can consist of multiple sentences. finally, we extract 6,902 vocabulary items in kiriol and portuguese from the bilingual dictionary. we use nllb (nllb, 2024) to automatically translate the portuguese vocabulary items to english, removing any translations that are more than two words long to remove likely errors. this creates a three-way parallel lexicon of 1,983 lexical items. we also extract 1,603 kiriol and portuguese glosses from the dictionary, using nllb to translate the portuguese sentences into english, then manually checking and correcting the english translations with reference to the original kiriol glosses. the dictionary sentences are not from the religious domain, but cover a wide range of day-today topics such as food, family, work and lifestyle. we describe this as 'general domain' data. in total, we have 38,578 parallel sentences (see we construct our training and test datasets to best reflect real-world translation scenarios. the bilingual dictionary sentences cover a broader range of topics in the general domain. we therefore take 1,000 of these sentences to comprise our out-ofdomain test set. we construct a validation set of 500 jw wt sentences and 500 bible sentences (1,000 total). this leaves us with 36,578 training sentences from the bible, jw wt, jw donations and bilingual dictionary, covering a range of domains but with mostly religious data. ## models we describe here the training procedure for all our translation models, which are trained from-scratch using an encoder-decoder transformer architecture @xcite . this is implemented using the eole toolkit, 10 an updated version of the open-nmt toolkit @xcite . for each model, we train a byte-pair encoding tokeniser @xcite with a vocabulary size of 10k (unless otherwise specified), implemented in sentence piece @xcite . in each experiment, the tokeniser is trained on the same data as used for the model training. we then train models for up to 20k steps using the adam optimiser (kingma and ba, 2015), validating with bleu @xcite every 1,000 steps with an early stopping of 5 validations: the best model is then evaluated on the test set. 11 for each experiment, we run five seeds on all four language directions (kiriol-portuguese, kiriol-english, portuguese-kiriol, english-kiriol). this setup improves the robustness of the results, even where bleu scores are low. for each experiment, we report the average test and validation bleu scores over the five seeds in appendix b. a full list of model parameters is found in appendix a. ## experiments we run a series of experiments to explore the utility of the different datasets for training models from scratch to generalise across domains, and to investigate differences in performance between translation directions and language pairs. ## conclusion and future work we introduce a new dataset for the machine translation of guinea-bissau creole, comprising around 40k sentences. this dataset is mostly from the bible and jehovah's witnesses texts, with a small amount of general-domain data, and we use this to investigate how to optimise real-world translation performance with mostly religious text. while all of our translation models show limited overall performance, our results provide insights into both the utility of different data types for domain generalisation and also the consistent differences in translation performance over different language pairs. while the performance of all of the models we train is limited, and human vali-dation shows that overall results remain poor, we show that adding even a few hundred sentences of the target domain dictionary data to the bible and wt training data considerably improved bleu scores on the test set, indicating that this may be a strategy worth exploring in other low-resource scenarios. collecting a small amount of manually translated sentences in the target domain should be possible in many low-resource mt settings where there is a specific domain of interest, and we show that these small sets can also be oversampled in the training data mix to further improve performance. we also investigate two well-known properties of creole languages, morphological simplicity and lexical overlap with a lexifier language. we show that differences in morphological complexity between kiriol and english or portuguese appear to result in differential tokenisation efficiency, whereby more tokenisation space is allocated to the more morphologically complex languages which disadvantages the more morphologically-simple kiriol. we also show that using shared embeddings results in better average model performance over all languages, but that the difference is most pronounced for kiriol-portuguese and portuguese-kiriol models, indicating that the overlap in vocabulary between kiriol and portuguese can be better leveraged during training with shared embeddings. there are many avenues for future research even within these limited data scenarios, including exploring tokenisation strategies that specifically benefit creole-lexifier translation scenarios, or investigating how lexical overlap impacts other areas of model training outside of tokenisation and learning the embedding space; for example, by exploring patterns in attention mapping over input sentences. however, developing functional translation models for kiriol will require either using kiriol data to finetune pre-trained multilingual models, as in @xcite , or collecting much more domain-diverse data from the kiriol community to create better quality from-scratch models. as has been made clear elsewhere @xcite , those research avenues will require careful consultation with the kiriol-speaking community, to explore how they may want kiriol to be represented-or not-in digital tools and technologies, and to find ways of making such research and language technology development maximally accessible to and beneficial for kiriol speakers.