Leveraging Adapters for Improved Cross-lingual Transfer for Low-Resource Creole MT (2024)

Abstract: ———– EXTENDED ABSTRACT INTRODUCTION ———–Creole languages are low-resource languages, often genetically related to languages like English, French, and Portuguese, due to their linguistic histories with colonialism (DeGraff, 2003). As such, Creoles stand to benefit greatly from both data-efficient methods and transfer-learning from high-resource languages. At the same time, it has been observed by Lent et al. (2022b) that machine translation (MT) is a highly desired language technology by speakers of many Creoles. To this end, recent works have contributed new datasets, allowing for the development and evaluation of MT systems for Creoles (Robinson et al., 2024; Lent et al. 2024). In this work, we explore the use of the limited monolingual and parallel data for Creoles using parameter-efficient adaptation methods. Specifically, we compare the performance of different adapter architectures over the set of available benchmarks. We find adapters a promising approach for Creoles because they are parameter-efficient and have been shown to leverage transfer learning between related languages (Faisal and Anastasopoulos, 2022). While we perform experiments across multiple Creoles, we present only on Haitian Creole in this extended abstract. For future work, we aim to explore the potentials for leveraging other high-resourced languages for parameter-efficient transfer learning.

## methodology and experiments to train adapters for haitian, we use monolingual data from nllb-opus @xcite , and the parallel creolem2m training split from creole-val @xcite . for evaluation, we leverage two evaluation datasets from creoleval: the creolem2m evaluation split and the mit-haiti corpus for mt; we also evaluate over @xcite an mbart-50 model fine-tuned on the kreyòl-mt dataset @xcite . furthermore, we apply dnllb-200, a 600m-parameter distillation of the original 54b-parameter nllb-200 model as a baseline @xcite . both models have 12 encoder and 12 decoder layers, 16 attention heads, and 1024 dimensions, and each model have their own model vocabularies of over 250,000 sentence-piece tokens shared across all languages. ## conclusion and future work while gains over baselines were reached via the monolingual adaptation, most creoles do not have lage amounts of web-scraped data, as found in nllb-opus. thus, the ability to leverage data 1 https://docs.adapterhub.ml/overview.html and transfer from closely related languages to creoles has great potential for bolstering creole mt. to this end, we plan experiments for parameterefficient transfer learning, inspired by @xcite , who found success with phylogenetically-motivated adaptation. the application of phylogenetic adaptation for creoles is not straight-forward, however. there is no consensus phylogeny of creoles or even their broader language families @xcite . simultaneously, previous works have shown that transfer learning to creoles from related languages is nontrivial @xcite @xcite . thus an important area of creole mt remains selecting favorable languages for transfer learning. in addition to phylogentic relation, we are exploring selection of transfer languages via embedding clustering. we cluster nllb-200 language token embeddings with cosine and euclidean distance, and identify afrikaans, igbo, and yiddish as the nearest neighbors of haitian. these are interesting findings, since igbo is one of haitian's hypothesized relatives @xcite , and yiddish and afrikaans are indo-european languages influenced by afroasiatic and south african khoisan languages, respectively-which appears analogous to haitian's mixed indo-european and niger-congo influences. we also explore measuring vocabulary subword evenness, as introduced by @xcite , as a more helpful language selection method than simple typological proximity. while experiments are still underway, these explorations will help establish the languages amenable to crosslingual transfer for creoles, and ultimately the degree to which cross-lingual adaptation methods can benefit speakers of creoles.