Few-Shot Pidgin Text Adaptation via Contrastive Fine-Tuning (2022)

Abstract: The surging demand for multilingual dialogue systems often requires a costly labeling process for each language addition. For low resource languages, human annotators are continuously tasked with the adaptation of resource-rich language utterances for each new domain. However, this prohibitive and impractical process can often be a bottleneck for low resource languages that are still without proper translation systems nor parallel corpus. In particular, it is difficult to obtain task-specific low resource language annotations for the English-derived creoles (e.g. Nigerian and Cameroonian Pidgin). To address this issue, we utilize the pretrained language models i.e. BART which has shown great potential in language generation/understanding – we propose to finetune the BART model to generate utterances in Pidgin by leveraging the proximity of the source and target languages, and utilizing positive and negative examples in constrastive training objectives. We collected and released the first parallel Pidgin-English conversation corpus in two dialogue domains and showed that this simple and effective technique is suffice to yield impressive results for English-to-Pidgin generation, which are two closely-related languages.

## introduction task-oriented dialog systems are becoming prevalent in various daily activities such as ticket booking and restaurant reservations @xcite . in a typical task-oriented dialog system, the natural language generation (nlg) module plays a crucial role by converting semantic representations into responses in natural language @xcite . as such, nlg plays a significant impact on the users' experience. unfortunately, these dialogue systems are mostly built for high resource languages such as english, which makes it less user-friendly in regions where english is not widely-spoken or is spoken differently @xcite . this includes african countries such as nigeria and cameroon where the demand for these technologies are growing at a rapid rate @xcite . there is then a need to adapt foot_0 /translate these existing well-developed dialogue systems in high resource languages into its target language counterparts. however, techniques in machine translation are still under-developed as some language pairs are completely devoid of parallel corpus @xcite . this situation is further exacerbated by the fact that the targeted nlg requires a domain-specific translation and annotation. to this end, we present a preliminary study on the domain-specific adaptation of high resource english language model into the english-related pidgin languages consisting of the nigerian (naija) and cameroonian pidgin (yaounde). in particular, we explore on a low resource scenario where the goal is to translate the english utterance into conversation (dialogue) pidgin sentences. this scenario consists of a few parallel data and a larger quantity of monolingual text in both languages. to facilitate the research, we collected the first parallel english-pidgin dialogue corpus and release it along with the public nigerian (naija) @xcite and cameroonian pidgin spoken corpus @xcite . we further propose a simple technique that allows to finetune a pretrained english language model into generating pidgin that can be used to collect dialogue text in dialogue systems. to do so, we employ the technique of contrastive learning where the off-the-shelf english language model adjusts its useful prior knowledge into producing pidgin, which is a closely-related language @xcite . this process is enhanced by having the model to discern between positive/negative examples as defined by (1) the in/out-domain english utterances, and (2) the english and pidgin texts. overall, we made the following contributions: contrastive learning has been widely used in various tasks -language modeling @xcite , unsupervised word alignment @xcite , caption generation @xcite , and machine translation @xcite . representations are learned with contrasting positive pairs and negative pairs: chopra et al. (2005); weinberger and saul (2009); schroff et al. (2015) leverage a triplet loss to separate positive examples from negative examples in metric learning. chen et al. (2020) shows that contrastive learning can boost the performance of self/semi-supervised learning in computer vision tasks. in natural language processing, contrastive learning has been widely used. @xcite predicts neighbouring words from context with noise-contrastive estimation @xcite while @xcite samples two contiguous sentences for positive pairs and the sentences from other document as negative pairs. our work is in the same vein where positive/negative examples are provided to enhance cross-lingual sentence representations. ## relations of pidgin to english variations of nigerian pidgin are spoken across west and central africa in countries such as benin, ghana, and cameroon @xcite as a result of contacts between africans and englishspeaking sailors and traders @xcite foot_1 . altogether, these languages evolved into a closely related group of languages whose vocabulary is predominantly english, spoken by africans in west africa and by their descendants in the western hemisphere @xcite . importantly, we postulate that the relatedness of the pidgin language to english can be utilized to formulate an effective contrastive learning setup such that an english language model can be readily adapted into its pidgin variant. this assumption is not unfounded as english is the lexifier language of pidgin language @xcite , thereby consisting of a large amount of loaned english vocabulary. this motivates our use of an english language model as a strong prior to generate pidgin text. ## methodology in order to adapt the english language model to generate pidgin dialogue texts, we propose a contrastive learning framework to expose the model to various valid or incorrect output sequences for a given input sentence. specifically, we train the model in a two-stage process where for each stage its contrastive loss goes below the threshold 0.1: both stages follow the contrastive learning framework @xcite , where we train the model to learn the representations of the ground stage 1: + (dialogue english): there is a pub blue spice in the riverside area. t ] ∈ r d×t is a concatenation of the decoder hidden states h (i) t of the target sentence y (i) across the sequence of length t for sequence s. sim(•, •) is a cosine similarity function and τ is the temperature factor set to 0.5. ## pidgin dialogue corpus to create the pidgin dialogue corpus, we consider two publicly available english dialogue datasets (i.e. e2e @xcite of restaurant domain and droneparrot 3 ) ) and translate the english utterances into pidgin. we show the statistics for 3 dialogue corpus for drone-human communication. ## results and analysis in ## limitations while there are substantial improvements over the translation model baseline, we also observe the limitation of the approach in overcoming extreme low resource conditions as in yaounde, which does not share much similarities with english. yaounde is mostly spoken in cameroon, and therefore evolved differently from naija pidgin. this influences how effective the contrastive objectives are in regulating the latent space of the models, which in turn limit the applications of the approach. moreover, we also notice some discrepancies between the contents of reference and generated outputs, meaning that the models tend to hallucinate fluent, unrelated sentences. this is therefore one other aspect that the approach is unable to address. ## conclusion in this paper, we show that the proposed two-stage training approach can help to adapt the high resource english conversation texts into natural, highfidelity pidgin sentences in low resource scenarios. moreover, we conclude that while naija and yaounde are two similar languages, augmenting with either languages provide no further improvements, while separating the representations with contrastive objectives is hugely beneficial.