D IALOGUE P IDGIN T EXT A DAPTATION VIA C ONTRASTIVE F INE -T UNING
A BSTRACT
The surging demand for multilingual dialogue systems often requires a costly labeling process for each language addition.
For low resource languages, human annotators are continuously tasked with the adaptation of resource-rich language utterances for each new domain.
However, this prohibitive and impractical process can often be a bottleneck for low resource languages that are still without proper translation systems nor parallel corpus.
In particular, it is difficult to obtain task-specific low resource language annotations for the English-derived creoles (e.g. Nigerian and Cameroonian Pidgin).
To address this issue, we utilize the pretrained language models i.e. BART which has shown great potential in language generation/understanding – we propose to finetune the BART model to generate utterances in Pidgin by leveraging the proximity of the source and target languages, and utilizing positive and negative examples in constrastive training objectives.
We collected and released the first parallel PidginEnglish conversation corpus in two dialogue domains and showed that this simple and effective technique is suffice to yield impressive results for English-to-Pidgin generation, which are two closely-related languages.
I NTRODUCTION
Task-oriented dialog systems are becoming prevalent in various daily activities such as ticket booking and restaurant reservations
Peng et al.
In a typical task-oriented dialog system, the natural language generation (NLG) module plays a crucial role by converting semantic representations into responses in natural language Langkilde & Knight (1998).
As such, NLG plays a significant impact on the users’ experience.
Unfortunately, these dialogue systems are mostly built for high resource languages such as English, which makes it less user-friendly in regions where English is not widely-spoken or is spoken differently Khalil (2020); Yusupujiang & Ginzburg (2021).
This includes African countries such as
Nigeria and Cameroon where the demand for these technologies are growing at a rapid rate Khalil (2020).
There is then a need to adapt 1 /translate these existing well-developed dialogue systems in high resource languages into its target language counterparts.
However, techniques in machine translation are still under-developed as some language pairs are completely devoid of parallel corpus Adelani et al.
; Chang et al.
This situation is further exacerbated by the fact that the targeted
NLG requires a domain-specific translation and annotation.
To this end, we present a preliminary study on the domain-specific adaptation of high resource
English language model into the English-related Pidgin languages consisting of the Nigerian (Naija) and Cameroonian Pidgin (Yaounde).
In particular, we explore on a low resource scenario where the goal is to translate the English utterance into conversation (dialogue) Pidgin sentences.
This scenario consists of a few parallel data and a larger quantity of monolingual text in both languages.
To facilitate the research, we collected the first parallel English-Pidgin dialogue corpus and release it along with the public Nigerian (Naija) Ogueji & Ahia (2019) and Cameroonian Pidgin spoken corpus Green et al.
We further propose a simple technique that allows to finetune a pretrained
English language model into generating Pidgin that can be used to collect dialogue text in dialogue systems.
To do so, we employ the technique of contrastive learning where the off-the-shelf English language model adjusts its useful prior knowledge into producing Pidgin, which is a closely-related language Ayafor (2008);
Faraclas (2008).
This process is enhanced by having the model to discern between positive/negative examples as defined by (1) the in/out-domain English utterances, and (2) the English and Pidgin texts.
Overall, we made the following contributions:
We release the first parallel data consisting of ∽ 200 Nigerian/Cameroonian Pidgin and English pairs in multiple dialogue domains including the restaurant and drone simulation.
We showed the efficacy of the proposed constrastive finetuning technique as being both simple and effective in creating natural Pidgin text with high-fidelity.
R ELATED W ORK
Contrastive learning has been widely used in various tasks – language modeling (Huang et al., 2018), unsupervised word alignment (Liu & Sun, 2015), caption generation (Mao et al., 2016; Vedantam et al., 2017), and machine translation (Yang et al., 2019).
Representations are learned with contrasting positive pairs and negative pairs: Chopra et al.
(2005); Weinberger & Saul (2009);
Schroff et al.
(2015) leverage a triplet loss to separate positive examples from negative examples in metric learning.
Chen et al.
(2020) shows that contrastive learning can boost the performance of self/semi-supervised learning in computer vision tasks.
In natural language processing, contrastive learning has been widely used.
(Mikolov et al., 2013) predicts neighbouring words from context with noise-contrastive estimation (Gutmann & Hyv ¨ arinen, 2012) while Logeswaran & Lee (2018) samples two contiguous sentences for positive pairs and the sentences from other document as negative pairs.
Our work is in the same vein where positive/negative examples are provided to enhance cross-lingual sentence representations.
R ELATIONS OF P IDGIN TO E NGLISH
Variations of Nigerian Pidgin are spoken across West and Central Africa in countries such as Benin,
Ghana, and Cameroon Faraclas (2013) as a result of contacts between Africans and English-speaking sailors and traders Gilman (1980) 2 .
Altogether, these languages evolved into a closely related group of languages whose vocabulary is predominantly English, spoken by Africans in West Africa and by their descendants in the Western Hemisphere Gilman (1980).
Importantly, we postulate that the relatedness of the Pidgin language to English can be utilized to formulate an effective contrastive learning setup such that an English language model can be readily adapted into its Pidgin variant.
This assumption is not unfounded as English is the lexifier language of Pidgin language Gilman (1980), thereby consisting of a large amount of loaned English vocabulary.
This motivates our use of an English language model as a strong prior to generate Pidgin text.
M ETHODOLOGY
In order to adapt the English language model to generate Pidgin dialogue texts, we propose a contrastive learning framework to expose the model to various valid or incorrect output sequences for a given input sentence.
Specifically, we train the model in a two-stage process where for each stage its contrastive loss goes below the threshold 0 .
(1) Stage 1 is the domain-targeting phase where the language model is finetuned to generate indomain dialogue utterances.
As such, the primary source of positive samples are the English dialogue
Stage 1: + (Dialogue English) :
There is a pub Blue Spice in the riverside area.
− (English) : Mauritius was voted Vice President.
Stage 2: + (Pidgin) :
Na for di main general hospital for di regional capital, Mekelle dem dey treat sick pipo.
+ (Dialogue Pidgin) : One pub Blue Soice dey for riversde area.
− (English) : We treat our youth differently because they are just coming up. utterances.
Further, we include a general-domain, non-conversation English texts.
Non-conversation
English texts are closer in the embedding space to the dialogue English texts, so they serve as a meaningful source of negative examples that help to pull the embedding projections to separate in/out domain texts.
(2) On the other hand, stage 2 is the language-converting phase that primarily aims to ensure that the model learns to distinguish Pidgin from English sentences.
In this phase, all monolingual Pidgin texts are defined to be the positive samples.
In order to induce the language model to generate the corresponding Pidgin texts in stage 2, one source of negative samples are thus the randomly sampled
English sentences that exclude the dialogue English texts.
We obtained these monolingual English sentences from a general domain and force the model to predict them to be negative.
Lastly, when available for the setting, we consider the few-shot Pidgin dialogue utterances obtained from the dialogue system can be used as positive samples.
We display some examples in Figure 1.
Both stages follow the contrastive learning framework (Chen et al., 2020), where we train the model to learn the representations of the ground truth sentence by contrasting the positive pairs with the negative pairs.
We project the source and target text sequences onto the latent embedding space.
Then we maximize the similarity between the pair of source and target sequence; while minimizing the similarity between the negative pairs as follows:
[Equation]
z ( i ) =
[ h ( i ) 1 · · · h ( i ) T ] ∈ R d × T is a concatenation of the decoder hidden states h ( i ) t of the target sentence y
( i ) across the sequence of length T for sequence S .
Sim ( · , · ) is a cosine similarity function and τ is the temperature factor set to 0 .
Sequence-to-sequence Finetuning.
To fine-tune the pretrained BART Lewis et al.
(2020) models for generation, we assume a dataset where each example ( x, y ) is an (English, Pidgin) pair.
We train the student model using the standard cross entropy loss:
[Equation]
where T in the target sequence length p is the model’s predicted probability for the correct word.
As such, the total objective for each batch update is given as L total =
L seq + L cont .
P IDGIN D IALOGUE C ORPUS
To create the Pidgin dialogue corpus, we consider two publicly available English dialogue datasets (i.e. E2E Novikova et al.
(2017) of restaurant domain and DroneParrot 3 ) )
and translate the English utterances into Pidgin.
We show the statistics for each datasets in Table 1 and observe that
the
Pidgin language generally has less characters (e.g. “kontri” as opposed to “country”) – owing to the evolution of its make-shift morphology where words are spelled in a reduced way based on its spoken, colloquial form Gramley & P¨atzold (2004).
[Table] [Table]
While the language tend to have a smaller vocabulary set than its lexifier (i
[Table]
e. English) 4 , Pidgin diverges profusely in terms of syntax.
Thus, the difference in word-ordering is where the anticipated challenges come from in terms of utterance adaptation.
E XPERIMENTS
Training Details.
We fine-tune the large BART Lewis et al.
(2020) model for 200 steps using an
Adam optimizer Kingma & Ba (2014) with β 1 = 0 .
9 , β 2 = 0 .
1 weight decay, 0 .
1 dropout, 0 .attention dropout, 0 .
1 label smoothing, 6% warmup steps and a learning rate of 3e-5 (See Appendix).
The final outputs are generated using beam search with a beam size of 3 .
Compared Approaches.
We compare the proposed technique with contrastive learning setup with various methods including simply using the source as target ( COPY ):
NMT : It is a semi/un-supervised technique that takes sentences from bilingual/monolingual corpora in two different languages and maps them into the same latent space via iterative back-translation Lample et al.
In addition, we include the supervised bidirectional training objectives.
XLM
-R :
This approach finetunes the pretrained multilingual language model in Conneau et al.
(2020) on parallel data following the proposed objectives as in Ours as we now discussed.
Ours : As our proposed approach, the pretrained language model BART Lewis et al.
(2020) is finetuned as in §4 where BART simply uses the objective L seq , and Stage-1 and Stage-2 are incrementally added.
Experimental Scenarios.
To validate our approach, we finetune BART with both Naija and
Yaounde utterances.
In Table 2, Un-Naija means that only the monolingual English and Naija texts are provided.
Naija and Yaounde means training on parallel/monolingual data and testing on their respective languages; while N+Y means training on both Naija/Yaounde data and testing on Yaounde.
R ESULTS AND A NALYSIS
In Table 2, we display the results for experiments in both unsupervised and few-shot scenarios.
We observe that even with an extremely small amount of annotations, the proposed technique ( Ours ) consisting of both training stages outperforms all benchmarks ( NMT and XLM-R ) by as much as 28 . 09 BLEU – which also goes to show that having multilingual representation ( XLM-R ) is not more beneficial.
Next, we show the effectiveness of the contrastive learning setup where BART without contrastive losses is substantially lower than BART+Stage-1 by 2 .
BLEU points, and 6 .
49 points lower than Ours .
This shows that providing the positive/negative examples can help project the embedding space into more accurate representations for each examples.
Further, we observe a consistent lower scores for the Yaounde Pidgin – which corroborates with past findings that indicates its influence from other sources of European languages such as the Portuguese and
French Gilman (1980).
However, when both Naija and Yaounde corpus are combined, we observe a drop in performance, which suggests that the morphological differences between the two languages (i.e. Naija and Yaounde) are interfering with the latent representations, causing it to performing sub-par.
[Table]
Human Evaluations.
We also perform human evaluation with two expert who are well-versed in both English and Pidgin and show the results on naturalness and wrong in Table 3.
Wrong is a content selection metric that measures how accurate the generated Pidgin texts (See Appendix) are adhering to the dialogue semantic frames.
We observe that results on both metrics are consistent with the BLEU-4 scores where our proposed approach consistently generate more natural text with fewer mistakes on both Naija and Yaounde .
Embedding Space Separation.
To get a clear picture of the sentence representations, we visualize the respective embedding projections in Figure 2.
We show the before and after projections of English and Pidgin utterances, which shows that the clusters of dialogue English and Pidgin , and general
Pidgin sentences are more separated.
C ONCLUSION
In this paper we show that the proposed two-stage training approach can help to adapt the high resource English conversation texts into natural, high-fidelity Pidgin sentences in low resource scenarios.
Moreover, we conclude that while Naija and Yaounde are two similar languages, augmenting with either languages provide no further improvements, while separating the representations with contrastive objectives is hugely beneficial.
A T RAINING C ONFIGURATIONS
All models were trained on 1 Nvidia V100 GPUs (32GB and CUDA Version 10.2) for 4k steps and averaged over 10 initialization runs for approximately 20 minutes each.
All models are selected based on optimal validation BLEU4.
Sentences are generated with a beam size of 5 .
B P IDGIN (N AIJA ) G ENERATION Reference
[Table]