Machine Translation into Low-resource Language Varieties (2021)

Abstract: State-of-the-art machine translation (MT) systems are typically trained to generate “standard” target language; however, many languages have multiple varieties (regional varieties, dialects, sociolects, non-native varieties) that are different from the standard language. Such varieties are often low-resource, and hence do not benefit from contemporary NLP solutions, MT included. We propose a general framework to rapidly adapt MT systems to generate language varieties that are close to, but different from, the standard target language, using no parallel (source–variety) data. This also includes adaptation of MT systems to low-resource typologically-related target languages. We experiment with adapting an English–Russian MT system to generate Ukrainian and Belarusian, an English–Norwegian Bokmål system to generate Nynorsk, and an English–Arabic system to generate four Arabic dialects, obtaining significant improvements over competitive baselines.

## introduction despite tremendous progress in machine translation @xcite and language generation in general, current stateof-the-art systems often work under the assumption that a language is homogeneously spoken and understood by its speakers: they generate a "standard" form of the target language, typically based on the availability of parallel data. but language use varies with regions, socio-economic backgrounds, ethnicity, and fluency, and many widely spoken languages consist of dozens of varieties or dialects, with differing lexical, morphological, and syntactic patterns for which no translation data are typically available. as a result, models trained to translate 1 code, data and trained models are available here: https: //github.com/sachin19/seq2seq-con from a source language (src) to a standard language variety (std) lead to a sub-par experience for speakers of other varieties. motivated by these issues, we focus on the task of adapting a trained @xmath0, having access only to limited monolingual corpora in tgt and no src-tgt parallel data. tgt may be a dialect of, a language variety of, or a typologically-related language to std. we present an effective transfer-learning framework for translation into low resource language varieties. our method reuses @xmath1. this allows for rapid adaptation of mt models to new varieties without having to train everything from scratch. using word-embedding adaptation techniques, we show that mt models which predict continuous word vectors @xcite rather than softmax probabilities lead to superior performance since they allow additional knowledge to be injected into the models through transfer between word embeddings of high-resource (std) and lowresource (tgt) monolingual corpora. we evaluate our framework on three translation tasks: english to ukrainian and belarusian, assuming parallel data are only available for @xmath2; english to nynorsk, with only english to norwegian bokmål parallel data; and english to four arabic dialects, with only @xmath3. our approach outperforms competitive baselines based on unsupervised mt, and methods based on finetuning softmax-based models. ## a transfer-learning architecture we first formalize the task setup. we are given a parallel @xmath4, which allows us to train a translation model f (•; θ) that takes an input sentence x in src and generates its translation in the standard veriety std, ŷstd = f (x; θ). here, θ are the learnable parameters of the model. we are also given monolingual corpora in both the standard std and target variety tgt. our goal now is to modify f to generate translations ŷtgt in the target variety tgt. at training time, we assume no src-tgt or std-tgt parallel data are available. our solution (figure 1 ) is based on a transformerbased encoder-decoder architecture @xcite which we modify to predict word vectors. following @xcite , instead of treating each token in the vocabulary as a discrete unit, we represent it using a unit-normalized d-dimensional pre-trained vector. these vectors are learned from a std monolingual corpus using fasttext @xcite . a word's representation is computed as the average of the vectors of its character n-grams, allowing surfacelevel linguistic information to be shared among words. at each step in the decoder, we feed this pretrained vector at the input and instead of predicting a probability distribution over the vocabulary using a softmax layer, we predict a d-dimensional continuous-valued vector. we train this model by minimizing the von mises-fisher (vmf) loss-a probabilistic variant of cosine distance-between the predicted vector and the pre-trained vector. the pre-trained vectors (at both input and output of the decoder) are not trained with the model. to decode from this model, at each step, the output word is generated by finding the closest neighbor (in terms of cosine similarity) of the predicted output vector in the pre-trained embedding table. we train f in this fashion using src-std parallel data. as shown below, training a softmax-based @xmath5. by replacing the decoder input and output with pretrained vectors, we separate the vocabulary from the mt model, making adaptation easier. now, to finetune this model to generate tgt, we need tgt embeddings. since the tgt monolingual corpus is small, training fasttext vectors on this corpus from scratch will lead (as we show) to low-quality embeddings. leveraging the relatedness of std and tgt and their vocabulary overlap, we use std embeddings to transfer knowledge to tgt embeddings: for each character ngram in the tgt corpus, we initialize its embedding with the corresponding std embedding, if available. we then continue training fasttext on the tgt monolingual corpus @xcite . last, we use a supervised embedding alignment method @xcite to project the learned tgt embeddings in the same space as std. std and tgt are expected to have a large lexical overlap, so we use identical tokens in both varieties as supervision for this alignment. the obtained embeddings, due to transfer learning from std, inject additional knowledge in the model. finally, to obtain a @xmath6, we finetune f on psuedo-parallel src-tgt data. using a @xmath7. naturally, these synthetic parallel data will be noisy despite the similarity between std and tgt, but we show that they improve the overall performance. we discuss the implications of this noise in §4. ## experimental setup datasets we experiment with two setups. in the first (synthetic) setup, we use english (en) as src, russian (ru) as std, and ukrainian (uk) and belarusian (be) as tgts. we sample 10m en-ru sentences from the wmt'19 shared task @xcite , and 80m ru sentences from the conll'17 shared task to train embeddings. to simulate lowresource scenarios, we sample 10k, 100k and 1m uk sentences from the conll'17 shared task and be sentences from the oscar corpus @xcite . we use ted dev/test sets for both languages pairs @xcite . the second (real world) setup has two language sets: the first one defines english as src, with modern standard arabic (msa) as std and four arabic varieties spoken in doha, beirut, rabat and tunis as tgts. we sample 10m en-msa sentences from the unpc corpus @xcite , and 80m msa sentences from the conll'17 shared task. for arabic varieties, we use the madar corpus @xcite which consists of 12k 6way parallel sentences between english, msa and the 4 considered varieties. we ignore the english sentences, sample dev/test sets of 1k sentences each, and consider 10k monolingual sentences for each tgt variety. the second set also has english as src with norwegian bokmål (no) as std and its written variety nynorsk (nn) as tgt. we use 630k en-no sentences from wikimatrix @xcite , and 26m no sentences from paracrawl @xcite combined with the wikimatrix no sentences to train embeddings. we use 310k nn sentences from wikimatrix, and ted dev/test sets for both varieties @xcite . preprocessing we preprocess raw text using byte pair encoding (bpe, @xcite with 24k merge operations on each src-std corpus trained separately on src and std. we use the same bpe model to tokenize the monolingual std data and learn fasttext embeddings (we consider character n-grams of length 3 to 6). 2 splitting the tgt words with the same std bpe model will result in heavy segmentation, especially when tgt contains characters not present in std. 3 to counter this, we train a joint bpe model with 24k operations on the concatenation of std and tgt corpora to tokenize tgt corpus following @xcite . this technique increases the number of shared tokens between std and tgt, thus enabling better cross-variety transfer while learning embeddings and while finetuning. we follow @xcite to train embeddings on the generated tgt vocabulary where we initialize the character n-gram representations for tgt words with std's fasttext model wherever available and finetune them on the tgt corpus. ## results and analysis synthetic setup considering std and tgt as the same language is sub-optimal, as is evident from the poor performance of the non-adapted sup(@xmath11. clearly, special attention ought to be paid to language varieties. direct unsupervised translation from src to tgt performs poorly as well, confirming previously reported results of the ineffectiveness of such methods on unrelated languages @xcite ). translating src to tgt by pivoting through std achieves much better performance owing to strong unsup(@xmath12. however, when resources are scarse (e.g., with 10k monolingual sentences as opposed to 1m), this performance gain considerably diminishes. we attribute this drop to overfitting during the pre-training phase on the small tgt monolingual data. ablation results (appendix c) also show that in such low-resource settings the learned embeddings are of low quality. finally, langvarmt consistently outperforms all baselines. using 1m uk sentences, it achieves similar performance (for @xmath13, softmax, and small gains over unsupervised methods. however, in lower resource settings our approach is clearly better than the strongest baselines by over 4 bleu points for uk (10k) and 3.9 points for be (100k). to identify potential sources of error in our proposed method, we lemmatize the generated translations and test sets and evaluate bleu @xcite . across all data sizes, both uk and be achieve a substantial increase in bleu (up to +6 bleu; see appendix d for details) compared to that obtained on raw text, indicating morphological errors in the translations. in future work, we will investigate whether we can alleviate this issue by considering tgt embeddings based on morphological features of tokens @xcite . ## discussion fairness the goal of this work is to develop more equitable technologies, usable by speakers of diverse language varieties. here, we evaluate the systems along the principles of fairness. we evaluate the fairness of our arabic multi-dialect system's utility proportionally to the populations speaking those dialects. in particular, we seek to measure how much average benefit will the people of different dialects receive if their respective translation performance is improved. a simple proxy for fairness is the standard deviation (or, even simpler, a max -min performance) of the bleu scores across dialects (a higher value implies more unfairness across the dialects) beyond that, we measure a system's unfairness with respect to the different dialect subgroups, using the adaptation of generalized entropy index @xcite , which considers equities within and between subgroups in evaluating the overall unfairness of an algorithm on a population @xcite (see appendix f for details and additional discussion). ## conclusion we presented a transfer-learning framework for rapid and effective adaptation of mt models to different varieties of the target language without access to any source-to-variety parallel data. we demonstrated significant gains in bleu scores across several language pairs, especially in highly resource-scarce scenarios. the improvements are mainly due to the benefits of continuous-output models over softmax-based generation. our analysis highlights the importance of addressing morphological differences between language varieties, which will be in the focus of our future work.