Penalizing Divergence: Multi-Parallel Translation for Low-Resource Languages of N orth A merica (2022)

Abstract: This paper explores a special case in multilingual machine translation: so called multi-parallel translation, where the target data for all language pairs are identical. While multi-parallelism offers benefits which are not available in a standard translation setting, translation models can easily overfit when training data are limited. We introduce a regularizer, the divergence penalty, which penalizes the translation model when it represents source sentences with identical target translations in divergent ways. Experiments on very low-resourced Indigenous North American languages show that an initially deficient multilingual translator can improve by 4.9 BLEU through mBART pre-training, and 5.5 BLEU points with the strategic addition of monolingual data, and that a divergence penalty leads to further increases of 0.4 BLEU. Further experiments on Germanic languages demonstrate a improvement of 0.5 BLEU when applying the divergence penalty. An investigation of the neural encoder representations learned by our translation models shows that the divergence penalty encourages models to learn a unified neural interlingua.

## introduction bilingual neural translation models typically require millions of parallel sentences to achieve adequate quality. a vast majority of the world's languages lack sufficient parallel corpora, and translation efforts for low-resource languages have turned to multilingual methods, leveraging linguistic similarity to augment a deficient signal with plentiful, albeit sometimes noisy, data from related languages @xcite . in this paper, we explore a very specific multilingual translation setting: multi-parallel translation. here, models are trained on documents, such as the proceedings of the european parliament, collections of subtitles, and the bible. each sentence has translations in many languages, providing not just a bilingual signal, but one that is bilingual in many directions. we explore a massively multi-parallel document: the bible, which has hundreds of translations. while it represents a very restricted domain, the bible is the only parallel document available for many languages and multi-parallel translation is, therefore, of key importance for low-resource nlp. earlier work shows that multi-parallel translation systems can in practice deliver poor results when available training data for individual languages are very small @xcite . in this setting, the performance of the translation models degrades when the number of source languages is increased. adapting pre-trained multilingual models such as mbart has also not led to significant progress @xcite . for languages that are not closely-related to the languages in the model, highquality translation remains an unsatisfied goal. we hypothesize that this drop in performance is a consequence of an inability of the model to learn a neural interlingua @xcite , that is, a shared semantic representation for source languages. this prevents knowledge transfer between languages, degrading translation performance. to counteract this tendency, we present the divergence penalty-a modification to the standard loss of a multilingual translation system, which encourages identical encoder and decoder representations for parallel source sentences. our experiments on indigenous north american languages show several avenues for improving the quality of the embedding space. we are able to stabilize the embeddings through mbart pretraining 1 and the addition of monolingual corpora, with gains of up to 5.5 bleu in the super-low setting. it can be supplemented, however, with our di-vergence penalty, which further coerces the embeddings to adopt interlingual representations. both visual inspection of t-sne plots @xcite and quantitative examination of learned representations show that the divergence penalty encourages the encoder and decoder representations to cluster according to semantics of the source sentence, regardless of source language. this is not the first work to explore approaches to strengthening the embedding space of a multilingual translation model. @xcite use cross-lingual word embeddings. others have used explicit neural interlinguas @xcite , and multiple encoders with tied attention @xcite . more closely related to our work, @xcite apply an agreement objective which encourages similar representations for artificially code-switched sentence variants and the original sentences. @xcite use contrastive learning to encourage shared representations for semantically similar sentences which resembles our divergence penalty. finally, @xcite introduce an auxiliary loss which enforces multilingual similarity to improve zero-shot translation results. all of the aforementioned works, however, investigate translation in a substantially higher-resourced setting and none of them investigate multi-parallel translation. a second class of related research falls into the broad category of data augmentation for lowresource translation. @xcite demonstrate that neural translation can learn in low-resource settings without significant modifications, but that these systems can be very sensitive to hyper-parameter tuning. @xcite demonstrate that the expedient method of copying source data to the target can improve low-resource translation quality, while madaan and sadat (2020) further leverage back-translations to boost the signal of low-resource translations. likewise, @xcite extol the virtues of monolingual data and back-translations in low-resource settings. ## methods in a multi-parallel translation setting, each sentence in the training data has been translated into several languages. a standard multilingual transformer model learns from sentences in isolation, and does not leverage co-dependencies between source sentences in a multi-parallel scenario. we introduce the divergence penalty as an auxiliary loss which penalizes models which do not learn similar encoder and decoder representations for parallel sentences. within each training batch, we identify sentences that are parallel with each other (via their targets), and compute a pairwise cosine comparison of their representation vectors at each position in the sentence. in a true interlingua, these representations would be identical. the transformer modifies the embedding space at several points, and we compare three variants that utilize a snapshot of the embeddings at a specific point. first, we calculate the cosine distance after context-attention has been applied to the final layer of the encoder (ep). secondly, we calculate the distance on the output distributions (dp). finally, we sum the two together (both). given a sequence @xmath0, ..., r n of encoder, decoder or joint encoder and decoder representations, the divergence penalty takes the form: we then weight this distance (via a tunable hyperparameter α @xmath1, 1]), and add it to the batch loss. batches with parallel sources propagate higher loss if the model has learned divergent representations. batches that have no multi-parallel sentences see no modification. during training, all parallel translations of the same target sentence are added into the same batch. training is performed using the fairseq @xcite implementation of transformers, with 3 layers and 4 attentional heads. embedding dimensions are set at 512, while the feed-forward size is 1024. the model optimizes a label-smoothed cross entropy using adam(0.9, 0.98), and an inverse square-root learning rate schedule (5e-4 -1e-9). the model is run for 50 epochs, with the best model chosen via validation loss. these settings closely follow @xcite . α div is tuned for each model, with values in [0, 0.3] typically leading to the best results. ## data and architectures our experiments are conducted on bible data @xcite for three indigenous language families of north america: algic, athabaskan, and inuit-aleut. the target language in most of our experiments is english although we also train manyto-many translation systems. family data sets are constructed by concatenating individual language bibles, and prepending a language tag. we additionally perform experiments on germanic bibles @xcite . some languages have a full bible translation available, while most only have a subset. see for development and testing, we sample each book of the new testament at a rate of 1% for test, and a further 1% for development. all datasets are segmented using a joint source-target byte pair encoding with a vocabulary size of 16,000. explorations varying the vocabulary size suggested that this was a reasonable, stable choice for these data sets. 2 models are learned using a modified version of the fairseq @xcite implementation of transformers. each model is trained with 3 layers and 4 attentional heads, with an embedding size of 512, and a feed-forward size of 1024. models are optimized using adam(0.9, 0.98), and an inverse square root learning schedule starting at 1e-7. models are trained for a maximum of 50 epochs, with a batch size of 2000 tokens. dropout is 50%, while attentional dropout is 30%. ## experiments in our experiments, we investigate the performance of different multi-parallel configurations on our indigenous and germanic data. we start by training baseline bilingual x-to-english (2l) and multilingual f-to-english systems (m2e), where x is an indigenous or germanic language and f is one of the indigenous language families or the germanic family. we also continue training on mbart @xcite for a maximum of 50 epochs. 3 indigenous languages turning to the second sub-table (raw) in adding monolingual data mimics mbart, pushing both low and higher-resource languages beyond the divergence penalty alone. adding the encoder penalty on top of e2e does not improve results. however, the decoder penalty leads to a further average improvement of 0.3 bleu, while the combination both fares slightly worse. we hypothesize that monolingual data and the encoder penalty behave similarly, anchoring multilingual represen-lang. ## analysis to understand how the encoder and decoder penalties affect multi-parallel translation models, we use t-sne to plot the encoder representations of source encoder penalty baseline m2e nav: júdah bikéyahgi béthlehem nílíinii, júdahgi naat'áanii danilíinii bitahdóó doo akéédęę' ánít'ée da, [...] tli: 'judea nèk'e, kòta bethlehem gòyeh sìı ekǫ kòta gòlaa hazǫǫ̀ nahk'e wet'àaɂà hǫt'e. [...] eng: » bethlehem in the land of judah , you are not the least of the leading cities of judah . [...] sentences for our baseline multilingual athabaskan model and a model trained with the encoder penalty in figure 2 . 5 sentences are color-coded. for example, red dots include representations for one tlicho and one navajo source sentence with identical english translations. when we zoom into the red cluster, showing navajo representations in light blue and tlicho in green, we can see that representations for the baseline multilingual system (baseline m2e) form very tight clusters. however, in many cases, these cluster according to source language. this indicates that the model has failed to learn truly multilingual representations, which would instead cluster according to semantics of the source sentence, rather than source language. when we add in the encoder penalty (encoder penalty), we see a significant correction. tlicho and navajo representations for equivalent source sentences now intermingle in a joint red cluster. models supplemented by monolingual data also seem to cluster representations by meaning (included in appendix a), demonstrating a similar tendency which supports our conclusion that both monolingual data augmentation and the encoder penalty strengthen learning of shared multilingual representations. the decoder penalty does not seem to have a similar effect (see appendix a). the visual interpretation of t-sne plots is confirmed via mathematical analysis. we calculate the centroid of each cluster (formed by representations for a particular source language and sentence id), and determine the average cosine distance between centroids for clusters having the same sentence id across languages. formally, let r(i, l) be the set of representations for language l and sentence id 5 we first use pca to project into r 10 , and then use t-sne to further project the results to r 2 . t-sne plots ran for a maximum of 150,000 iterations, with perplexity of 30. ## conclusion multi-parallel translation has the ability to leverage cross-lingual information to supplement a weak low-resource translation signal, but not all languages can benefit. we have introduced a divergence penalty that forces multi-parallel models to learn shared embedding spaces that improve the quality of the translation. on its own, the penalty improves the quality of ultra low-resource indigenous translation by 4.1 bleu over a bilingual model, and by more than 4.3 bleu over a deficient multilingual alternative. furthermore, monolingual data provides a strong target for multilingual embeddings, but is complemented by our penalty for a further increase of 0.4 bleu. this trend continues even when the the number of large translation corpora is increased.