Auxiliary Subword Segmentations as Related Languages for Low Resource Multilingual Translation (2022)

Abstract: We propose a novel technique that combines alternative subword tokenizations of a single source-target language pair that allows us to leverage multilingual neural translation training methods. These alternate segmentations function like related languages in multilingual translation. Overall this improves translation accuracy for low-resource languages and produces translations that are lexically diverse and morphologically rich. We also introduce a cross-teaching technique which yields further improvements in translation accuracy and cross-lingual transfer between high- and low-resource language pairs. Compared to other strong multilingual baselines, our approach yields average gains of +1.7 BLEU across the four low-resource datasets from the multilingual TED-talks dataset. Our technique does not require additional training data and is a drop-in improvement for any existing neural translation system.

## introduction multilingual neural machine translation (nmt, @xcite ) models are capable of translating from multiple source and target languages. besides allowing efficient parameter sharing @xcite these models facilitate inherent transfer learning @xcite that can especially benefit low resource languages @xcite @xcite . a common technique to address lexical sharing and complex morphology in multilingual nmt is to decompose longer words into shorter subword units @xcite . since subword units are produced using heuristic methods, not all subwords are created equally. this can put low-and extremely low-resource languages at a disadvantage, even when these languages are paired with a suitable high resource language. to diminish the impact of rare subwords in nmt, @xcite leverage ciphertexts to augment the training data by constructing multiple-views of the source text. "soft" decomposition methods based on transfer learning @xcite address the problem of sub-optimal word segmentation with shared character-level lexical and sentence representations across multiple source languages @xcite . @xcite addressed this problem with a multiviewsubword regularization technique that also improves the effectiveness of cross-lingual transfer in pretrained multilingual representations by simultaneously finetuning on different input segmentations from a heuristic and a probabilistic tokenizer. while subword-regularization methods @xcite have been widely explored in nmt, this work is the first to study them together with multilingual training methods. concretely, we construct pairs of "related languages" by segmenting an input corpus twice, each time with a different vocabulary size and algorithm for finding subwords; we use these "languages" (really, views of the same language) for multilingual training of an nmt model. we propose multi-sub training, a method that combines multilingual nmt training methods with a diverse set of auxiliary subword segmentations which func-на@@ тура@@ льна , мы працуем , мы рых@@ ту@@ ем настаўні@@ каў . мы выкла@@ даем правы жанчын , правы чалавека , прынцы@@ пы дэ@@ ма@@ кра@@ ты@@ і , права@@ пара@@ дак . мы право@@ дзім разнастай@@ ныя трэні@@ н@@ гі . but of course , we &apos;re doing all our work , we were giving teacher training . we were training women &apos;s rights , human rights , de@@ mo@@ cr@@ acy , rule of law . we were giving all kind@@ s of training . figure 1 : an illustration of the interaction between the primary (bpe) and auxiliary (sp) subwords for the same sample from the be-en dev set where each type of segmentation is treated as a separate language. the model is taught to translate into a specific segmentation via multilingual training using the target "language" tags [2bpe] and @xcite . the sentence in bold type font shows both variants of the source sentence translating to the same target sentence. the colored spans show different segmentations of the same word(s) in source/target. we also introduce a cross-teaching technique in which a model is trained to translate source sentences from one subword tokenization into target sentences from a different subword tokenization. by using multi-sub training together with crossteaching, we obtain strong results on four lowresource languages in the multilingual ted talks dataset outperforming strong multilingual baselines, with the most significant improvements in the lowest-resource languages. in addition to improving the bleu scores, our technique captures word compositionality better leading to improved lexical diversity and morphological richness in the target language. multi-sub with cross-teaching is better at clustering different languages in the sentence embedding space than previous methods including multi-sub without cross-teaching. ## auxiliary segmentation as a related language pairing related languages is common in multilingual nmt 1 : nguyen and chiang (2017) combine uzbek/turkish and uzbek/uyghur; johnson et al. (2017) study multilingual translation to and from english with pairs such as spanish/portuguese or japanese/korean. neubig and hu (2018) pair low resource languages like azerbaijani with a related 1 here we do not distinguish between languages which are related in the linguistic sense (having some genetic affiliation) and those which are related in a more pragmatic sense of having high lexical overlap. rather than segmenting a corpus with a single tokenizer prior to training a translation model, we produce multiple segmentations using different tokenizers. consider the example sentences in figure 1. on both the source and target sides, the same sentence is represented using both byte-pair encodings (bpes, @xcite , with a "@@" separator) and in parallel as sentencepieces (sp, kudo 2018, with a " " separator). each segmentation uses a different vocabulary size, which guarantees that their subword sequences are to some extent distinct. the two tokenizations still resemble one other in many ways: (i) they have a nontrivial degree of lexical overlap (mostly between subwords which do not fall along word boundaries); (ii) they share the same grammatical structure, as both represent the same underlying language; and (iii) both sequences have the same semantic interpretation. we thus refer to the two segmentations as a pair of "related languages". applying two segmentations to a parallel corpus yields a total of four "languages": the source and target represented as bpe subwords, and the same represented using sp subwords. we obtain two source "languages" (each containing data from both high and low resource languages) and two target "languages". using this four way configuration, we train a model following a common multi-lingual training method @xcite : depending on the segmentation we want to translate into, we prepend a target token [2bpe] or [2sp] to the source side. we explore two different multilingual training configurations: bpe [src] → bpe [tgt] sp [src] → sp [tgt] cross-teaching: in addition to [bpe+sp], in this setting, each source sentence with a particular segmentation is translated into the target with alternate segmentation. this multilingual model is therefore trained on the following pairs: bpe [src] → sp [tgt] sp [src] → bpe [tgt] using multilingual training, our model is able to transfer information between bpe and sp segmentations in much the same way that conventional multilingual models transfer information between languages with a shared linguistic affiliation. unlike data augmentation techniques which generate synthetic training data, multi-sub training uses only the content of the original training corpus. furthermore, contrary to other works which employ multiple segmentations @xcite , multi-sub training and crossteaching do not affect model architecture and do not require specialised training. thus multi-sub training can be used as a simple, drop-in improvement to an existing neural translation model. ## qualitative analysis we list translations for the baseline sub-sep and sde models along with our multi-sub model in ## related work several techniques have been proposed to improve lexical representations for multilingual machine translation. @xcite propose to first train a hrl parent model, then transfer some of the learned parameters to the lrl child model to initialize and constrain training. similarly, @xcite pair related languages together and transfer source word embeddings from parent-hrl words to their child-lrl equivalents. @xcite ; @xcite , on the other hand, learn a joint vocabulary over several languages and train a single nmt model on the concatenated data. @xcite introduce a latent embedding space shared by all languages to enhance parameter sharing in lexical representation. @xcite ; @xcite use a similar idea but use character n-gram encodings (sde) instead of the conventional subword/word embeddings. by contrast multi-sub does not in11 in this respect, cross-teaching has a similar effect to bpedropout @xcite , which serves to eliminate monolingual subspaces at the level of subword embeddings (but recall our prior comments on the distinction between bpe-dropout and multi-sub in section 3.3). ## conclusion this work introduces multi-sub training with cross-teaching-a novel technique that combines multiple alternative subword tokenizations of a source-target language pair-to improve the representation of low-resource languages. our proposed methods obtain significant gains on lowresource datasets from multilingual ted-talks. we performed exhaustive analysis to show that our methods also increase the lexical and morphological diversity of the output translations, and produce better multilingual representations which we demonstrate by performing zero-shot ner by exploiting representations from a high resource language. multi-sub training and cross-teaching are simple architecture-agnostic steps which can be easily applied to existing single or multilingual neural machine translation models and do not require any external data.