Using Encipherment to Isolate Conditions for the Successful Fine-tuning of Massively Multilingual Translation Models (2025)

Abstract: When fine-tuning massively multilingual translation models for low-resource languages, practitioners often include auxiliary languages to improve performance, but factors determining successful auxiliary language selection remain unclear. This paper investigates whether syntactic similarity or lexical overlap is more important for effective multilingual fine-tuning. We use encipherment to create controlled experimental conditions that disentangle these confounded factors, generating novel languages with identical syntax but no lexical overlap, and conversely, languages that preserve lexical overlap. Through extensive NLLB-200 fine-tuning experiments across Europarl and AmericasNLP datasets, we demonstrate that lexical overlap is the dominant factor. Syntactically identical auxiliary languages provide negligible benefits ( 1.0 ChrF), while languages with significant lexical overlap provide substantial improvements ( 5.0 ChrF), with effectiveness strongly correlated to KL-divergence between token distributions (r = -0.47, p .001). Our findings provide clear guidance: when selecting auxiliary languages for multilingual fine-tuning, prioritize lexical overlap over syntactic similarity.

## introduction a popular modern approach to low-resource machine translation is the fine-tuning of massively multilingual encoder-decoder transformers @xcite . for example, the top two entrants in the americasnlp 2023 foot_0 shared task on machine translation into indigenous languages (which solicits systems that translate spanish into indigenous american languages) both used this technique @xcite . the teams from the university of sheffield @xcite and the university of helsinki @xcite fine-tuned distillations of @xcite simultaneously on all eleven language pairs of the shared task. simultaneous training on a set of related language pairs (as opposed to training a separate system per language pair) has frequently been reported to yield performance benefits @xcite . one survey @xcite on low-resource machine translation claims: "this is mainly due to the capability of the model to learn an interlingua (shared semantic representation between languages)". but when the parent model (as in the case of nllb-200) has already been pre-trained on 200 language pairs, when (and why) does it remain beneficial to simultaneously fine-tune on multiple language pairs? what do the fine-tuning languages learn from each other? in the context of multilingual language modeling, investigators have focused on two candidates: syntax and lexicon. according to a recent review @xcite : "in previous research, syntax has been suggested as potentially the most important linguistic contributor for better crosslingual transfer." 2 the same article also reports that "lexical overlap is particularly important when the pre-training corpus for the source language is small or when the word order between the source and target languages is dissimilar," but concludes that "lexical overlap is not a sufficient standalone factor to explain cross-lingual transfer." an obstacle to drawing definitive conclusions is the difficulty of isolating the confounding factors of shared syntax and lexicon -related languages typically share both. in this work, we use encipherment to disentangle these factors. encipher-figure 1 : we use encipherment to create statistically realistic languages that are unseen by pre-trained translation models, affording experimental control over the often-confounded factors of syntactic and lexical overlap. for the top two rows, we apply different encipherments to disjoint subsets of english-spanish europarl, creating two novel languages with identical syntax but no lexical overlap. for the bottom two rows, we apply the same encipherment to disjoint subsets of english-spanish and english-portuguese europarl, creating two novel languages that preserve the lexical overlap between spanish and portuguese. ment allows us to generate statistically realistic languages that have not been previously included in translation model pre-training, while also providing experimental control: the goal of this paper is to provide practical guidance to those seeking to build translation engines for low-resource languages. specifically, when fine-tuning a massively multilingual translation model like nllb-200, how should one select auxiliary languages to include in the fine-tuning (or should one include them at all)? we ultimately arrive at the following recommendations: ## preliminaries this paper uses the following formalisms to describe enciphered parallel corpora. let t and l be finite alphabets that respectively correspond to a token vocabulary foot_5 and a set of language ids (e.g. eng_latn, rus_cyrl, etc.). let t * be the set of all sequences of tokens from t . define a parallel corpus as a function π : @xmath0, where d ⊂ l × z + and ∀(l 1 , i), (l 2 , i) ∈ d, π(l 1 , i) and π(l 2 , i) have the same meaning (i.e. they are translations of one another). define an encipherment ϵ as a permutation foot_6 of token vocabulary t , i.e. a bijection ϵ : @xmath1. for a token sequence π(l, i) = ⟨t 1 , ..., t k ⟩ from parallel corpus π, denote the ϵ-enciphered sequence as π ϵ (l, i) = ⟨ϵ(t 1 ), ..., ϵ(t k )⟩. we extract a bitext from parallel corpus π using the following notation: ## experiment design the index sets i 1 , . . . , i k are pairwise disjoint. we use these bitexts to train several translation models: we fine-tune each model using a batch size of 64 for a maximum of 60,000 training steps. validation is performed every 500 steps, and training is terminated early if the validation loss does not decrease for five consecutive evaluations. we use the adafactor optimizer @xcite , as implemented in the transformers library @xcite , with a fixed learning rate of 1 × 10 -4 , disabling both parameter scaling and relative step sizing. gradient clipping is applied with a threshold of 1.0, and a weight decay of 1 × 10 -3 is used for regularization. we adopt a constant learning rate schedule with warm-up, increasing the learning rate linearly over the first 1,000 steps. we evaluate the resulting models by translating held-out test sets, then deciphering the translations and scoring them using standard machine translation metrics (e.g. bleu @xcite and chrf @xcite ). because one of the languages in our experiments is polysynthetic, we report results using chrf, but the choice of metric does not affect the experimental conclusions. we run 5 trials for each of the following training bitext sizes: 1024, 2048, 4096, 8192, and 16392 (so 25 trials in total). a bilingual fine-tuning trial consists of a fine-tuning for each bitext -the resulting systems are then evaluated and their scores are averaged. a multilingual fine-tuning trial is a single fine-tuning over all bitexts -the resulting system is then evaluated on each language pair, and these scores are averaged. ## base models we focus on fine-tuning nllb-200 models @xcite . these models were trained on a large-scale multilingual corpus covering 200 languages, using a transformer-based encoder-decoder architecture, following the general design of the m2m-100 model @xcite . several distillations of this model are provided, including a 600m parameter model (6 encoder/decoder layers, 768 hidden size, 12 attention heads) and a 1.3b parameter model (12 encoder/decoder layers, 1024 hidden size, 16 attention heads). to increase experimental throughput, our experiments focus on the 600m parameter model. ## datasets this section describes the parallel corpora that we use in our experiments. ## experiment 2: the impact of lexical overlap on the success of multilingual fine-tuning 8.1 lexical overlap of target languages the only difference between this experiment and experiment 1 is that we extract the following k bitexts from parallel corpus π: we use different languages but the same encipherment, so that shared tokens remain shared after encipherment (see the bottom two rows of figure 1 ). this produces unseen languages that have the same amount of lexical overlap as their unenciphered analogues. again, we freeze the encoder during fine-tuning to eliminate the confounding impact of encoder domain adaptation to the source language. ## conclusion this work addresses a fundamental question in lowresource machine translation: when fine-tuning massively multilingual models like nllb-200, which factors determine the success of multilingual fine-tuning with auxiliary languages? through controlled experiments using encipherment to disen-tangle syntactic similarity and lexical overlap, we provide empirical evidence that lexical overlap is the primary driver of performance improvements. our key findings are: even when auxiliary languages share identical syntax with the target language, multilingual fine-tuning shows little advantage over bilingual approaches. the benefits are most pronounced (but still minor) only for languages from families not represented in the pre-training corpus. the encipherment methodology introduced here also provides an experimental framework for future research on cross-lingual transfer, allowing researchers to control for confounding factors that typically make it difficult to isolate the impact of different linguistic properties. future work should explore whether these findings generalize to other massively multilingual models, investigate optimal methods for measuring and maximizing lexical overlap, and examine whether the relative importance of syntax versus lexicon changes with different model architectures or pre-training objectives.