Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024)

Abstract: LLMs have become a go-to solution not just for text generation, but also for natural language understanding (NLU) tasks. Acquiring extensive knowledge through language modeling on web-scale corpora, they excel on English NLU, yet struggle to extend their NLU capabilities to underrepresented languages. In contrast, machine translation models (MT) produce excellent multilingual representations, resulting in strong translation performance even for low-resource languages. MT encoders, however, lack the knowledge necessary for comprehensive NLU that LLMs obtain through language modeling training on immense corpora. In this work, we get the best both worlds by integrating MT encoders directly into LLM backbones via sample-efficient self-distillation. The resulting MT-LLMs preserve the inherent multilingual representational alignment from the MT encoder, allowing lower-resource languages to tap into the rich knowledge embedded in English-centric LLMs. Merging the MT encoder and LLM in a single model, we mitigate the propagation of translation errors and inference overhead of MT decoding inherent to discrete translation-based cross-lingual transfer (e.g., translate-test). Evaluation spanning three prominent NLU tasks and 127 predominantly low-resource languages renders MT-LLMs highly effective in cross-lingual transfer. MT-LLMs substantially and consistently outperform translation-test based on the same MT model, showing that we truly unlock multilingual language understanding for LLMs.

## introduction large language models (llms) have become the swiss-army knife for natural language understanding (nlu) in english. when pretrained with language modelling on trillions of tokens, llms excel at complex nlu tasks with minimal or no labeled data @xcite @xcite . although these models are pre-dominantly trained on english texts, typically comprising more than 80% of their training data @xcite @xcite , they show strong nlu capabilities also in other high-resource languages @xcite . 1 however, llm performance degrades in cross-lingual transfer to languages that are typologically distant from english or virtually unseen at pretraining @xcite @xcite . this performance degradation restricts the effectiveness of llms primarily to english and a tiny subset of high-resource languages and underscores shortcomings in their adaptability to underrepresented low-resource languages, thereby amplifying the cross-lingual language technology gap @xcite . in contrast, publicly available machine translation models like nllb @xcite and madlad-400 @xcite are by design oriented towards and showcase ever more inclusiveness; they provide some machine translation capabilities between more than 200 and 400 languages, respectively, in any language direction. unlike llms, machine translation (mt) models, and specifically mt encoders, are designed to semantically align textual representations in a unified embedding space, as demonstrated by their sentence retrieval performance on the flores200 dataset (cf. figure 1 ). however, the mt models lack various types of knowledge (e.g., world knowledge, commonsense knowledge), commonly acquired through large-scale language modeling pre-training. consequently, in multilingual nlu tasks and respective cross-lingual transfer they underperform even smaller multilingual encoders like xlm-r (conneau et al., 2020) (see appendix a.2 for an empirical comparison). because of this, mt models are typically used fully downstream, to translate training and/or test data from the source to the target languages, extending the wide availability of task-annotated english corpora to the target languages @xcite @xcite . translating training data (ttrain) involves substantial computational resources but yields strong xlt performance. ttrain nevertheless requires llms to support the target languages, which does not hold true for low-resource languages @xcite . translating test data, on the other hand, enables zs-xlt with monolingual lms, but it incurs an additional inference overhead from mt and generally offers performance that is slightly inferior to ttrain. both ttrain and ttest aim to align the input to accommodate the shortcomings of the llm representation space, resorting for this to discrete natural language translations coming from the mt decoder. these methods fail to preserve the rich latent representations from the mt encoder and propagate translation errors to llms, thereby reducing downstream performance @xcite . in this work, we thus propose to merge mt encoders directly with llms, creating a unified multilingual llm for enhanced cross-lingual nlu, termed mt-llm. the merger of the two models unlocks the potential to combine 1) the general knowledge available in the original llm for english and a handful of high-resource languages and 2) powerful multilingual representations and their cross-lingual semantic alignment available in the mt encoder (see figure 1 ). the key idea involves enabling the llm to directly integrate the output representations from mt encoders, this way extending its nlu performance to virtually all languages supported by the mt encoder. we align mt encoders with llms via self-distillation in two steps. the objective in the first, self-supervised adaptation step is sequence-level alignment between the original llm and the mtequipped llm (mt-llm). second, we then address the distributional shifts inherent to adaptation from general-purpose data to downstream task data through task-specific self-distillation. we fine-tune the llm on labeled task data, then transfer this task knowledge to the mt-llm by aligning the task-specific output representations. contributions. 1) to the best of our knowledge, we are the first to successfully integrate mt encoders into language model backbones for xlt, thereby enabling zs-xlt to all languages supported by the mt encoder. this integration yields two key benefits: zs-xlt performance consistently improves over ttest, while simultaneously reducing inference cost by eliminating the need to translate test instances. in turn, we show that the integration is highly efficient and only requires a few self-supervised adaptation steps to yield performance improvements over the llm backbone. ## related work translation-based xlt is a strong xlt baseline @xcite @xcite . previous studies have explored various techniques for leveraging translated training data in xlt (ttrain): these include training on translated data in a single target language @xcite , using concatenated data from all target languages @xcite , sequential training starting with the source language followed by the translated target language (aggar-wal et al., 2022), and jointly training on both combined @xcite . recent studies have also benchmarked translating test data (ttest) @xcite , which enables zs-xlt without the need for extensive finetuning for each target language, as in the case of ttrain. moreover, both paradigms can be combined by training on round-trip translated noisy source data (translating source-language data to the target language and back) and evaluating on target language test data translated to the source @xcite @xcite . translating training or test data is essentially a discrete approach for adjusting the input (i.e., its language) to the llm (i.e., language that the llm is proficient in). in contrast, we propose to align latent representation of input, produced by the mt encoders, to the representation space of the llm backbone via self-distillation, effectively bypassing translation errors that arise from the discrete translation, output of the mt decoder. by retaining continuous mt encoder representations and avoiding their discretization in the mt decoder, our approach also reduces the time and cost of inference vis-a-vis ttest. this also means that the mt-llms (unlike english-centric llms) can also reap further gains from ttrain, particularly for low-resource languages unseen during pretraining. while few studies investigated the integration of rich mt representations into lms, these efforts have generally focused on task-specific integration, without achieving a global representation alignment between the mt encoder and the (large) language model @xcite . our approach addresses this limitation by achieving task-agnostic representation alignment between mt and lm before task specialization. cross-lingual transfer with llms. widely used llms are predominantly trained on english data with english text accounting for 80-90% of their pretraining corpora @xcite . despite this imbalance, llms demonstrate a surprisingly strong performance in (high-resource) languages, which account for only a small fraction of their pretraining corpora @xcite . the pretraining focus on english limits the nlu capabilities of llms in many low(er)-resource languages, and languages linguistically distant from english @xcite . various methods adapt llms to languages not covered during pretraining, including continued pretraining @xcite , self-instruction @xcite , and vocabulary extension @xcite . these methods yield gains in model's target language generation capabilities; however, recent work shows that better generation does not translate to stronger nlu performance @xcite . more concretely, we hypothesize that by finetuning additional modular parameters on top of the base llm, we can align the output representations of the merged mt-llm with the original output representations of the base llm. in other words, we learn the mt-llm alignment via distillation with the llm itself as the teacher. as a result, the mt encoder representations, which are of high-quality for many languages, act as input for the merged mt-llm. the mt-llm merge happens in two stages: 1) self-supervised (general, task-agnostic) adaptation and 2) task-specific distillation; we describe both in what follows. stage 1: self-supervised general adaptation. figure 2 illustrates the approach in the first stage. we 'vertically' fuse a multilingual mt encoder e and an llm m into a multilingual mt-llm e × m . to this end, we introduce two sets of new, trainable parameters θ: we first initialize a projection @xmath0. we then insert low-rank adapters (loras) @xcite ## results and discussion ## conclusion llms quickly emerged as the catch-all solution to nlu in english. however, llms still cannot extend their nlu abilities to languages typologically distant from english or virtually unseen at pretraining. in this work, we propose a novel approach to fuse mt encoders with llm backbones via self-distillation to compile a massively multilingual mt-llm. the mt-llm not only strongly improves zs-xlt performance over ttest but also removes the overhead of mt decoding at inference. we demonstrate that our distillation procedure is highly efficient and requires only a few thousand steps to convert llms into multilingual mt-llms, enabling nlu in all languages supported by the mt encoder. we further show that our mt-llm alignment benefits both encoder and decoder llms. in future work we will seek to (1) further improve generalization of mt-llm by incorporating token-level alignment objectives and (2) extend the mt-llm to support further languages by posthoc adaptation of the mt encoder.