U ni B ridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages (2024)

Abstract: In this paper, we introduce UniBridge (Cross-Lingual Transfer Learning with Optimized Embeddings and Vocabulary), a comprehensive approach developed to improve the effectiveness of Cross-Lingual Transfer Learning, particularly in languages with limited resources. Our approach tackles two essential elements of a language model: the initialization of embeddings and the optimal vocabulary size. Specifically, we propose a novel embedding initialization method that leverages both lexical and semantic alignment for a language. In addition, we present a method for systematically searching for the optimal vocabulary size, ensuring a balance between model complexity and linguistic coverage. Our experiments across multilingual datasets show that our approach greatly improves the F1-Score in several languages. UniBridge is a robust and adaptable solution for cross-lingual systems in various languages, highlighting the significance of initializing embeddings and choosing the right vocabulary size in cross-lingual environments.

## introduction recently, multilingual pre-trained language models (lms) have significantly advanced natural language processing (nlp) tasks, narrowing the performance gap between english and various other languages. multilingual pre-trained models such as xlm-r @xcite and mbert @xcite are currently strong models for effectively cross-lingual transfer @xcite @xcite . however, these models pose a limitation that they are pretrained on a limited set of approximately 100 languages, leaving a substantial void for the vast array of the world's nearly 7000 languages @xcite . the resultant disparity disproportionately affects low-resource languages that are * equal contribution. † corresponding author. not covered in their pre-trained corpora @xcite , impeding their performance compared to their high-resource counterparts. figure 1 : some languages/scripts are not covered in the pre-trained corpora. hence, the pre-trained tokenizer will eventually produce many unknown tokens which corrupts the sentence's meaning and results in poor performance. recent efforts propose the use of adapters to mitigate the knowledge gap in low-resource languages prior to transferring knowledge for specific tasks @xcite @xcite . these methods adapt the pre-trained lms to a new language by utilizing monolingual data, enabling the model to acquire a robust representation of the target language before receiving knowledge from the source language. despite enhanced performance in languages not included in the pre-trained corpora, these approaches still exhibit poor performance in languages with unseen scripts (i.e., the scripts that are not presented in the pre-training corpora; see figure 1 ). to address the issue of unseen scripts, existing studies @xcite propose acquiring a new vocabulary embedding for newly discovered languages. however, these methods heavily rely on manually configuring the vocabulary size and initializing the embedding matrix. furthermore, recent cross-lingual transfer learning studies focus on english due to its abundant pre-trained data and impressive task performance, our experiments reveal that high performance in english tasks does not necessarily guar-antee successful transfer to other languages, particularly low-resource languages. therefore, we suggest an automated method utilizing the lms to identify the most suitable set of source languages for knowledge aggregation, leading to notable performance improvements over single-source language transfer. our research empirically tested the effectiveness of newly random initialized embeddings and fixed vocabulary size. we then introduce an efficient technique for determining the optimal vocabulary size for new languages, utilizing the syntactic and semantic insights from the pre-trained lms. in addition, we present an innovative method for transferring knowledge from multiple sources, which allows the model to choose the best combination of source languages to improve the overall performance. our results contribute to the ongoing discussion about managing linguistic diversity in nlp, particularly for languages with limited resources, emphasizing the importance of a detailed and inclusive strategy in creating multilingual pre-trained lms. we evaluate our approach on sequence tagging tasks (e.g. ner, pos) and classification (e.g. nli) with two strong baselines, mbert and xlm-r, and observe a significant increase in the f1 and accuracy score foot_0 . in summary, our contributions are: ## methodology our proposed framework includes five stages as illustrated in figure 2 . in the following section we will detail each stage of the framework: 1) vocabulary size searching, 2) language-specific embedding initialization, 3) model adaptation to new languages not covered in the pre-training data, 4) downstream task training, 5) multi-source transfer downstream task inference. ## experimental setup language set: the set of source languages are: english, chinese, russian, arabic and japanese. for the target languages, we evaluate 14 low-resource languages from wikiann @xcite whose training set consists of only 100 samples per language, 9 low-resource languages from universal dependencies (ud) @xcite whose training set consists of just few thousands samples per language and 10 languages from the americasnli @xcite . monolingual data: for the language adaptation part, we extract from the wikipedia dataset from huggingface foot_1 10k samples for simulating the low-resource settings, each sample consists of 128 words, for each target language. for source languages, the number of samples is 50k per language to simulate the rich-resource languages. for languages in americasnli, we use one side of the translation dataset from @xcite . tokenizer: we use the sentencepiece @xcite to learn the token from the monolingual data with the vocab size determined by our algorithm 1. downstream data: ner: we train unibridge on the train split of wikiann for all the source language sets and perform inference for the target language on the test split. pos: we train unibridge from the train split of ud for all the source languages sets. nli: we train unibridge from the train split of xnli @xcite for english, chinese, arabic and russia due to the missing japanese set. baseline: we evaluate unibridge against the mad-x framework and zero-shot cross-lingual fine-tuning using pre-trained language models (lms). in the zero-shot approach, we fine-tune the entire pre-trained lm on the combined training data of all source languages and then directly infer on the target languages. with mad-x, we adhere to its standard setup, training on monolingual data. to perform multi-language training, we combine training data from all source languages to train a "universal" task adapter. for inference, we swap the language adapter for each target language and integrate the "universal" task adapter. for unibridge, we implement the language adaptation and task training stages as detailed in section 2.3. during inference, we combine the task adapters from 5 source languages for multi-source transfer and report the f1 score for ner and accuracy score pos, nli on the target language's test split. the hyperparameters for training, inference as well as the computational resources are given in appendix c. ## results and analysis we present the result of our method and the baselines in and potential in leveraging various llm architectures for improved language representation. the results are presented in appendix d.8, showcasing the comparative performance and strengths of unibridge in diverse settings. although unibridge can successfully improve cross-lingual generalization, there are still some inconsistencies in the performance of a language on different tasks, e.g., amharic (amh), ligurian (lij), and sanskrit (san) on ner and pos tasks. we hypothesize that the inconsistency arises from the misalignment in the subspace between the language adapter and the task adapter. one approach to mitigate this misalignment is to regularize the representation so that the newly learned representation is shared between the source and target languages. unibridge leverages kl divergence as a regularization approach. this may not be sufficient to completely resolve the inconsistency, but given our constrained resources, kl divergence fits our requirements well. we leave other advanced methods, such as optimal transport or contrastive learning, for future work. ## conclusion in this paper, we investigate cross-lingual transfer learning, focusing on languages with constrained resources. our contribution lies in an algorithm that autonomously determines the optimal vocabulary size for a new language, informed by its monolingual corpus, and an innovative method for initializing a new embedding matrix, drawing from both semantic and lexical facets of the pre-trained language models. additionally, we introduce a novel technique for aggregating multi-source transfer learning, enhancing the efficacy of cross-lingual transfer tasks. our empirical tests demonstrate the adaptability of our method across different models, yielding significant enhancements in performance. a thorough investigation of key elements highlights unibridge's effectiveness in various situations, offering an in-depth understanding of the robustness of our approach.