Soft Language Clustering for Multilingual Model Pre-training (2023)

Abstract: Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typologyfrom the source language or when pre-training data is limited in size. In this paper, we propose XLM-P, a method that contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our space-efficient and model-agnostic XLM-P approach enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods. On the tasks of XTREME, which include text classification, sequence labeling, question answering, and sentence retrieval, both base- and large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer.

## introduction multilingual pre-trained language models (mplms) such as mbert @xcite , mbart @xcite , xlm-r @xcite and mt5 @xcite have lately produced notable advancements in a number of downstream nlp tasks. in particular, the use of mplms significantly enhances few-shot fine-tuning and makes possible efficient zero-shot cross-lingual transfer @xcite . essentially, an ideal mplm should satisfy two properties: alignment between language pairs, which has been widely studied in the literature @xcite @xcite ; and a good trade-off between high-resource and low-resource languages, which remains largely unexplored despite the success of mplms. in this paper, we focus on the second property, specially the potential for model performance to suffer when a large number of languages are added. this can occur due to restricted model capacity or computational limitations, resulting in underrepresented languages being allocated less capacity @xcite . furthermore, the model's coverage of world's languages remains inadequate, limiting the range of language technology applications it can support @xcite . a typical solution for the coverage-performance trade-off in multilingual learning is to assign additional model parameters to specific languages, such as language identity embeddings @xcite , adaptors @xcite @xcite , and language-aware layers @xcite . however, it is impractical for multilingual pre-training to maintain a separate component for each language, which can lead to more complicated and challenging optimization, especially for low-resource languages. we propose to approach the above languageaware components from a different perspective. in linguistic typology, some patterns such as nominative-accusative alignment have broad global distributions, whereas others like morphology are more specific and detailed @xcite . to take advantage of this, we introduce xlm-p, which uses a set of compact embeddings to represent soft clustering of the language patterns beyond language identity. we refer these embeddings as prompts, due to their similarity to prompt tuning @xcite . concretely, we build a key-value prompt pool and use the attention mechanism to look up the prompts for each input. the retrieved prompts are then prepended to the input embeddings, and serve as categorization information to adapt the model weights conditionally. this allows for more efficient and effective mul-tilingual learning by leveraging the patterns and similarities across languages rather than maintaining separate components for each language. we evaluate the proposed xlm-p on cross-lingual natural language understanding tasks and cross-lingual sentence retrieval tasks of the xtreme benchmark, and the consistent improvement in performance demonstrates its effectiveness. in addition, we conduct empirical analyses to investigate the underlying reasons of the improvement of xlm-p. the advantages of xlm-p can be summed up as follows: overall, xlm-p is a versatile and efficient approach for improving multilingual pre-training. ## analysis we carry out a number of analyses in order to comprehend the design of our proposed xlm-p better. ## conclusion this paper presents xlm-p, a new multilingual pre-trained language model equipped with contextually retrieved prompts. in particular, we prepend prompts-like learnable vectors to the input for modeling language interdependence and other potential sharing information. compared with other language-aware components, the retrieved prompts are parameter-efficient and more flexible without the requirement of language detection. experiments and analyses validate the effectiveness and robustness of xlm-p. in addition, our method is compatible with various existing multilingual pretraining objectives.