m O thello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models? (2024)

Abstract: Many pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. However, it remains unclear what factors contribute to the learning of a language-neutral representation, and whether the learned language-neutral representation suffices to facilitate cross-lingual transfer. We propose a synthetic task, Multilingual Othello (mOthello), as a testbed to delve into these two questions. We find that: (1) models trained with naive multilingual pretraining fail to learn a language-neutral representation across all input languages; (2) the introduction of “anchor tokens” (i.e., lexical items that are identical across languages) helps cross-lingual representation alignment; and (3) the learning of a language-neutral representation alone is not sufficient to facilitate cross-lingual transfer. Based on our findings, we propose a novel approach – multilingual pretraining with unified output space – that both induces the learning of language-neutral representation and facilitates cross-lingual transfer.

## introduction one of the primary desired properties of multilingual models is their cross-lingual transfer abilitythe ability to enhance task performance in a target language when being finetuned exclusively with labeled data from the same task, but in a different source language. many pretrained multilingual models, such as mbert @xcite and xlm-r @xcite , are found to exhibit this ability across a wide range of tasks, such as natural language inference, named entity recognition, and part-of-speech tagging @xcite @xcite . crosslingual transfer also serves as a central justification figure 1 : illustration of three multilingual training approaches. blue and green blocks represent contexts in 2 different languages, and tokens from the same language have the same color. a multilingual model m consumes a,b,c,d and predicts the corresponding output e. top: a model is trained on multilingual corpora, with an objective to predict the next tokens specific to each language. middle: a model is trained on multilingual corpora, where there are tokens shared across language pairs. these tokens are named as anchor tokens. the objective is still to predict the next tokens specific to each language. bottom: a model is trained on multilingual corpora, with an objective to predict the next tokens in a unified output space. for why we would prefer multilingual models over a collection of monolingual models, since the multilingual setting might introduce competition among languages on the model capacity, which has been referred to as "the curse of multilinguality" @xcite . multilingual pretraining @xcite @xcite . however, there is no consensus on what factors lead to such language-neutral representations @xcite @xcite . additionally, to our knowledge, it remains unclear whether a shared, language-neutral space is by itself sufficient to facilitate cross-lingual transfer. in this work, we use a controlled language learning environment to investigate the essential factors for learning language-neutral representations and whether they are sufficient to facilitate the crosslingual transfer ability of multilingual models. to approach these questions, we introduce multilingual othello (mothello), a sequence modeling task based on the othello board game @xcite . in mothello, a model is given a sequence of game moves in a specific "language" l k , and the task is to predict the next legal move in the same "language" l k . this environment is appropriate for our purposes, since it separates the ground truth "world" (i.e., the game state) which is assumed to be singular, from the language used to describe it, which can take any number of forms (languages). with mothello, we first train gpt2-based (radford et al., 2019) models (mothellogpts) and analyze under what conditions language-neutral representations are learned. to quantitatively measure the alignment of representations across languages, we propose cross-lingual alignment probing, which is to recover board states in language l 2 using a probe trained on language l 1 . we observe that mothellogpts trained with the naive multilingual pretraining (figure 1(a) ) do not learn languageneutral hidden space across all languages. following, we show that anchor tokens (i.e., shared tokens across languages, figure 1(b) ) facilitates mothellogpts to learn aligned representations across all languages connected via the anchor tokens. however, we observe that these models do not show cross-lingual transfer. this contradicts with the common hypothesis that crosslingual representation alignment suffices for crosslingual transfer ability of multilingual models. lastly, we further investigate what encourages the emergence of cross-lingual transfer ability. we propose the use of a unified language-neutral output space during multilingual pretraining (figure 1(c )), which brings both aligned representations across languages and cross-lingual transfer. to summarize, our main contributions are: ## discussions in section 5.4, we introduced the unified output space approach to induce both representation alignment and cross-lingual transfer in mothellogpts. however, it is important to note that modeling mothello is considerably simpler than modeling natural languages. while it is simple to identify a next token in the unified output space in mothello, it is comparatively challenging to identify a language-neutral next token for each languagespecific context in natural languages. nevertheless, our results could inform future strategies in designing training objectives for multilingual models. traditional training of such models primarily focuses on the causal task of predicting the next language-specific token. our results suggest that incorporating a language-neutral next-token prediction task into the training process could potentially enhance the cross-lingual transfer abilities of multilingual models. some such approaches have been explored in @xcite . ## conclusion in this paper, we propose the multilingual othello (mothello) sequence modeling task as a testbed to investigate the factors which help align representations across languages, and to study the relationship between representation alignment and cross-lingual transfer in multilingual models. we introduce a new metric, the cross-lingual alignment probe accuracy, on measuring the alignment of representations across languages. we train mothel-logpts on the mothello task, and conducted analyses on the representation alignment. we found that models trained with a naive approach fail to learn a language-neutral hidden space across all input languages, but the introduction of anchor tokens helps the alignment of representations. then, we conduct finetuning experiments on mothellog-pts pretrained on a prefix-limited training corpus. to our surprise, we found that the learning of a language-neutral space alone is not sufficient to facilitate cross-lingual transfer. upon further investigation, we propose an alternative training approach -the unified output space approach -that both induces the learning of the language-neutral space and facilitates cross-lingual transfer.