Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners (2024)

Abstract: Recently, Large Language Models (LLMs) have shown impressive language capabilities, while most of them have very unbalanced performance across different languages. Multilingual alignment based on the translation parallel data is an effective method to enhance LLMs’ multilingual capabilities. In this work, we first discover and comprehensively investigate the spontaneous multilingual alignment of LLMs. Firstly, we find that LLMs instruction-tuned on the question translation data (i.e. without annotated answers) are able to encourage the alignment between English and a wide range of languages, even including those unseen during instruction-tuning. Additionally, we utilize different settings and mechanistic interpretability methods to analyze the LLM’s performance in the multilingual scenario comprehensively. Our work suggests that LLMs have enormous potential for improving multilingual alignment efficiently with great language generalization and task generalization.

## introduction large language models (llms) have recently shown impressive language capabilities across numerous downstream language tasks @xcite . however, most of the existing llms are trained on extensive high-resource languages text @xcite @xcite , which lead to a significant performance gap between high-resource languages and low-resource languages @xcite @xcite . for the same task and question contents, using different languages for inputs may have a significant impact on the model's performance. some studies have conducted comprehensive exploration about how to enhance the llms' ca-pabilities across different languages. the classical approach typically follows the translate-based paradigm @xcite . considering llms' great performance on the high-resource languages, some cross-lingual alignment and transfer methods are proposed @xcite @xcite . question alignment @xcite is an outstanding paradigm among these methods which effectively improves multilingual alignment at lower cost, i.e. only utilizes the x-english parallel question translation data. meanwhile, some studies have further explored the llms, revealing that english also participate in the intermediate latent reasoning of these models even when llms are prompted in non-english @xcite . these findings suggest that for llms, different languages are not isolated, and llms are able to leverage the connections between various languages to address problems in the multilingual scenarios. researchers also reveal the shared semantic space for different languages @xcite , which is consistent with the findings above and indicates the importance of the multilingual alignment. also, @xcite discover that multilingual instruction-tuning with three languages improves model's cross-lingual transfer abilities on some generative tasks. intuitively, llms have abilities to acclimatize themselves to the multilingual environment through appropriate training @xcite . many existing methods rely on instruction-tuning on the multilingual instruction-tuning datasets @xcite . however, given the question alignment paradigm, utilizing multilingual alignment is also helpful for improving llms' multilingual abilities. additionally, we focus on question alignment in our work to eliminate the interference of task-related data with annotated answers from our analysis of multilingual alignment. based on the findings above, can llms achieve better multilingual alignment across different languages efficiently through appropriate methods? in this work, we investigate the multilingual alignment of llms, where we only train the llms on the parallel data without annotated answers (only queries) in a few languages. following question alignment, we conduct the experiments on models in different types (english-centric or not) and parameter sizes, and test across a wide range of languages on different benchmarks. we find that question alignment following @xcite can effectively enhance the multilingual capabilities of llms, which indicates that models can effectively utilize the relevant knowledge and capabilities learned during the pretraining process with question alignment, consisting with the "superficial alignment hypothesis" @xcite . our results also indicate that conducting question alignment in a small number of languages brings much better multilingual alignment even between english and many languages unseen during instruction-tuning process, which implies good language generalization. furthermore, we also use logit lens (nostalgebraist, 2020) and dimensionality reduction techniques @xcite to study the latent states of llms, providing more comprehensive perspectives and empirical results for the alignment improvements in our experiments. ## analysis pipeline we investigate the effect of question translation parallel data on llms' performance across a wide range of languages even unseen during the finetuning process. we define the universal set of languages as u: we use question translation data for training to eliminate the impact of annotated answers themselves. and we use in-context learning for test while the model haven't been trained on the corresponding task. we test the trained model on all languages @xmath0. we construct the testing dataset q @xmath1, where q l consists of all test questions in the language l. ## experimental setup we conduct our experiments on both englishcentric and non-english-centric models. and we utilize different representative tasks and different model parameter sizes to further strengthen our conclusions. in this section, we introduce our experimental settings in detailed. ## results in this section, we report the main results across different experimental settings and conduct some discussions based on the results. ## conclusion in this paper, we find that llms only trained on question translation data without annotated answers are able to get a significant multilingual alignment improvement between english and a wide range of languages, even those unseen during instructiontuning. we conduct the experiments on different models, different benchmarks and 20 different languages to strengthen our conclusions. our results indicate that utilizing question alignment greatly enhances the multilingual alignment and the incontext learning capabilities of llms. and these improvements demonstrate the excellent model and language generalization. furthermore, we also conduct comprehensive analysis based on some mechanistic interpretability methods, including logit lens and dimensionality reduction technique. our work demonstrates the enormous potential of llms for efficient multilingual capability improvement. we hope our work can inspire the community to further explore this promising direction for the better multilingual alignment.