An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference (2024)

Abstract: The development of state-of-the-art generative large language models (LLMs) disproportionately relies on English-centric tokenizers, vocabulary and pre-training data. Despite the fact that some LLMs have multilingual capabilities, recent studies have shown that their inference efficiency deteriorates when generating text in languages other than English. This results in increased inference time and costs. Cross-lingual vocabulary adaptation (CVA) methods have been proposed for adapting models to a target language aiming to improve downstream performance. However, the effectiveness of these methods on increasing inference efficiency of generative LLMs has yet to be explored. In this paper, we perform an empirical study of five CVA methods on four generative LLMs (including monolingual and multilingual models) across four typologically-diverse languages and four natural language understanding tasks. We find that CVA substantially contributes to LLM inference speedups of up to 271.5%. We also show that adapting LLMs that have been pre-trained on more balanced multilingual data results in downstream performance comparable to the original models.

## introduction generative large language models (llms) obtain strong generalization performance in many downstream natural language processing tasks (ope-nai, 2023; @xcite across various languages. for example, bloom @xcite supports 46 languages while open ai's chatgpt reportedly supports 90 languages @xcite . despite the multilingual capabilities of state-ofthe-art llms, their development disproportionately relies on english-oriented tokenizers, vocabulary and pre-training data. for example, around 30% of the training data in bloom is english. this to test our hypothesis, we perform an empirical study of five cva methods, on four generative llms, including a wide range of downstream tasks, from text classification, and span prediction, to summarization in zero-shot and few-shot settings across four diverse languages (i.e. german, japanese, arabic, and swahili). our contributions are as follows: ## analysis in-language vs. english prompting. figure 3 shows the performance difference between english and in-language prompting across models, languages and tasks for two representative cva methods (heuristics and clp+). 11 recent studies have showed that in-language prompting yields lower performance than prompt-10 tigerbot is built on top of llama2, with german being the largest among non-english languages in the pre-training corpus (0.17%) @xcite . 11 one heuristic-and one semantic similarity-based method. we also observe a drop of 0.03 or larger (nonshaded areas in figure 3 ) in only 6 out of 16 cases. the few-shot settings also exhibited similar trends with substantial performance degradation of 0.03 or more in 4 out of 12 cases, and similar or better performance in 7 out of 12 cases. some in-language prompting cases with lower performance than english, such as in german across tasks and zero-shot nli in arabic and swahili, can be related to the tokenization effects discussed in §5.1. previous studies have also found a strong correlation between tokenization and performance @xcite @xcite . ## recommendations our findings suggest that cva offers substantial inference speedups across tasks, languages and models regardless of the target embedding initialization approach. however, downstream performance is sensitive to the embedding initialization and lora fine-tuning. therefore, we provide the following recommendations to researchers and practitioners foot_9 : ## conclusion we have conducted an extensive study on the effectiveness of cva on llm inference efficiency and performance. our experiments in four diverse languages demonstrated that cva substantially contributes to llm inference speedups of up to 271.5% while maintaining comparable downstream performance to baselines when adapting multilingual llms. we supplement our results and analysis with specific recommendations for effective cva with llms. in future work, we plan to explore various inference-aware methods for cross-lingual transfer, such as cost-effective subword vocabulary selection @xcite .