Accelerating Multilingual Language Model for Excessively Tokenized Languages (2024)

Abstract: Recent advancements in large language models (LLMs) have remarkably enhanced performances on a variety of tasks in multiple languages. However, tokenizers in LLMs trained primarily on English-centric corpora often overly fragment a text into character or Unicode-level tokens in non-Roman alphabetic languages, leading to inefficient text generation.We introduce a simple yet effective framework to accelerate text generation in such languages. Our approach involves employing a new language model head with a vocabulary set tailored to a specific target language for a pre-trained LLM. This is followed by fine-tuning the new head while incorporating a verification step to ensure the model’s performance is preserved.We show that this targeted fine-tuning, while freezing other model parameters, effectively reduces token fragmentation for the target language. Our extensive experiments demonstrate that the proposed framework increases the generation speed by a factor of 1.7 while maintaining the performance of pre-trained multilingual models on target monolingual tasks.

## introduction modern large language models (llms) @xcite @xcite have exhibited remarkable capabilities for a variety of tasks in multiple languages @xcite . although these models are predominantly trained on english-centric data, they have shown a significant degree of multilingual proficiency @xcite . however, when applied to non-alphabetic languages, these models often suffer from slower text generation due to english-centric tokenization @xcite @xcite . current tokenization techniques used in large language models (llms) are data-driven and optimize segmentation based on the frequency of characters or bytes within a specific corpus @xcite . as a result, the tokenizers of multilingual models, which are heavily influenced by english-dominant training data, are predominantly composed of english subwords. this leads to excessive fragmentation, where non-english words are overly segmented into a large number of subword units @xcite @xcite . the autoregressive nature of llms further amplifies this inefficiency, as it sequentially requires the generation of text. to address these challenges, previous studies @xcite @xcite have proposed replacing or augmenting the existing vocabulary of pre-trained multilingual models with language-specific vocabularies to more effectively encode monolingual text corpora. specifically, @xcite improved mbert @xcite by replacing its tokenizer with a monolingual one and incorporating an additional 100,000 pre-training steps. on the other hand, @xcite enhanced llama (touvron et al., 2023a) by expanding the chinese vocabulary and further pre-training it on a 120gb text corpus that includes chinese texts. however, this approach requires an extensive pre-training phase with a substantial amount of data. another approach to address the challenges is the use of small draft models @xcite . these models generate draft output tokens, which are then verified by the original language model. however, a significant challenge arises when trying to identify or train a suitable small model that can handle multiple languages with reliable performance @xcite . in response to these challenges, our research introduces mumo, accelerating multilingual language models for a targeted monolingual text generation, particularly in non-alphabetic languages. mumo incorporates a new vocabulary of a target language into the output layer, also known as the language model (lm) head, and predicts the next token from this expanded vocabulary. this approach requires training only the extended portion of the output layer and specific layers of the feed-forward network. importantly, mumo eliminates the need for extensive text corpora or a draft model, requiring only a modest corpus of the target language, approximately 44m tokens in size. empirical results across summarization, and translation tasks in korean and japanese demonstrate that mumo significantly accelerates text generation, achieving over a 1.7-fold increase in speed without significantly compromising output quality. ## proposed framework we propose a framework named mumo to accelerate the inference speed of a pre-trained multilingual lm for a non-alphabetic monolingual language via a given small monolingual dataset. in the section, we introduce 1) the model architecture, 2) the finetuning process on a small targeted language dataset, and 3) the inference process of the proposed framework. ## conclusion our study has successfully tackled the challenges in generating text for non-alphabet languages, particularly those associated with excessive fragmentation issues. the approach not only speeds up text generation but also paves the way for more efficient multilingual language applications. our future work will broaden our experimental scope to languages that were not sufficiently represented in the pre-trained multilingual language model.