Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024)

Abstract: Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup of inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

## introduction large language models (llms) such as gemini @xcite , gpt @xcite , and llama @xcite have remarkable success across various natural language processing tasks. their deployment in commercial settings has expanded to include applications such as coding assistance, writing support, conversational interfaces, and tools for search @xcite . despite their potential, the practical deployment of these models is often limited by prohibitively high inference time, particularly in multilingual contexts @xcite . for example, characterlevel and byte-level models exhibit encoding length discrepancies exceeding fourfold for certain language pairs, resulting in significant disparities in cost and inference time available to different language communities @xcite . these challenges present substantial hurdles to scalable and cost-efficient applications of llms. ## discussion drafting in speculative decoding has been treated akin to n-gram prediction @xcite , often relying on straightforward pretraining using datasets designed to replicate target llm behaviors, such as the sharegpt dataset @xcite . this approach posits that generating a limited sequence of future tokens suffices for speculative inference. contrary to this belief, our empirical result presents a different narrative. figure 5 illustrates that even in seemingly straightforward translation tasks, such as from german to english, outcomes are not as effective. this suggests that drafting requires a broader array of language modeling capabilities to manage complex linguistic structures and context variations effectively. drafters, therefore, benefit significantly from a robust pretrain-and-finetune approach, where they are first exposed to a wide array of linguistic contexts and then finely tuned to specific tasks. this training regimen transforms them into compact, yet comprehensive, language models capable of handling diverse and challenging speculative decoding scenarios with better alignment. ## conclusion this paper has demonstrated that the pretrain-andfinetune strategy for training drafters significantly enhances speedup ratio relative to standard autoregressive decoding in multilingual translation tasks. this gain grows logarithmically with the increase in the number of training tokens. supported by qualitative analysis, out-of-domain analysis, and gpt-4o evaluation, this strategy substantially outperforms the state-of-the-art methods in various language pairs. our study uncovers approaches to maximize the benefits from drafter models, thereby setting a new benchmark in this area.