The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics (2024)

Abstract: Deploying large language models (LLMs) encounters challenges due to intensive computational and memory requirements. Our research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. While such modifications have been proven effective in tasks like machine translation, tailoring them to LLMs demands specific modifications given the diverse nature of LLM applications. We apply two language heuristics to trim the full vocabulary—Unicode-based script filtering and corpus-based selection—to different LLM families and sizes. The methods are straightforward, interpretable, and easy to implement. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed. Yet, we reveal the limitations of these methods in that they do not perform consistently well for each language with diminishing returns in larger models.

## introduction large language models (llms) are gaining increasing attention given their strong performance @xcite @xcite . llms, especially multilingual ones, hold vocabulary items for many languages and scripts, which entail a costly matrix multiplication h × |v | in the output layer, where h is the hidden size and |v | is the size of a vocabulary v . this expensive operation leads to increased costs of both memory and time given the autoregressive nature of llm decoding. given their substantial size, this latency in inference significantly escalates the expense of llm deployment. in practice, creating a sub-vocabulary v ′ with |v ′ | ≪ |v | and only loading its corresponding embedding entries for inference seems favourable since most logits from the output layer do not affect the hypothesis token(s) at each time step. vocabulary trimming (vt) has been actively explored in machine translation (often called shortlisting, @xcite @xcite -it computes token-level alignments and makes potential target tokens a subvocabulary. while anticipating certain limitations such as domain mismatch @xcite , vocabulary shortlisting in llms poses a fundamental challenge: often llm outputs are variable and open-ended, complicating the determination of the required lexicons. recent attempts at multilingual pre-trained models select tokens in a task's language @xcite . nonetheless, research in this direction is still limited, especially in speed considerations. we follow the idea of fitting vocabulary to the language of the downstream task. specifically, we examine two strategies: unicode-based filtering where vocabulary items are removed if they do not belong to the task language, and corpus-based selection where we record vocabulary hits from a large representative corpus. after experimenting with llms from two families of different sizes, we identify a good upper bound of memory reduction with several limitations and outlooks: 1) unicode-based script filtering maintains quality for latin-based languages but harms languages requiring code-mixing. 2) corpus-based selection leads to fewer alterations but is less effective in reducing the embedding size. 3) embeddings are proportionally smaller in larger models (with smaller vocabularies). yet we argue that vt can be applied orthogonally to other efficiency methods like efficient attention, quantization, etc. ## language-based vocabulary trimming we explore two ways to prepare sub-vocabulary for llms, focusing on only retaining tokens relevant to the language being generated. we test a batched setting on the fly: we determine a sub-vocabulary for an entire batch because creating the sub-vocabulary separately for each input is too expensive in practice. furthermore, we always include all tokens appearing in the inputs. script-based filtering this is done by filtering token strings that fall out of a language's unicode range-keeping tokens in the writing script of that language. it should be especially effective for languages operating on unique scripts, such as armenian, chinese, korean, etc since it allows for concise vocabulary restriction. this method might be less practical if a writing system is shared among multiple languages (e.g. cyrillic or latin alphabets), because it would be infeasible to limit the lexicons to those solely belong to a specific language, resulting in a relatively large sub-vocabulary. moreover, this method would strictly rule out codemixed texts, emojis, etc, which are used in realworld communications. corpus-based selection another way is to tokenize a representative corpus in the desired language in advance and use the vocabulary entries that have been recorded to build a sub-vocabulary. this method is non-exhaustive because we could miss rare but valid tokens or suffer from domain mismatch between the vocabulary selection corpus and the downstream tasks at inference time. ## experimental setup languages and test sets we experimented on four languages: bulgarian, chinese, english, and spanish, to offer distinct conditions that cover different degrees of writing script overlap, codemixing, etc, with details in appendix a. we sample 50 prompt questions from openassistant @xcite which are then human-translated into test languages. we decode them with beam size 1. metrics we consider efficiency-quality tradeoffs. in terms of speed, we report end-to-end time to decode the entire test, including model loading and embedding slicing. as a quality indicator, we count the chances a model fails to produce the exact same output (miss) with a full vocabulary and with vt. in addition, we report the bleu and chrf of the vt output w.r.t. to the original output with the full vocabulary (not the reference). note that there is no gold reference due to the open domain nature; we hence prefix the two string metrics with an "o-". ## cpu results and discussions upper bound performance first of all, we conduct an oracle vocabulary selection experiment to find the theoretical upper bound for speed and memory improvements: we run inference using the full vocabulary and we add the used vocabulary items to a oracle sub-vocabulary. ## gpu results in addition to cpu tests, we performed the same bloom experiments on a gpu and observed that all three selection criteria including the oracle do not lead to improved inference speed. small performance differences might amount to little more than noise when the overhead of model slicing is considered. we hypothesize that gpus are designed for multiplying large matrices, so reducing the matrix size, even to the extremity of an oracle subvocabulary, is not able to offer any speedup. this is consistent with @xcite 's findings in applying shortlists to neural machine translation on gpus. we list gpu results for bloom in appendix b ## conclusion we presented a study of two straightforward language-inspired vocabulary trimming methods to speed up inference and save memory for large language model deployment. experiments reveal ups and downs. while we can achieve speed improvements, it does not guarantee that the output is not altered compared to full vocabulary generation. with the models tested, we see the feasibility of our proposed approaches for english and spanish, but there are shortcomings when considering languages written in non-latin script and requiring code-mixing. in terms of efficiency, the reduction in inference time is less pronounced compared with memory saving.