Multi-word Tokenization for Sequence Compression (2023)

Abstract: Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

## introduction the field of natural language processing (nlp) has seen major breakthroughs with the advent of large language models (llms) @xcite @xcite . despite their successes, llms like chatgpt @xcite possess hundreds of billions of parameters that entail enormous computational cost by design. traditional model compression methods such as knowledge distillation @xcite , pruning @xcite , and quantization @xcite have focused on creating lighter models either by shrinking the architectural size or by reducing the number of flops. recently, llms have been shown to produce impressive performance on inputs that have been carefully designed to contain all the necessary information for a given instruction. as such, there is an increasing trend in designing longer and longer prompts that has led to a significant rise in computational cost. to address this, interest has grown in compressing the input sequences from the tokenizer @xcite @xcite . indeed, various works have shown the importance of tokenization in determining the length of a sequence in specialized domains @xcite or on underrepresented languages @xcite . in this paper, we propose a method for reducing the computational cost of a llm by compressing the textual inputs using multi-word tokenizers (mwts). to achieves this, we enrich the vocabulary of the tokenizer with statistically determined multi-word expressions. by encoding the frequent n-grams with single tokens, the sequences produced are both shorter and more informative, thus allowing for major speedups via early sequence truncation. additionally, mwts are shown to be compatible with the aforementioned traditional compression methods. experimentally, we assess mwts on three text classification datasets. we show how our approach still performs well when combined with distilled models @xcite and other sequence compression techniques @xcite . the code for our paper is publicly available foot_0 . the rest of the paper is organized as follows. first, we review the related works in section 2. then, we describe our approach in section 3 and present the experiments in section 4. finally, we draw our conclusions in section 5. ## related works most model compression research falls into one of the following categories: knowledge distillation @xcite @xcite @xcite , pruning @xcite , and quantization @xcite . the family of approaches is somewhat complementary and input: an energizable member is operably coupled to the outer sleeve . tgen: an, en, ##er, ##gi, ##zable, member, is, opera, ##bly, coupled, to, the, outer, sleeve, . ## multi-word tokenizer tokenization is a necessary step in the feeding of textual data to a llm. typically, tokenizers split a text into a sequence of symbols which can be entire words or only subparts. to do this, a vocabulary is first constructed by statistically learning the most frequent tokens from a large general-purpose corpus @xcite @xcite . the resulting tokenizer can then be used to segment an input text by greedily looking for the solution with the least number of tokens. building upon this, we inject into the tokenizer new symbols formed by n-grams of words. we do this by first selecting the most frequent n-grams to include in its vocabulary. then, we place an n-gram merging step within the tokenization pipeline as sketched in figure 2 . the added n-grams will be treated as single tokens further down the tokenization pipeline. n-gram selection. in order to maximize the sequence reduction, we statistically estimate the top-k most frequent n-grams in a reference training corpus. although the approach is greedy, hence sub-optimal, it still effectively yields significant compression while being extremely fast and easy to compute. more formally, given a corpus d and @xmath0, we compute all the possible n-grams g @xmath1, where @xmath2, . . . , n . then, we count their frequency f (g n ), ∀g @xmath3. the k most frequent n-grams g k are included in the vocabulary v ← v ∪ g k of the tokenizer t . energizable + + member_is en ##er ##zable member is treated pre-trained lm embeddings adapted tokenizer embeddings ##gi figure 3: fast vocabulary transfer. the pre-trained embeddings of existing tokens are combined to form the embeddings of the newly adapted vocabulary. ## experiments given a fixed number of tokens, a more compact input sequence preserves a greater amount of infor-mation. this can be used to either achieve a better performance with limited benefits in speedup, or vice versa, i.e. making the model faster with negligible drops in performance. the experiments aim to analyze how these two aspects interact with one another. we focus on text classification as it is a problem of particular interest for many industryoriented applications. ## conclusion in this work, we proposed a sequence compression approach that reduces textual inputs by exploiting the use of multi-word expressions drawn from the training set according to their top-k frequencies. we conducted an investigation on 3 different datasets by evaluating each model in conjunction with other compression methods @xcite . our approach is shown to be highly robust to shorter sequence lengths, thus yielding a more than x4 reduction in computational cost with negligible drops in performance. in the future, we expect to extend our analysis to other language models and tasks such as language generation in the scope of sequence compression.