T - FREE : Subword Tokenizer-Free Generative LLM s via Sparse Representations for Memory-Efficient Embeddings (2024)

Abstract: Tokenizers are crucial for encoding information in Large Language Models, but their development has recently stagnated, and they contain inherent weaknesses. Major limitations include computational overhead, ineffective vocabulary use, and unnecessarily large embedding and head layers. Additionally, their performance is biased towards a reference corpus, leading to reduced effectiveness for underrepresented languages.To remedy these issues, we propose T-Free, which directly embeds words through sparse activation patterns over character triplets and does not require a reference corpus. T-Free inherently exploits morphological similarities and allows for strong compression of embedding layers. In our exhaustive experimental evaluation, we achieve competitive downstream performance with a parameter reduction of more than 85% on these layers. Further, T-Free shows significant improvements in cross-lingual transfer learning.

## classic tokenization principles before we derive t-free in detail, let us first establish some basics of how llms traditionally encode and decode text. most llm operations are performed in floating-point numbers through a series of matrix multiplications and non-linear activation functions. consequently, we require techniques that map discrete textual inputs into floating-point representations and inversely transform the predictions of the model back to text. traditionally, the first step in this process is to split any textual input into small chunks referred to as tokens. generally, these tokens can take arbitrary formats, spanning numerous characters, a single or even multiple words, and may also contain special characters. the latter can be particularly useful to encode programming languages. a tokenizer com-prises the steps and rules that are necessary to dissect a text into a sequence of tokens. importantly, the total number of tokens is restricted, and we refer to the set of all unique tokens as the vocabulary. each token in the vocabulary is assigned an integer token-id, wherefore tokenizers produce a sequence of token-ids for any textual input. next, a large matrix of dimensions vocab size × hidden size, an llm's embedding layer, maps each tokenid to an internal representation as a floating point vector (cf. fig. 1a ). to produce new text, generative models are trained auto-regressively. that is, they iteratively predict the next token, which is appended to the input text. therefore, the training objective is formulated as a classification problem: a one-label prediction of the next token over the entire vocabulary. consequently, the last layer of the model-the lm head-is a projection into the size of the vocabulary and thus also of dimension vocab size × hidden size. for decoding, we can, for example, always select the token with the highest assigned value, which is called greedy sampling. the output text is produced by looking up the corresponding text snippet of each predicted token-id in the vocabulary. generally, it is desirable to encode any text in as few tokens as possible to reduce computational cost. at the same time, different semantic concepts should be separated into distinct tokens to ensure good language comprehension. the combination of both objectives is usually best satisfied by encoding each word as one token. ## t-free a key motivation for t-free is the intuition that minor differences in spelling, like leading whitespaces or capitalization, do not hold enough entropy to justify entirely independent tokens. t-free directly encodes morphological similarities by representing each word as a multi-label encoding of its character triplets. this designed overlap between words allows us to significantly reduce the size of embedding layers. we now derive t-free's approach to text encoding and decoding and discuss implications on llms in general. we provide a visualization of each step in fig. 1b and pseudo-code in app. a. ## empirical evaluations we continue with an empirical demonstration of the performance of t-free, and how it remedies the flaws of standard tokenizers as outlined in sec. 2.2. to thoroughly analyze the performance differences, we designed three consecutive experiments: first, we perform hyperparameter ablations on a series of 1b parameter models, which achieve competitive scores on standard benchmarks with a reduced vocabulary, which in turn addresses f1. second, we analyze the duplicates in the tokenizers of recent figure 2 : example of the next word prediction with t-free. to the list of predictable words of dimension d we generate once the corresponding patterns within the available vocabulary size v, as described in the encoding step 2 of sec. 3.1. note how morphologically close words will generate overlapping patterns. the element-wise sigmoid values of the output of the last hidden layer, σ(h), is multiplied with this pattern matrix using standard dot product. finally, we use h ′ for the sampling process, the average sigmoid value of a word. c.f. app. a for further details. llms with respect to f2. notably, t-free is by design free of duplicates. lastly, we look at f3 and evaluate the performance of various tokenizers across languages. further, we trained 3b parameter models on english and continued training on german data to practically investigate language adaptability. t-free has better tokenization performance across languages and outperforms classic tokenizers on language transfer. ## discussion prior research has demonstrated that the mapping into a sparse hidden representation and the training of a dense aggregation layer as applied in t-free, is a universal function approximator @xcite . these results provide further theoretical motivation for our approach. t-free allows for significant compression of an llms' vocabulary by more than 85% without performance degradation. notably, the affected embedding and head layers are by far the largest in llms in terms of parameter count. they are also the most influential to an llm, as they dictate the mapping between text and numerical representations. for one, these massive improvements allow for better utilization of billions of parameters in large models. the compression of t-free in particular paves the way to building better lowresource models, by reducing model size and training cost and improving adaptability. for example, in our experiments without pipe or modelparallelism, we were able to triple the micro-batch size, yielding faster training iterations. furthermore, we observed more stable loss curves for t-free, in particular for higher learning rates. these improvements may be attributed to the explicit modeling of similar words, the removal of duplicates, and the less volatile multi-label training target. further, the uniform hashing distributes gradients evenly amongst the available vocab size, in contrast to classical approaches. we provide further details in app. d,g. the rules we use for obtaining word representations are universal and well-defined at pre-training time. they do not change over time, particularly neither when adding languages later on. t-free also lowers computational costs due to its low fertility and easy-to-process whitespace splitting. consequently, pre-processing, training and inference of an llm all require less compute. lastly, t-free allows to explicitly model and steer the decoding process at inference time, by altering the available dictionary. consequently, hallucinations will likely be reduced due to fewer "generic fall-back" word splits. moreover, one can dynamically add or remove words. it is worth pointing out that t-free's compression benefits can also be combined with traditional tokenizers. in-stead of the simple whitespace splitting one could keep traditional tokenization and trigramify "classic tokens". ## related work few alternatives to bpe and unigram have been found in recent llms and research. @xcite propose a gradient-based trainable tokenization module in contrast the otherwise statistical based approach. the naive approach of splitting the input text into bytes or characters maximizes fertility and thus increases computational requirements. @xcite employ a mix of multiple models to improve this drawback of byte-wise processing. i.p. they introduce a fixed character-embedding aggregation and a second character-decoder model. however, they use a fixed byte-width that is processed at once, which is not aligned with word splits. consequently, prior research has proposed methods for merging bytes, e.g., through state-space models @xcite . however, these approaches still result in performance degradation. finally, linguistically motivated approaches have built tokenizers based on known morphological rules @xcite . however, these methods are usually tailored to specific applications and are usually too costly and error-prone for large, generalpurpose models. @xcite in particular discusses how adding subword informations, such as trigrams, enriches the encoding of words and leads to reliable compressions. @xcite conduct research on the overloading of different hashfunctions to further improve and compress embedding representations. xue and aletras (2022) train bert-style encoder models based on a different set of hashes on words. @xcite propose a multistage encoding scheme that uses hash functions and convolutions to enhance the bert-encodings of words. another line of work to reduce the vocabulary parameter count is the utilization of weight tying, effectively halving it, as the embedding and head layers become "tied" to the same matrix (press and wolf, 2017). however, the effects on performance are still not sufficiently explored, and it arguably imposes a more difficult training objective. ## conclusion in this work we present t-free, an alternative to subword tokenizers with a simple and explicitly modeled robust hash function on words. it removes the need and pitfalls to limit "a models potential" to a "pre-pre-trained" vocabulary. we, moreover, fundamentally shift the established target of training language models, previously designed as a single-label problem, into a multi-label prediction based on word similarities. similarities in particular include leading whitespaces and uppercase variations, for which subword tokenizers add specific tokens that are independently trained from scratch. these contributions allow us to train language models more robust, more adaptable when continuing pre-training with a new language, and with a significantly (to 12.5%) reduced parameter size without a decrease in benchmark scores. due to the special role of the matrices, the latter in particular allows one to increase micro-batchsize, which further accelerates training time. finally, the consequent convolution-like encoding achieves sota fertility scores across most languages and enables by design synergies to similar language groups. we demonstrated the latter showing that our 3b almost achieved "native-language" performance after a small amount of language-transfer training steps, in contrast to the tokenizer baseline.