Tokenization with Factorized Subword Encoding (2023)

Abstract: In recent years, language models have become increasingly larger and more complex. However, the input representations for these models continue to rely on simple and greedy subword tokenization methods. In this paper, we propose a novel tokenization method that factorizes subwords onto discrete triplets using a VQ-VAE model. The effectiveness of the proposed tokenization method, referred to as the Factorizer, is evaluated on language modeling and morpho-syntactic tasks for 7 diverse languages. Results indicate that this method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm.

## introduction a typical subword tokenizer consists of a vocabulary of 10 000s of subwords, each of them mapped onto a single and independent index. instead, we propose a method that learns to project subwords onto triplets of 256 indices: @xmath0. this mapping is learned by a vector-quantized variational auto-encoder @xcite from a large word-frequency list, resulting in a projection where orthographically different words use different indices and similar words share similar indices: @xmath1, @xmath2. modeling this projection with a vq-vae also automatically gives the estimated probability of every subword given an index triplet. maximizing the joint probability allows for an optimal subword tokenization of words not contained in the vocabulary. in this paper, we present factorizer, a subword encoding method that serves as a drop-in replacement for any subword tokenizer used in modern nlp pipelines. we release the source code, trained models and ready-to-use tokenizers online. 1 our approach demonstrates the following advantages: in this paper, we propose a novel tokenizer that utilizes a vector-quantized variational auto-encoder (vq-vae; @xcite as its central component. vq-vae is a powerful technique for learning discrete latent variables that can encode words and reconstruct them back to their original form. this process can be broken down into three main steps, as illustrated in figure 1 : gradient backpropagation. the model is optimized jointly with the backpropagation algorithm. however, special attention must be given to the codebook quantization q as this operation is not differentiable: during the forward phase, the quantized codebook vectors are simply passed to the decoder. however, during the backward phase, the gradient of the loss function ∇ d l is passed directly from the decoder to the encoder. this technique, known as straightthrough gradient estimation, is possible because the output of the encoder and the input of the decoder share the same latent space. the output e(x) is sufficiently similar to the decoder input z, such that the gradient carries information about how to change e(x) to lower the reconstruction loss. loss terms. in addition to the standard reconstruction loss l @xmath3, which measures the auto-encoding performance, the vq-vae model incorporates two auxiliary loss terms that align the encoder outputs e(x) with the nearest codebook vectors. specifically, the codebook loss ## factorizer we utilize the vq-vae architecture to train a model capable of mapping discrete triplet representations z to subword strings w (and vice versa). furthermore, we employ the vq-vae decoder to estimate the probabilities of the subword strings p(w|z). after the model is trained, we infer its vocabulary, consisting of a set of tuples ⟨w i , z i , log p(w i |z i )⟩. finally, we use this vocabulary to perform optimal (in regards to the logprobabilities) subword tokenization. ## experiments the effectiveness of factorizer is experimentally verified on a typologically diverse set of languages. we train masked language models on each language and then evaluate it on morpho-syntactic tasks from the universal dependencies (ud) treebanks @xcite . additionally, we demonstrate that the factorizer-based language models exhibit greater robustness to noise and superior performance in low-resource settings. we present four sets of experiments in this section: languages. in total, our method is evaluated on 7 different languages to investigate its performance on different morphological typologies. for the most part, we follow @xcite in their selection of languages according to the traditional categories of morphological systems. the text corpus for each language is drawn from the respective part of the mc4 corpora @xcite , unless specified otherwise. note that we use the same corpus for training factorizer models (where we extract a word-frequency list for every language) as for training language models. the languages chosen for evaluation are: in order to evaluate performance on lowresource languages, this language, which has about 57 000 fluent speakers, was chosen as it is the smallest language from mc4 with a full train-dev-test split in universal dependencies. ## related work the neural era in nlp has brought about a change in how sentences are typically tokenized into atomic units. tokenization in current nlp typically involves the segmentation of the sentence into subwords, smaller units than the traditional word tokens of the pre-neural era. the question of what these subword units should consist of has received a fair amount of attention in previous research. linguistically motivated or rule-based approaches are found @xcite , however the majority of this work is based on unsupervised morphological segmentation. while this field has a long-standing history @xcite , subwords have in recent years been based on bpe @xcite and variations such as wordpiece @xcite or sentencepiece @xcite . the recent pixel-based encoder of language (pixel; @xcite reframes language modeling as a visual recognition task, employing a transformer-based encoder-decoder trained to reconstruct the pixels in masked image patches, and disposing of the vocabulary embedding layer completely. the method shows strong results for unseen scripts, however, more modest performance for languages with latin script, such as english. as in our work, they find morpho-syntactic tasks to benefit the most and more semantic tasks to show more mixed results. ## conclusion we proposed a novel tokenization method where every subword token is factorized into triplets of indices. the main benefit of this factorization is that the subwords-units maintain some characterlevel information without increasing the length of tokenized sequences. in practice, factorizer even slightly decreases the number of subword splits (figure 3 ), while noticeably improving the performance of large language models on morphosyntactic tasks, especially on lemmatization ( we hope that this work clearly demonstrated the potential of utilizing factorized subwords for language modeling and future work will improve their performance even further.