Effects of sub-word segmentation on performance of transformer language models (2023)

Abstract: Language modeling is a fundamental task in natural language processing, which has been thoroughly explored with various architectures and hyperparameters. However, few studies focus on the effect of sub-word segmentation on the performance of language models (LMs). In this paper, we compare GPT and BERT models trained with the statistical segmentation algorithm BPE vs. two unsupervised algorithms for morphological segmentation — Morfessor and StateMorph. We train the models for several languages — including ones with very rich morphology — and compare their performance with different segmentation algorithms, vocabulary sizes, and model sizes. The results show that training with morphological segmentation allows the LMs to: (1) achieve lower perplexity, (2) converge more efficiently in terms of training time, and (3) achieve equivalent or better evaluation scores on downstream tasks. Lastly, we show that (4) LMs of smaller size using morphological segmentation can perform comparably to models of larger size trained with BPE — both in terms of (1) perplexity and (3) scores on downstream tasks. Points (2) and (4) impact on sustainability, since they reduce the model cost; and while 2 reduces cost only in the training phase, 4 does so also in the inference phase.

## introduction a key component required to train language models is the tokenizer. one "basic" naïve approach is word-based tokenization-splitting the input into words, and assembling a training sequence as the list of word tokens. the problem with this approach is out-of-vocabulary tokens (oov) during inference-i.e., tokens not seen during training. one solution is to pick a set of most frequent words, and replace all other words with a special [unk] token @xcite . this works well only when the unknown vocabulary is small @xcite . another solution is to use an extensive dictionary @xcite . this approach is problematic for morphologically-rich languages, such as finnish or russian. the wordbased approach results in a huge vocabulary, and still very many oov tokens, due to the possibly rare morphological variants @xcite . at the other extreme is character-based tokenization-treating each character as a separate token; but this leads to long sequences, and nonmeaningful tokens, which inevitably increase the computational complexity @xcite . hence, the most common approach is in the middle-"sub-word" tokenization. a segmenter is an algorithm that splits words into sub-word units, between word-and character-based tokenization. the most common choice for a segmenter is the "byte-pair encoding" (bpe) algorithm, originally designed for simple data compression @xcite . currently most state-of-the-art neural language models, such as gpt @xcite and bert @xcite , use it as the segmenter. the idea behind bpe and its variants is to compute the frequencies of all segments, to iteratively merge the most frequent consecutive pair of segments into one segment, and add it to the vocabulary of tokens. this yields a compression of the data, and also a way to represent the data using a finite vocabulary of segment tokens-much smaller than by using entire word tokens. bpe works well in training language models, as it reduces the size of the vocabulary while still preserving some frequent words in the language as tokens. however, bpe has several problems. it is a greedy algorithm, and gives a crude approximation to the "true" linguistic structure of words, that is, morphemes. a morpheme is a linguistically meaningful segment of a word-it carries either semantic or syntactic content, and cannot be split further into smaller segments. also, morpheme retains its meaning in different contexts. the goal of this work is to determine whether we can improve the performance of language models by using segmenters that are more sophisticated than bpe-linguistically motivated and morphologically informed. for example, bpe may decide to segment the english word baking into two frequent segments: ba•king. however, we may consider bak•ing a "better" segmentation: root morpheme bak and suffix morpheme ing. it stands to reason that "knowing" about morphology-having more meaningful segments in its vocabularyshould help the lm predict data better. to be fair, the lm might be able to "recover" from the naïve segmentation, and still achieve excellent performance on various tasks, but it will have to do so at a higher cost!-in terms of more parameters and longer training-which degrades sustainability. which segmentation algorithm yields a better language model? we explore the "goodness" of a lm through four research questions: the paper is organized as follows: section 2 briefly reviews prior work on language modeling with morphological features, and the segmentation methods. section 3 introduces the language models and the downstream tasks used for evaluation. section 4 presents the results of our experiments. section 5 concludes and discusses future work. ## prior work sub-word tokenization is a well-studied topic in natural language processing. several approaches are proposed to segment words into sub-word units, @xcite @xcite . however, few papers have studied the effect of sub-word segmentation on the performance of neural language models. hofmann et al. (2021) discuss how sub-word segmentation affects bert's interpretation of complex words. they conduct a series of semantic probing tasks and show that applying morphological segmentation is beneficial for language models. park et al. (2021) trained models with several segmentation algorithms, including bpe and morfessor @xcite , on a corpus of bible verses in 92 languages. they evaluate the models according to surprisal (negative loglikelihood) per verse. their results show that morfessor segmentation yields much lower surprisal per verse than character or bpe segmentation for the overwhelming majority of the tested languages. bostrom and durrett (2020) compare the output of bpe and unigram tokenization @xcite on english and japanese. comparing with goldstandard morpheme boundaries, they found that unigram tokenization segments words more closely to morphological references, while bpe greedily merges sub-words according to their frequency, even if the sub-word pair is not semantically meaningful. they experimented with evaluating language models on downstream tasks, and pre-trained models from scratch with various segmentation methods. they found that unigram tokenization outperforms bpe on both english and japanese. toraman et al. (2022) conduct a comprehensive study to evaluate how tokenization affects the performance of turkish lms. besides bpe and its variant wordpiece @xcite , they apply a morphological analysis tool for turkish, and use it as sub-word segmenter. similarly to @xcite , they conducted experiments by evaluating six downstream lm tasks. although bpe and wordpiece achieve the best performance, lm with morphology-based segmentation achieve comparable results. they also point out that lm with a bigger vocabulary size can generally perform better. however, a large vocabulary increases the computational complexity. ## methods to evaluate the impact of the segmentation on the language models, we conduct several experiments. for each lm-bert and gpt, we perform four evaluations, corresponding to the four research questions. each model is trained with the subword lexicon resulting from the three segmentation algorithms: bpe, morfessor, and statemorph. ## experiments and results this section presents a series of experiments that we conduct to address the research questions. (a) english gpt (b) turkish gpt figure 2: learning curves for pre-training english and turkish gpt with different segmentation methods. ## conclusions and future work we have explored the impact of intelligent segmentation algorithms on the performance of language models. we experiment with four languages: finnish, russian, turkish, and english, with an in-depth investigation of the first two. 10 we train two language models-gpt and bert-and compare the statistical segmentation algorithm, bpe, with two morphological segmentation algorithms: morfessor and statemorph. we aim to show that lms trained on a vocabulary based on morphological information are better than lms trained on "naïve" sub-word segments produced by bpe. although bpe does not explicitly model morphology, it will inevitably stumble into discovering some morphemes as well-because many morphemes also happen to be frequent subwords. this makes bpe a tough baseline to beat. we explore four research questions: does morphological segmentation help lms-1: reach lower perplexity; 2: learn and converge faster; 3: perform at least as well on downstream tasks; 4: perform at least as well with smaller model size. we show that lms trained with morphological segmentation reach much lower perplexity (except morfessor, which has the largest vocabulary) than lms trained with bpe (rq1). we also show that lms trained with morphological segmentation converge faster than lms trained with bpe (rq2). we evaluate the performance of language models on several downstream tasks (rq3). the results show that the performance of lms with morphological segmentation (including smaller lms) is comparable to models with bpe. while performance on the topic classification tasks is quite convincing, we meet several challeges with other downstream tasks. the tasks are so high-level and so difficult, and the baseline performance is so low, that the gains from "smarter" segmentation may not be easy to demonstrate directly. the languages we work with have a paucity of "standard" datasets for downstream tasks with sufficient labeled data. in the future, we will investigate more languages and more downstream tasks. however, the theoretical results from rq1, 2 and 4 are convincing. to investigate the impact of segmentation on model size (rq4), we pre-train a smaller version of each lm with statemorph, for each language. we show that-for a fixed vocabulary size-small lms with statemorph segmentation have lower perplexity than regular-sized lms with bpe segmentation. this suggests that morphological segmentation can reduce the size of language models, and improve the sustainability of lms (rq4). sustainability is impacted by rq2, but even more so by rq4, since rq2 affects only the training, whereas rq4 affects training and inference. in future work, we plan to experiment with more morphological segmentation algorithms, a broader range of languages, and more types of language models, such as transformer-xl @xcite and xlnet @xcite . as a final point, the smarter segmentations that we have tested have much room for improvement-e.g., we can expect that supervised or rule-based morphological segmentation will be still better than the unsupervised segmentation that we have tested so far.