Analyzing Cognitive Plausibility of Subword Tokenization (2023)

Abstract: Subword tokenization has become the de-facto standard for tokenization although comparative evaluations of their quality across languages are scarce. Existing evaluation studies focus on the effect of a tokenization algorithm on the performance in downstream tasks, or on engineering criteria such as the compression rate. We present a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. We analyze the correlation of the tokenizer output with the reading time and accuracy of human responses on a lexical decision task. We compare three tokenization algorithms across several languages and vocabulary sizes. Our results indicate that the Unigram algorithm yields less cognitively plausible tokenization behavior and a worse coverage of derivational morphemes, in contrast with prior work.

## introduction when we develop natural language processing (nlp) models, we first need to segment a stream of text into small processable units. this preparatory step is known as tokenization and it is more challenging than the segmentation of continuous sensor signals because human language uses symbolic representations. traditionally, the space-delimited word was considered a meaningful basic unit, but the concept cannot intuitively be mapped to languages with a rich morphological structure such as turkish or finnish, or to languages that use a writing system without white spaces such as chinese. more recently, the focus has shifted to smaller character sequences known as subwords, with the explicit goal of limiting the necessary vocabulary size (which affects model size and performance), and the implicit hope of better approximating semantically-meaningful linguistic units below the word level, i.e., morphemes. in practice, today's dominant subword tokenization algorithms are purely data-driven. they treat frequent sequences as single tokens (e.g., seafood), and split less frequent ones into multiple tokens composed of frequently occurring character sequences (e.g., @xmath0. subword splits might coincide with morpheme boundaries (the plural marker s), but not necessarily (seab). previous comparisons of tokenization algorithms focused on engineering-oriented desiderata such as processing speed and encoding efficiency, and on the performance of models on downstream nlp tasks @xcite . in this paper, we evaluate subword tokenizers from a cognitive perspective, utilizing lexical decision task measures as a proxy for the processing complexity of individual words. we evaluate the split rates of three tokenization algorithms on various languages. we find significant correlations in line with cognitive expectations, allowing systematic analyses of the influence of parameters such as vocabulary size. we observe that the unigramlm tokenization algorithm @xcite produces less correlative splits than bpe @xcite and word-piece @xcite , in contrast with previous evaluations made over corpus statistics and downstream tasks @xcite . in further experiments, we find that multilingual token vocabularies inhibit tokenizers' ability to predict cognitive performance as well as signs that current popular vocabulary sizes are insuffi-cient for morphologically-rich languages, echoing recent findings @xcite ). foot_0 2 self-supervised tokenization a tokenizer takes as input a sequence s of characters [c 1 , . . . , c n ] and splits it into non-overlapping substrings, the tokens [t 1 , . . . , t k ], where @xmath1. each token t i consists of a variable number of j consecutive characters (1 ≤ @xmath2, such that the concatenation of the tokens t i yields the sequence s. a tokenizer consists of a vocabulary consisting of m tokens and an algorithm that determines the best splits of the input s into vocabulary items t i . see @xcite for a detailed survey of tokenization approaches. evaluating vocabularies comparative evaluations of tokenization algorithms commonly focus on downstream performance and on cross-lingual differences. @xcite calculate the tokenization compatibility for pairs of languages and find that the vocabulary size of a tokenizer needs to be adapted to the characteristics of the language. multilingual language models use a single shared vocabulary for a large number of languages to facilitate cross-lingual transfer, however, @xcite find improvements when these are replaced with targeted monolingual tokenizers. @xcite propose to increase the vocabulary size for multilingual models and assign per-language budgets in a dynamic manner, in order to mitigate effects on the splitting ratio for languages less represented in the vocabularies. they de-emphasize token sharing between languages with little lexical overlap, in line with @xcite . @xcite propose to incorporate context sensitivity in order to generate more cohesive tokenization and show that their approach leads to increased downstream performance for both english, and the morphologically more complex language turkish. morphological evaluation more linguistically motivated evaluations of subword tokenization focus on morphological plausibility. @xcite compare the bpe and unigramlm algorithms for english and japanese and find that the segmentation produced by the latter aligns more closely with morphology and leads to better results on downstream tasks, especially for japanese. in a similar vein, park et al. (2021) find that bpe does not properly reflect morphological complexity and that enriching the model with explicit morphological information leads to reduced language modeling surprisal. @xcite show that a vocabulary with better morphological coverage leads to better performance in genre classification of english titles, and might lead to better generalization capabilities @xcite . other studies have shown that these consistent results in english do not necessarily generalize to other languages @xcite , particularly morphologically-rich ones @xcite . morphological segmentation is related to tokenization, but it is sensitive to phonotactic variations, e.g., discernible is segmented into discern and -able @xcite . nevertheless, the winning system at the sigmorphon shared task was based on subword tokenization and outperformed character-based approaches, indicating that subwords can approximate morphological boundaries @xcite . cognitive plausibility from a cognitive perspective, it remains an open question to which extent lexical processing is driven by morphological units. one of the most robust effects in lexical decision tasks @xcite is that morphologically structured non-words cause longer response times and lead to decreased accuracy in word detection. @xcite find that this effect is stronger for german than for french, and suggest that this is due to its larger degree of morphological productivity. @xcite compare lexical decision times for english words and find that priming with morphological components (teach) leads to faster responses (teacher) even if the prime has no semantic relation (@xmath3. @xcite show that the prediction of eye fixations is facilitated for english and dutch readers by operating on subtokens determined by unsupervised tokenizers instead of word units. @xcite claim that effects attributed to morphological decomposition cannot be easily disentangled from frequency effects, and urge nlp researchers to integrate response times into the evaluation of distributional approaches. ## experimental setup we train three tokenization algorithms on 100,000 sentences from the news domain of the leipzig tokenization models we use the huggingface implementations of three corpus-based subword algorithms: byte-pair encoding (bpe), wordpiece (wpc), and unigram (uni). foot_2 bpe originated as a compression algorithm @xcite and has been used in large pre-trained language models such as gpt-2 @xcite . bpe vocabularies are built bottom-up, starting with an initial vocabulary of all characters. the algorithm then iteratively merges sequences of characters frequent in the corpus and adds them as tokens to the vocabulary until reaching the maximum size. during inference, an input sequence is greedily split into tokens aiming for a minimum number of splits, see @xcite for a more detailed description. wordpiece @xcite is a variant of bpe which adds tokens to the vocabulary when they maximally increase the likelihood of an n-gram-based language model in the corpus, and is decoded greedily left-to-right to find the locally longest token available at each step. the unigramlm @xcite algorithm, in contrast, takes a top-down approach starting with an overly large vocabulary of all possible tokens, followed by iteratively pruning those that lead to minimal loss of likelihood over the corpus when they are removed from a token-unigram language model. cognitive data we use data from lexical decision tasks in british english @xcite , dutch @xcite , french @xcite , and spanish @xcite . in these tasks, participants are presented with a sequence of characters, e.g., thornier, and decide whether the sequence forms a valid word in their first language. the datasets contain information about the average response time (i.e., the number of milliseconds it took the participants to make a decision) 4 and accuracy for each stimulus. we base our metric on the intuition that fewer splits generally indicate a better fit and that longer sequences are more likely to be split. our metric therefore takes both n and k into account to measure how well the tokenizer handles the sequence. we define the chunkability of a sequence as a value that decreases as the ratio of tokens over characters increases: if a token is split into individual characters, chunkability is zero. for tokens that are not split, chunkability approaches one as the number of characters increases, reflecting the increasing challenge of fitting long words into the vocabulary. foot_4 ## results we test the hypothesis that sequences with higher chunkability are easier to process for humans and are more likely to be considered words. figure 1 visualizes pearson's correlation between the chunkability of a sequence and the response time and accuracy observed from humans rating the sequence, and for non-words, we observe the reverse tendencies: non-word sequences with a high chunkability such as catchwind require longer response times and are less accurately identified as non-words than an unusual sequence such as brithbloom. these patterns are consistent across algorithms and languages, whereas a baseline that only considers character length cannot capture the effect (longer sequences generally lead to longer response times). the correlation for the unigramlm algorithm is systematically lower than for the other two algorithms, suggesting a contrast with morphology-and corpus-based measures considered by @xcite . for french, chunkability seems to be less correlated with human responses, echoing findings related to cognitive performance in a morphologically-primed environment (beyersmann et al., 2020). while the absolute correlation scores might be of limited explanatory value, we find the relative differences between conditions a relevant point of information for further development. cross-lingual vocabulary it has been shown that the "curse of multilinguality" reduces the performance of multilingual models compared to their monolingual counterparts @xcite . @xcite show that this difference in performance is strongly related to the tokenizer. we compare cognitive plausibility of monolingual and cross-lingual tokenizers of pretrained models in figure 2 , and affirm that the monolingual tokenizer aligns much better with human responses than the cross-lingual ones. foot_6 see also appendix d. ## conclusion we propose a new evaluation paradigm for comparing subword tokenization algorithms using cognitive data. we introduce a novel metric to capture the chunkability of a sequence that correlates with cognitive phenomena of lexical recognition. the overall trends suggest that the connection between plausibility tasks and segmentation is meaningful enough to be used as a benchmark. we find a lower cognitive correlation for the unigramlm algorithm than for wordpiece and bpe, which does not nec-essarily align with previous work evaluating tokenizers on morphological segmentation and downstream performance, suggesting that our framework provides a complementary perspective to tokenizer vocabulary evaluation. our analyses on vocabulary size and morphological coverage provide initial insights towards the development of cognitively and linguistically more plausible tokenizers.