Splintering Nonconcatenative Languages for Better Tokenization (2025)

Abstract: Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER’s merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.

## introduction large language models (llms) have become pivotal in natural language processing (nlp), offering extensive utility across diverse applications. central to constructing an llm is producing basic input units from the text sequence, for which subword tokenization is still the standard approach, using methods such as byte-pair encoding (bpe) @xcite , wordpiece @xcite , and unigramlm @xcite . however, subword tokenizers exhibit diminished effectiveness in nonconcatenative languages (ncls) such as hebrew and arabic @xcite . while tokenizers assume linear segmentation of words, ncls' units of meaning are typically intertwined within words, and cannot be separated linearly, as the root letters are not adjacent @xcite . a hebrew example is provided in we develop splinter as a pre-tokenization step designed to address the challenges of subword tokenization in ncls. the core idea of the algorithm is that in certain ncls, many words are formed by embedding root letters into specific morphological templates. for instance, the hebrew word ‫לעבוד‬ is derived from the root ‫עבד‬ placed in the template ‫ו‬ ‫.ל‬ since there are far fewer templates than roots, template letters tend to appear in specific positions within a word more consistently than root letters do (e.g., for ‫:לעבוד‬ "0: ‫:3","ל‬ ‫.)"ו‬ empirically, we observed that in hebrew and arabic, when a word is longer than 3 characters, there is always at least one deletion that, when applied, transitions the word to a different, existing template while preserving the same root. foot_1 for instance, the word ‫לעבוד‬ turns into ‫לעבד‬ when the template letter ‫ו‬ is removed. by repeating this process iteratively, the word eventually reduces to only its root letters. this method can be seen as a way to isolate the root letters from the template. in this method, ncl words are transformed into a sequence of single-letter reductions, with the goal of rearranging them to better align with existing subword tokenizers. this is achieved by expanding the language's alphabet to include not only its original letters but also single-letter reductions, where each reduction consists of a letter paired with the index from which it was removed. 3 from another perspective, this method explores the impact of expanding a language's alphabet to enhance word representation. we illustrate the high-level application of splinter into the nlp pipeline in figure 1 . ## intrinsic evaluation to evaluate the effects of using splinter as a pre-processing step before tokenization, we trained multiple tokenizers on raw text from nonconcatenative languages and their splinter-treated counterparts. we examined the performance of both the bottom-up bpe tokenization algorithm, which works by iteratively merging tokens based on corpus co-occurrence statistics, and the top-down uni-gramlm approach, which starts with a very large vocabulary and iteratively removes from it tokens which contribute minimally to the corpus's likelihood. we train over a wide range of vocabulary sizes in order to assess the utility of splinter in multiple scenarios: from multilingual models which can allocate roughly 1,000 tokens for a given language, to dedicated monolingual models with room for two orders of magnitude more tokens. we selected three languages for our experiments: hebrew and arabic are semitic languages featuring root-template morphology as discussed above, while malay is an austronesian language rich in circumfixes and infixes-morphemes which break either the stem or the affix when forming the composite inflection. we computed the splinter operations for each language using the reductions map generated from the november 2023 wikipedia dump of the respective language. the wikipedia dump sizes were 1.9gb for hebrew, 3.0gb for arabic, and 0.4gb for malay, and were downloaded using the hugging face "datasets" library. we trained the tokenizers on the same wikipedia dump used for splinter training, using google's sen-tencepiece library with default settings, except for the tokenizer type (unigramlm or bpe) and vocabulary size (800, 1k, 2k, 10k, 32k, 64k, 128k). we qualitatively examine the differences between the vanilla tokenizer and the splinterenhanced one. 7 figure 1 illustrates such differences: the vanilla bpe tokenizer linearly segments the word ‫לעבוד‬ 'to work' into " ‫"לע‬ and " ‫,"בוד‬ thereby splitting the root ‫עבד‬ across two tokens. in contrast, the non-linear bpe + splinter tokenizer separates the word into two tokens: one for the template letters "3: ‫:0,ו‬ ‫"ל‬ and another for the root " ‫."עבד‬ a similar pattern is seen in the noun ‫חישוב‬ xišuv 'computation', where the vanilla bpe tokenizer linearly segments it into " ‫"ח‬ and " ‫,"ישוב‬ while bpe + splinter produces a token for template letters "1: ‫:3,י‬ ‫"ו‬ and another token to the root " ‫."חשב‬ likewise, for the definite adjective ‫הפשוט‬ ha-pašut 'the simple.sg.masc', the vanilla tokenizer segments it into three tokens: " ‫,"הפ‬ " ‫"ש‬ and " ‫,"וט‬ whereas bpe + splinter separates it into "0: ‫:3,ה‬ ‫"ו‬ and the root " ‫."פשט‬ for direct evaluation of the tokenizer vocabularies, independent of further language model architecture and training, we follow the analytical procedures collected in @xcite , adding pairwise comparative measures from other sources as well. vocabulary overlap first, we validate that splinter provides models with vocabularies that are different enough from raw-text tokenizers. hypothetically, if many common words are learned in full as single tokens from raw text, there is no need for a special pre-processing step to account for an edge case. however, in figure 2 we show that this is not the case. in all three languages, the maximal vocabulary size of 128k stays at an intersection level below 75%, with the slope of added shared token rate declining to near constant. moreover, even if we assume a linear extrapolation rate, the intersection rate would only exceed 85% at a vocabulary size of around 780k, which is exceptionally large and is not used even in sota english-dominated llms like gpt-4o. we note that we used a generous calculation for the intersection rate, as not all tokens in splinter-enhanced tokenizers can be directly compared to those in a regular tokenizer. for instance, a token representing the reduction "0: ‫:3,ל‬ ‫"ו‬ cannot be linearly converted into a standard token. to make the comparison as permissive as possible, we applied the reductions within the token (e.g., converting "0: ‫:3,ל‬ ‫"ו‬ into " ‫)"לו‬ and counted it towards the intersection if the resulting token existed in the vanilla tokenizer's vocabulary. as a result, the actual intersection percentage may be significantly lower than reported. we conclude that splinter-enhanced tokenizers produce substantially different vocabularies at all stages of the vocabulary creation process. cognitive plausibility we use the metric introduced in @xcite to measure the correlation of the tokenizer output with the response time and accuracy of human performance on a lexical decision task. this metric is based on the hypothesis that an effective tokenizer encounters difficulty with character sequences that are also challenging for humans, and vice versa. we use the hebrew cognitive plausibility dataset (help; @xcite to evaluate both the bpe and uni-gramlm tokenizers. each tokenizer was compared across seven vocabulary sizes, with the vanilla tokenizer evaluated against the splinter-enhanced tokenizer. following @xcite , we report the average of the absolute value correlation scores across the four linguistic setups (word/nonword × accuracy/response time). higher scores mean better correlation with human performance. as shown in ## language modeling with splinters as noted above, standard subword tokenizers are suboptimal for nonconcatenative languages such as hebrew and arabic. thus, we proceed now to evaluate splinter's impact on downstream nlp tasks for hebrew. specifically, we evaluate prefix segmentation and syntactic parsing @xcite @xcite , and question answering @xcite . we begin with the open dictabert bert-base model @xcite , which delivers current sota-level performance on the aforementioned tasks @xcite the splinter-processed tokenization. we then fine-tune the new base model for the aforementioned tasks, and evaluate performance vis-a-vis fine-tuning the original dictabert. we follow the same task parameters defined by @xcite and @xcite for the three tasks. we present the results in ## conclusion in this work, we introduced splinter, a novel pre-processing method for subword tokenizers designed to improve downstream performance on nonconcatenative languages (ncls). by applying an iterative reduction process, splinter restructures words in a way that better aligns with existing subword tokenizers. our approach was designed with key constraints in mind: ensuring lossless transformation, compatibility with existing tokenization frameworks, and applicability across different vocabulary sizes and model types, whether for an ncl, a single-language lm, or a multilingual model based on english with a limited number of tokens allocated for nonconcatenative languages. through intrinsic evaluations, we demonstrated that splinter-enhanced tokenizers exhibit distinct vocabulary distributions compared to vanilla tokenizers. cognitive plausibility metrics indicated that splinter improves alignment with human-like lexical processing, while our analysis of compression-related metrics revealed that splinter trades off slight reductions in compression efficiency for potentially better linguistic representation. our downstream evaluation highlights splin-ter's impact, particularly on higher-level nlp tasks such as question answering and on charactercritical tasks such as prefix segmentation. the intermediate syntactic level appears to be less affected by the nonconcatenativity of hebrew text. in future work, we will extend the downstream evaluation to arabic and other semitic languages, as well as more languages exhibiting non-templatic nonconcatenative phenomena. additionally, we plan to evaluate the performance of a large multilingual generative model on various tasks after incorporating splinter, examining its effectiveness in a broader linguistic context.