The Effect of Model Capacity and Script Diversity on Subword Tokenization for S orani K urdish (2024)

Abstract: Tokenization and morphological segmentation continue to pose challenges for text processing and studies of human language. Here, we focus on written Soranî Kurdish, which uses a modified script based on Persian and Arabic, and its transliterations into the Kurdish Latin script. Importantly, Perso-Arabic and Latin-based writing systems demonstrate different statistical and structural properties, which may have significant effects on subword vocabulary learning. This has major consequences for frequency- or probability-based models of morphological induction. We explore the possibility that jointly training subword vocabularies using a source script along with its transliteration would improve morphological segmentation, subword tokenization, and whether gains are observed for one system over others. We find that joint training has a similar effect to increasing vocabulary size, while keeping subwords shorter in length, which produces higher-quality subwords that map onto morphemes.

## introduction different scripts for the same language may convey different linguistic and structural properties, such as phonological transparency, word boundaries (e.g., whitespace), morpheme boundaries, serial position within a word, or present different orthotactic and spelling constraints. in this work, we examine the relationship between script variation, morphological acquisition, and subword vocabulary construction. obtaining high-quality morphological annotations is critical for linguistic analysis, so unsupervised methods for learning morphemelike representations are often an acceptable compromise. here, we explore the usefulness of subword vocabulary training for morphological segmentation of written soranî kurdish, a central dialect of kurdish spoken mainly in iran and iraq. soranî is morphologically complex but relatively underresourced, with few large annotated corpora @xcite @xcite @xcite , and none with adequate morphological glosses or segmentations for downstream language model development @xcite or linguistic analysis. soranî has some unlabeled raw text corpora, which opens the possibility to leverage the statistical properties of the text for unsupervised subwordbased vocabulary induction. the existence of multiple writing systems for kurdish languages additionally presents a challenge for nlp systems, and jointly training subword tokenization models may be advantageous for central kurdish nlp in general. we thus ask whether training a subword vocabulary on multiple scripts can induce adequate morphological segmentations and compare such systems against models trained solely on single scripts of equivalent or larger sizes. ## soranî morphology and script variation our manipulation leverages script variability in the written kurdish dialects. kurdish dialects have been written with diverse writing systems including arabic, latin, yekgirtú (unified), cyrillic and armenian scripts. there is no unified orthography for kurdish despite previous efforts @xcite . this variation presents an intriguing opportunity to explore the impact of input diversity on the learning of subword vocabulary. furthermore, demonstrating the potential usefulness of joint training on multiple scripts could produce higher-quality multi-dialect transformer language models @xcite . the soranî writing system used for central and southern kurdish is written in a modified perso-arabic script and has an alphabetic structure with a high degree of phonological transparency, relative to arabic and persian scripts @xcite . the latin-based hawar al-phabet, used by northern kurdish dialects, shares this transparency, making it feasible to transliterate soranî script into a latin-based one @xcite . the latin script has two allographs per segment (e.g., h/h), which mostly encode sentence position, but the perso-arabic script has three for word-initial, -medial, and -final positions (e.g., the phoneme /h/ is represented by ' ' word initially, by ' ' word medially and by ' ' or ' ' word finally. ## subword tokenization we explore multiple tokenization models for soranî kurdish, with a primary focus on byte pair encoding (bpe; @xcite and unigram tokenization @xcite . we assessed these models' performance with respect to morphological and phonological structure and critically assess claims of better morphological induction by unigram relative to bpe @xcite . both bpe and unigram lm have probabilistic components based on frequency and vocabulary likelihood, respectively. these models were selected for their ability to handle diverse linguistic data and to learn meaningful linguistic units from large datasets without extensive annotated resources. the focus on these tokenization models that are used in modern neural methods is to align any prospective tokenization system with current trends, enabling scalability and robustness across different datasets and providing an exploration of their adaptability to the morphological richness of kurdish in both latin and arabic scripts. byte-pair encoding. bpe is a simple and common subword tokenization algorithm @xcite ) that grows a vocabulary from individual characters into more complex subwords by merging the most frequent co-occurring character sequence, up to a specified number of merges. given the frequency-based merging process of bpe, it is plausible that manipulating the relative frequency of subwords by training multiple scripts will influence the resulting vocabularies. we further hypothesize that the different character frequency distributions of latin and perso-arabic scripts may help bpe to learn subwords that better align with morphological boundaries and better capture the tendency of non-stem morphemes in kurdish to be short. unigram tokenization. this subword method iteratively splits words into subwords by optimizing the likelihood of the training data, which providing a probabilistic approach to subword segmentation that may capture more nuanced linguistic patterns compared to bpe's frequency-based merging strategy @xcite . unigram tokenization has been argued to produce better morphological segmentations than algorithms like bpe or wordpiece @xcite . we expect unigram tokenization to potentially provide more comprehensive coverage of soranî morphology compared to bpe, due to the likelihood objective of unigram. ## experiments we used the huggingface tokenizer package for bpe @xcite and unigram tokenization @xcite . for our experiments, we used the normalized version of the asosoft corpus @xcite consisting of 188 million word tokens and 4.66 million word types. the corpus was chosen for its comprehensive coverage of the soranî dialect. the corpus includes 58,000 documents from textbooks and magazines and 400,000 documents from web crawls. we removed newline characters, repeated characters @xcite , and redundant whitespace before subword training. we tested separate models for latin and arabic scripts, each with a 5k vocabulary size. additionally, we constructed a joint arabic-latin script corpus for data augmentation and further constrained model size based on script-specific vocabulary sizes derived from this joint corpus. vocabulary size. we explored various vocabulary sizes within a range of 1,000 to 10,000 subwords to identify the optimal balance between granularity and generalization. for bpe, we found that a larger vocabulary size of 5,000 subwords provided the best results, and so we use this size in all our experiments. this size qualitatively offered a good trade-off between identifying roots and affixes versus learning morphologically complex words, capturing the morphological complexity of soranî in both latin and arabic scripts. across all of our measures, 5,000 subwords each for the latin and arabic scripts led to the highest performance. transliteration. the latin-based script exhibits a one-to-one correspondence between phonemes and alphabet letters @xcite that can be deterministically transliterated from the arabic script using asosoft @xcite . in addition to changing character frequencies caused by multiple allographs, transliteration into the latin script introduces the letter "i" for the schwa, which is not encoded in the perso-arabic script. the relative transparency of the latin script may produce more accurate segmentations than the arabic script. data "augmentation." we define a joint tokenizer as a tokenization model trained on text data from multiple scripts simultaneously. for kurdish languages, which can be written in both latin and arabic scripts, a joint tokenizer aims to create a unified subword vocabulary that can effectively tokenize text for multiple dialects. this approach combines text data from both scripts for a balanced training set, which the bpe and unigram model then uses to develop a script-agnostic tokenization strategy based on subword frequency. this effectively doubles the training data set size and may alter the relative frequencies of subwords in the data. we measure the different tokenizers' precision against verified morphological segmentations of soranî kurdish, along with segmentation accuracy. we hypothesize changes in subword tokenization following from the fact that the two scripts have slightly different orthotactics (see section 2). transliteration is hypothesized to enhance subword vocabulary training by increasing the number of data points under consideration @xcite . ## future work a specific subset of words containing the zerowidth non-joiner (zwnj) was deliberately isolated to assess tokenization performance of the unigram tokenizer with 5000 subwords, particularly within the perso-arabic script. the presence of zwnj, which can act as a morphological delimiter in word or appear after the letter ' ' /a/ by pressing the e key on the keyboard, helps in achieving more accurate segmentation outcomes. for instance, in the word tokmetir ' ' 'stronger', the word tokme ' ' is separated from the comparative morpheme tir ' ' by a zwnj which gets tokenized as a two actual morphemes by the unigram tokenizer. this structural feature can provide clues to tokenization models, enabling more precise identification and segmentation of morphemes and a higher granularity in morpheme segmentation compared to the latin script. such findings underscore the importance of leveraging script-specific orthographic cues to improve tokenization models for tokenizer vocab word latin tokens morphemes unimoph 5k destîpêkird ['destîpêkird'] dest-î-pê-kird unimorph 5k meseley ['meseley'] mesele-y unimorph 5k pîlangêřîyekan ['pîlan', 'gêř', 'îyekan'] pîlan-gêřî-yek-an bpe 5k destîpêkird ['destî', 'pêkird'] dest-î-pê-kird bpe 5k lebexda ['lebexda'] le-bexda bpe 2514/2446 damezrawekanî ['damez', 'rawe', 'kanî'] damezraw-ekan-î bpe 2514/2446 lelayekewe ['lelay', 'ek', 'ewe'] yekêtî-ye bpe 2514/2446 gořanêk ['gořan', 'êk'] gořan-êk while recognizing the contributions of traditional unsupervised segmenters such as morfessor @xcite , adaptor grammars @xcite and dpseg (dirichlet process-based segmenter) @xcite in morphological analysis, this research primarily explores the application these subword tokenizers that are used in modern neural methods. we will extend this comparison to include these traditional segmenters, particularly focusing on their unigram versions which share similarities with the unigram model used in this study. for future work, we wish to explore the effects of using smaller training datasets with less bias in frequency distribution, build tokenization models based on vocabularies rather than corpora, and train greedy contextual decoding tokenizers (e.g., @xcite . ## conclusion in this study, we have explored the capacity of different tokenization models to segment soranî kurdish text into morphologically well-formed subwords. our findings highlight the differentical effects of pruning and merging on the inductive biases of these models, shedding light on their ability to capture morphological structures. we find that unigram tokenization leads to the highest quality off-the-shelf morpheme segmentation and find that data augmentation is a less effective strategy than increased vocabulary size in a monoscript context. this research will contribute to the development of more effective nlp tools for low-resource languages with smaller sources and only vocabulary lists, with a focus on morphologically and phonologically motivated analyses.