Subword Segmentation in LLM s: Looking at Inflection and Consistency (2024)

Abstract: The role of subword segmentation in relation to capturing morphological patterns in LLMs is currently not well explored. Ideally, one would train models like GPT using various segmentations and evaluate how well word meanings are captured. Since this is not computationally feasible, we group words according to their segmentation properties and compare how well a model can solve a linguistic task for these groups. We study two criteria: (i) adherence to morpheme boundaries and (ii) the segmentation consistency of the different inflected forms of a lemma. We select word forms with high and low values for these criteria and carry out experiments on GPT-4o’s ability to capture verbal inflection for 10 languages. Our results indicate that in particular the criterion of segmentation consistency can help to predict the model’s ability to recognize and generate the lemma from an inflected form, providing evidence that subword segmentation is relevant.

## introduction the linguistic abilities of large language models have been studied to a large extent, with many new abilities emerging as language models become ever larger and more powerful. while areas such as lexico-syntactic understanding, text generation and reasoning abilities have received much attention, morphology has only played a minor role, despite being of great interest to the nlp community. conceptually, the morphological abilities of a model are tightly linked to the internal representation of subwords: llms do not operate on complete words, but instead, most words are broken into subword pieces for better computational efficiency and to handle unknown words. subword segmentation strategies typically rely on frequency statistics and are not linguistically guided. this suggests that such segmentation strategies do not provide a suitable basis to fully capture morphology, e.g. @xcite ; @xcite . morphology relates to the construction of words, and thus represents the basis of understanding natural language. depending on the language, morphology can play a more or less relevant role, but even in a language with rather simple morphology such as english, morphology is indispensable, whether for rare words, or for more common ones. for instance, a botanizer is a person that botanizes, a baker's workplace is a bakery and a mathematician cares about mathematics. morphological processes are typically defined by general patterns, and, critically, understanding these patterns enables both the generation of novel words and the interpretation of previously unknown words. understanding the meaning of word parts in the larger context of a word, as well as the underlying patterns to compose new word forms is essential to fully comprehend language. this is particularly true for languages with complex morphology, where a larger proportion of information is encoded morphologically, leading to a comparatively high number of inflected forms that have insufficient coverage in the training data, and in the worst case do not occur in the training data at all. despite the impressive language capabilities of llms, the impact of the underlying segmentation is not clear. generally, llms are capable of modeling morphology and accessing morphological information, but presumably not on an ideal basis, because segmentation strategies, such as wordpiece or bpe @xcite rely on frequency-based heuristics that do not optimally capture morphological patterns. in the following, we study two criteria, adherence to morpheme boundaries and segmentation consistency of inflected forms of a lemma. we first analyze how well these criteria are met in existing llms, and then investigate to what extent words which have high or low values for these two criteria affect the performance of the llm on a linguistically interesting task. ## related work there is a large body of research concerning the representation of the training data of language models and translation systems: while the typical segmentation strategies are frequency-based such as wordpiece or bpe @xcite , there is also evidence that these segmentation approaches are not optimal for morphologically rich languages and fail to fully capture the morphological complexities of words @xcite ). @xcite show that a linguistically grounded segmentation can improve a model's performance. @xcite explore the effect of subword segmentation by training bert and gpt models on different segmentation algorithms, namely bpe and two morphological segmentation strategies. their experiments show that morphologically guided segmentation leads to lower perplexity and faster convergence during training; their models trained on morphologically segmented data reach a similar or better performance than models trained on bpe, depending on the task. furthermore, they find that models of smaller size trained on morphologically segmented data can perform comparably to models of larger size trained with bpe. while not specifically studying the impact of segmentation, but instead the multilingual capabilities of english-centric llms, @xcite assume that the quality of subword segmentation plays a part in the performance for languages different from english, as the segmen-tation is mostly based on the predominant english vocabulary and thus not representative of many other languages. their findings indicate that languages with more subword tokens per word tend to perform worse. there are many variants of language-specific plms trained on representations to accommodate the properties of a language, (e.g. antoun et al. (2020); nzeyimana and niyongabo rubungo (2022) ), mostly in a monolingual setting. jabbar (2024) proposes a linguistically-informed representation of the training data that relies on canonical forms instead of concatenable pieces. this makes the generation step less straightforward as the pieces cannot just be concatenated, but have to be reconstructed into inflected forms. the idea to combine linguistically guided segmentation with frequency-based segmentation has also been applied to machine translation, for example @xcite ; @xcite ; @xcite , and often found to be preferable to just frequency-based segmentation. a further task linked with the representation of sub words is that of morphological re-inflection (e.g @xcite ), where an inflected form needs to be generated for a given pair of word and morphological features. there is a growing interest in the quality of the underlying segmentation: beinborn and pinter (2023) look at the semantic plausibility of subword tokens; the segmentation strategy in @xcite aims at incorporating context information to obtain more meaningful splits. with regard to morphology, @xcite study the ability to create inflected forms for nonce words for typologically different languages, finding that gpt does not perform as well as systems specifically trained for morphological tasks. soler et al. (2024) study the impact of segmentation on the quality of word representation by comparing words that are segmented with those having a dedicated embedding, i.e. unsplit words, in a word similarity task. in general, they find that the representation of split words is often worse than for non-split words. interesting in the context of our work, their results show that a morphologically sound segmentation tends to lead to a better representation. with regard to over-splitting, their findings are mixed, but indicate that for split words, a higher number of tokens does not necessarily decrease representation quality. beinborn and pinter (2023) and @xcite propose to use the number of splits per word as an indicator for splitting quality, assuming that few splits per word suggest a "good" segmentation in contrast to a segmentation into many short pieces. to the best of our knowledge, there is no study that looks at segmentation criteria as outlined in this paper in combination with a linguistic task. ## methodology we study the quality of gpt-4o's segmentation for 10 languages (english, french, german, spanish, italian, portuguese, finnish, swedish, czech, hungarian). we look at the segmentation quality from two angles: first, we examine how well inflection suffixes are separated from the stem of the word, i.e. a linguistically-oriented criterion. second, we look at the segmentation consistency, i.e. whether all words from an inflection paradigm are segmented in a cohesive way. we assess whether the segmentation has an impact on the model performance. in previous work on subword segmentation, either on language modeling or on machine translation, the typical approach is to compare the performance of a model trained on a baseline subword segmentation with that of a model trained on a contrastive segmentation. working with an llm such as gpt, this strategy is not feasible due to the immense expense to train such a model. instead, we compare the outcome on a downstream task for words of different levels of segmentation quality, by selecting words with high and low values according to the criteria outlined previously. assuming that (i) the segmentation quality has an effect on the particular task and that (ii) the proposed criteria are suitable to capture the segmentation quality, we should be able to see a performance difference between the two sets. the linguistic task is that of predicting the lemma of an inflected verb form, which is applicable to every language in our data set; in a second experiment, we also generate inflected forms given the lemma and a morphological tag. we chose verbal morphology as it provides more variety than the inflection of nouns and adjectives. ## separation of stem and inflection in this first experiment, we apply a linguisticallyoriented criterion and study whether and how inflection suffixes are separated from the stem. we start from the hypothesis that a clean separation of inflectional suffixes allows for a better representation with regard to generalization due to separating the lexical content in the stem from the morphosyntactic information in the inflectional parts. we define five categories, as illustrated in the segmentation analysis in morphynet is in canonical notation, thus the concatenation of the segmentation analysis does not result in the inflected form itself, but in a sequence of the lemma and the inflectional suffix(es), for example (fr) @xmath0. as inflection suffixes, we consider all parts of the segmentation except for the first one, which is the lemma foot_1 . as we ignore the lemma part of the gold segmentation, there are no problems with irregular verb forms or stem changes between lemma and inflected form. many languages only have one suffix part, others like finnish or hungarian can have more. in the case of several suffixes, we only consider words where the concatenation of the suffixes in the gold segmentation also corresponds to the right side of the inflected word, but not forms like (es) @xmath1. the gpt segmentation was obtained for the target word without surrounding sentence context foot_2 . figure 1 shows the distribution of the segmentation categories for verbs. overall, the category while some of them are close to morphemes, the segmentation is not systematic in a linguistic sense. in particular the amount of the category concat is to a certain extent language-dependent as only languages with generally longer inflectional suffixes can have the suffix split into several pieces. however, even though many verbs are not split at the boundary between stem and inflectional suffix, there is still often some form of systematicity. ## segmentation consistency the criterion in the previous section was based on linguistic well-formedness; here, we look at segmentation quality from the angle of consistency, which also aims at capturing generalization abilities, but is formulated more robustly. we pursue the question whether a consistent segmentation across the inflected forms of a lemma provides a better basis for the representation than an inconsistent segmentation. the underlying assumption is that an internally coherent representation of different surface realizations of the same word should result in an overall better representation of that word, and thus provide a better basis for generalization and the modeling of potentially unseen words. ideally, a good segmentation should provide a consistent splitting of the stem part, with more necessary variation towards the end of the word. we use the overlap coefficient to measure the similarity between the sets of segments of two different verb forms, which is defined as the size of the intersection divided by the size of the smaller one of the two sets: the scores range between 0 (no overlap) and 1 (perfect match). a particular characteristic of this metric is that if a is a subset of b, then the coefficient is 1: this has the effect of comparing rather the segments of the stem part while disregarding suffixes that add to the overall length of the word, assuming that the stem part does not change much, whereas we expect comparatively more variation in the suffixes. in contrast, the jaccard index (ratio of intersection over union) might be less practical when the two compared forms are of different lengths, and we do not expect a subset to be similar. below are some examples for forms of the italian verb sorprendere (to surprise), and the respective overlap scores between lemma and form: lemma form overlap score s or pr endere s or pr endere bbe 1 s or pr endere s or pr end iamo 0.75 s or pr endere s or pre se 0.5 the splitting in the first line is linguistically questionable, but the lemma's segments are an exact subsect of the inflected form's segments, which is good in terms of consistency (@xmath2. for the other two words, the segments only partially match between the forms, and thus have a lower score. to obtain the overlap coefficient of an inflection paradigm, we computed the average of the overlap of every possible pair of forms. ## conclusion and future work we proposed two criteria to capture the quality of subword segmentation in llms and evaluated to what extent words which score high or low for these criteria affect the performance of the llm on a linguistic task for ten diverse languages. both criteria are targeted at the generalization abilities of the language model; the first one is more linguistically inspired and aims at a clear separation of stem and inflectional suffixes, whereas the second one rewards consistent segmentation within an inflection paradigm. the design of the criteria is in principal language-independent, but requires language-specific information: morphologically annotated data for the first one, and information about sets of inflection paradigms for the second one. the results of our experiments indicate that the subword segmentation does influence the behaviour of the model. in particular for the criterion of segmentation consistency, we could observe a better performance for the sets with higher segmentation overlap. in contrast, the morpheme-boundary criterion was found to be less suitable. with a view to linguistic resources, the consistency-based criterion departs from a more minimal point, as no morphological analysis is needed other than knowing the inflection paradigm of a word. the underlying segmentation is not only relevant for the representation within one language, but might also improve the multilingual competence of a model. conceptually, when adhering to morpheme boudaries, the resulting segmentation can separate between lexical parts and functional components, which might benefit multi-lingual and structure learning; for instance, through supporting the learning of lexical equivalents of words sharing the same (or close) orthographic forms of the stem with different inflections, such as symbol -ic en , -isch de , -ique f r , -iczne p ol . moreover, inflectional affixes contain context information such as tense or number, and an accessible and consistent representation can potentially contribute to the learning of syntactic structure across languages. finally, with view to the current efforts to include less-resourced languages into lms, segmentation strategies that promote a consistent representation and maximize the generalization abilities are a relevant and interesting research field.