How Can We Relate Language Modeling to Morphology? (2025)

Abstract: The extent to which individual language characteristics influence tokenization and language modeling is an open question. Differences in morphological systems have been suggested as both unimportant and crucial to consider (e.g., Cotterell et al., 2018; Park et al., 2021, Arnett & Bergen, 2025). We argue this conflicting evidence is due to confounding factors in experimental setups, making it hard to compare results and draw conclusions. We identify confounding factors in analyses trying to answer the question of whether, and how, morphology relates to language modeling. Next, we introduce token bigram metrics as an intrinsic way to predict the difficulty of causal language modeling, and find that they are gradient proxies for morphological complexity that do not require expert annotation. Ultimately, we outline necessities to reliably answer whether, and how, morphology relates to language modeling.

## introduction are certain languages inherently easier or harder to model @xcite ? the interplay between language modeling and individual differences among languages is an open problem. one angle it can be approached from is morphological complexity @xcite : if in one language the internal structure of words is more unpredictable according to some standard than another, then perhaps language models (lms) have a harder time learning to predict text in that language. morphological systems are widely recognized as being gradient, but coarse groupings are often used, especially in nlp @xcite . agglutinative languages (als) tend to add one grammatical feature to a word with each added morpheme, resulting in long words with many mor-phemes. fusional languages (fls) tend to express information through inflection, where a single morpheme can express multiple features, resulting in shorter words with fewer morphemes. results contrasting als and fls have been mixed, with some evidence pointing to als being harder to model than fls (e.g., @xcite whereas others have shown that there is no difference between the two groupings (e.g., @xcite . we outline what experimental conditions and metrics are necessary to reliably answer whether, and how, morphology relates to language modeling. our contributions: (1) we list confounding factors that have to be taken into account when attempting to answer the central question above. they can be seen as criteria for an "ideal" experiment. (2) we propose predicting clm difficulty with the variety and entropic efficiency of neighboring tokens, and find they are proxies for morphological complexity. ## confounding factors it is not obvious how morphology impacts language modeling. what is clear is that research that seeks to draw reliable conclusions relating the two must control for the following confounding factors: these factors show a way towards an ideal experimental setup. practically, one must work backwards from this to a feasible setup. ## accessor variety we need a reliable proxy for morphological complexity. harris (1955) first suggested to count the variety of predecessor and successor units of a given string, where unusual spikes would imply the string's edges delineated something meaningful like a morpheme. @xcite coined accessor variety (av) as the minimum of predecessor and successor variety. @xcite applied this to learn bpe merges. we use ulm tokens. in the word-based metrics recover the groupings somewhat, but are less reliable for clms, unless those models also use words instead of subword tokens. the token unigram metrics mtl, re, and mattr look rather even across the languages in europarl. since these estimators become more accurate with more data, their low variance calls into question higher-variance results computed for much smaller corpora like flores-200. lastly, av operates on tokens, which means it's applicable to other units. for character-or bytelevel tokenizers, av can still provide an estimate of the degree of choice of accessors for a given type. in the full paper, we discuss hypotheses of other papers, present results for a larger set of languages, and suggest general methodological improvements for future investigations.