Morphology Matters: A Multilingual Language Modeling Analysis (2021)

Abstract: Prior studies in multilingual language modeling (e.g., Cotterell et al., 2018; Mielke et al., 2019) disagree on whether or not inflectional morphology makes languages harder to model. We attempt to resolve the disagreement and extend those studies. We compile a larger corpus of 145 Bible translations in 92 languages and a larger number of typological features.1 We fill in missing typological data for several languages and consider corpus-based measures of morphological complexity in addition to expert-produced typological features. We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data. We also investigate linguistically motivated subword segmentation strategies like Morfessor and Finite-State Transducers (FSTs) and find that these segmentation strategies yield better performance and reduce the impact of a language’s morphology on language modeling.

## introduction with most research in natural language processing (nlp) directed at a small subset of the world's languages, whether the techniques developed are truly language-agnostic is often not known. because the vast majority of research focuses on english, with chinese a distant second @xcite , neither of which is morphologically rich, the impact of morphology on nlp tasks for various languages is not entirely understood. several studies have investigated this issue in the context of language modeling by comparing a 2. if so, how do different segmentation methods interact with morphology? in order to answer the first question, we train models using data sets segmented by characters and bpe units. our results show that bpe language modeling surprisal is significantly correlated with measures of morphological typology and complexity. this suggests that bpe segments are ineffective in mitigating the effect of morphology in language modeling. as for the second question, we consider more linguistically motivated segmentation methods to compare with bpe: morfessor @xcite and finite-state transducers (fsts) (see ## modeling difficulty across languages studies have demonstrated that different languages may be unequally difficult to model and have tested the relations between such modeling difficulty and morphological properties of languages, using different segmentation methods. @xcite compared the effectiveness of word representations based on different segmentation methods in modeling 10 languages with various morphological typologies. they trained word-level language models, but utilized segmentation methods to create word embeddings that included segment-level information. comparing character, bpe, and morfessor segmentations, they concluded that character-based representations were most effective across languages, with bpe always outperforming morfessor. however, models based on hand-crafted morphological analyses outperformed all other segmentation methods by a wide margin. @xcite trained n-gram and neural language models over 50 languages and argued that the type of morphological system is predictive of model performance. their results show that languages differ with regard to modeling difficulty. they attributed the differences among languages to four types of morphological systems: isolating, fusional, introflexive, and agglutinative. while they found a significant association between the morphological type and modeling difficulty, type-token ratio (ttr) was the most predictive of language modeling performance. @xcite arrived at a similar conclusion modeling 21 languages using the europarl corpus @xcite . when trained with n-gram and character-based long short-term memory (lstm) models, the languages showed different modeling difficulties, which were correlated with a measure of morphology, morphological counting complexity (mcc) or the number of inflectional categories @xcite . however, @xcite failed to reproduce the correlation with mcc when they increased the scope to 69 languages, utilizing a bible corpus @xcite . they also reported no correlation with measures of morphosyntactic complexity such as head-pos entropy @xcite and other linguistgenerated features @xcite . rather, they found that simpler statistics, namely, the number of types and number of characters per word, correlate with language model surprisal using bpe and character segmentation, respectively. ## morphological measures different measures of morphology are used to represent a language's morphology. ## methods we design our experiments to test whether a language's morphology is correlated with language model performance, depending on the segmentation method. we represent a language's morphology using wals features and corpus statistics. we train language models for bible translations in 92 languages based on five different segmentation methods: character, bpe, morfessor, and fst with bpe or morfessor back-off strategies (fst +bpe & fst+morfessor). we use surprisal per verse @xcite as the evaluation metric to compare language modeling performance across different languages and different segmentation methods. additionally, we quantify the difference in surprisal per verse between segmentation methods to compare the relative strength of each segmentation method with regard to morphological complexity. ## discussion our results show that bpe models' surprisal per verse is highly correlated with a language's morphology, represented by several wals features and corpus-based measures. morfessor shows weaker correlations with such measures and records better performance for most of the languages. fst-based models outperform others when available. in this section, we discuss the implications of these findings in the context of previous work and future research.