Subword Segmental Language Modelling for Nguni Languages (2022)

Abstract: Subwords have become the standard units of text in NLP, enabling efficient open-vocabulary models. With algorithms like byte-pair encoding (BPE), subword segmentation is viewed as a preprocessing step applied to the corpus before training. This can lead to sub-optimal segmentations for low-resource languages with complex morphologies. We propose a subword segmental language model (SSLM) that learns how to segment words while being trained for autoregressive language modelling. By unifying subword segmentation and language modelling, our model learns subwords that optimise LM performance. We train our model on the 4 Nguni languages of South Africa. These are low-resource agglutinative languages, so subword information is critical. As an LM, SSLM outperforms existing approaches such as BPE-based models on average across the 4 languages. Furthermore, it outperforms standard subword segmenters on unsupervised morphological segmentation. We also train our model as a word-level sequence model, resulting in an unsupervised morphological segmenter that outperforms existing methods by a large margin for all 4 languages. Our results show that learning subword segmentation is an effective alternative to existing subword segmenters, enabling the model to discover morpheme-like subwords that improve its LM capabilities.

## introduction subword segmentation has become a standard practice in natural language processing (nlp). the dominant approach is to run an algorithm like bpe @xcite as a preprocessing step, segmenting the corpus into subwords. this enables the model to learn features based on subwords, compose words, and handle rare and unknown words as an open-vocabulary model. subword segmentation is an active area of research, since no single technique outperforms others across all tasks, languages, and dataset sizes @xcite sides deterministic segmenters like bpe, stochastic algorithms like unigram lm (ulm) @xcite have also been proposed. subword segmentation is particularly important for the nguni languages of south africa (isixhosa, isizulu, isindebele, and siswati) because they are agglutinative languages that are written conjunctively. 1 these are morphologically rich languages in which words are formed by stringing together morphemes @xcite . morphemes are the primary linguistic units. for example, the isixhosa word "sesihambe" means "we are gone", where "se" means "we", "si" means "are", and "hamb-e" means "gone", with the "-e" suffix indicating past tense. as shown in the nguni languages are under-resourced, which compounds the importance of subword segmentation. available datasets are small, so any held-out dataset will contain rare or previously unseen words. therefore it is critical for models to learn useful subword features and effectively model morphological composition. in a low-resource setting it may then be more effective to learn subword segmentation as part of model training rather than as a distinct preprocessing step. the probabilistic models underlying existing subword segmentation methods such as ulm and morfessor @xcite assume that subwords are context-independent, making them unsuitable for language modelling. in this paper we propose a subword segmental language model that simultaneously learns how to segment words while training as an autoregressive lm. this allows the model to learn subword segmentations that optimise a left-to-right language modelling objective, thereby being conditioned on the context. our model learns the subwords that it can most effectively leverage for language modelling. we train our model in the 4 nguni languages of south africa. we compile lm datasets for these languages from publicly available corpora and release our train/validation/test sets. on intrinsic language modelling performance averaged across the 4 languages our model outperforms neural lms trained with characters, bpe, and ulm subwords. on the task of unsupervised morphological segmentation (which determines to what extent subwords correspond to actual morphemes) our model outperforms standard subword segmenters like bpe and ulm on average across the 4 languages. in addition to these lms, we train a second set of subword segmental models that train on single words in isolation (without having to model context for long-range language modelling). our word-level models outperform all existing methods on unsupervised morphological segmentation (including segmenters like morfessor) by a large margin across all 4 languages. finally, we discuss the importance of a subword lexicon to our model, analysing how hyperparameters that control lexicon construction affect performance. in summary, this paper makes the following contributions: 2 1. we propose a subword segmental language model (sslm) that unifies subword segmentation and language modelling in a single endto-end neural architecture. ## subword segmentation in this section we review the paradigm that currently dominates subword segmentation, discuss its limitations, and introduce the family of models we draw inspiration from for our approach to subword segmentation -segmental sequence models. ## subword segmental language model our sslm combines autoregressive language modelling and subword segmentation in a single model that can be trained end-to-end. the architecture is shown in figure 1 . it represents a radical divergence from segmenters like bpe and ulm, which view subword segmentation as contextindependent. the sslm views subword segmentation and language modelling jointly, so it can learn subwords that optimise conditional lm generation. ## language modelling we evaluate our sslm on intrinsic language modelling performance to determine whether learning subword segmentation during training can improve the inherent predictive capabilities of an lm. the lexicon is constructed before training. it contains all subwords shorter than a prespecified maximum length l that occur in the training corpus. the lexicon size v is also prespecified. the lexicon consists of the v most frequent subwords up to l characters long. the lexicon model p lex outputs a probability for each subword in the lexicon. ## analysis we analyse the effect of hyperparameter and architectural choices on the performance of our sslms. this section does not report findings for our wordlevel models, since we are primarily interested in investigating which model components contribute to the success of our long-range sslms (in terms of language modelling and morphological segmentation). @xcite found two components to be crucial to the success of their segmental lm for chinese word discovery: the lexicon and expected length regularisation. the former stores frequent subwords and the latter introduces a regularisation term to the training objective that encourages shorter segments. contrary to @xcite , we did not find length regularisation to be useful. this is because our datasets are much larger than those used in their ablation studies (they use the brent corpus of 27k words). when a segmental model is trained on a small dataset, it overfits by copying long segments from the lexicon. length regularisation prevents overfitting by biasing the model towards shorter segments. this is not a problem on larger datasets like ours, because the lexicon cannot cover all possible long segments in the corpus. in fact, our model tends to over-segment rather than under-segment. the lexicon proved to be essential for the sslm. during tuning it consistently improved validation bpc. the average lexicon coefficient (1g t in equation 6) of the isindebele sslm on the lm test set was 0.27, indicating a reliance on the lexicon for subword generation. we analyse two lexiconrelated hyperparameters: lexicon size and maximum segment length. figure 2 compares the performance of isindebele sslms across lexicon sizes and maximum segment lengths. figure 2 (a) plots intrinsic lm performance. smaller lexicon sizes improve lm performance up to a point, with 5k subwords being optimal. a maximum segment length of 10 characters achieves optimal performance across all lexicon sizes, striking a balance between memorising long segments where possible, and otherwise relying on short subwords. figure 2 (b) plots ums performance, where the picture is less clear-cut. since our model is an unsupervised morphological segmenter, we only considered lm performance (validation bpc) when tuning and selecting our final models. figure 2 shows that optimal lm performance does not necessarily imply optimal ums performance. longer maximum segment lengths tend to improve ums performance. biasing the model towards longer segments reduces the over-segmentation problem, but relinquishes some lm performance. however, there is at least some correlation between lm and ums performance, and the model selected on lm performance is not far off optimal ums accuracy. ## related work a few african languages have been included in large multilingual lms, such as mbert @xcite and xlm-r (conneau and lample, 2019). @xcite trained afrib-erta, a multilingual lm trained on 11 african languages. there has been less work on monolingual lms for african languages. ralethe (2020) trained afribert, a masked lm for afrikaans. @xcite trained bpe-based autoregressive lms for isizulu and sepedi. @xcite proposed kinyabert, a masked lm for kinyarwanda with a two-tier neural architecture that incorporates a morphological analyzer. ## conclusion in this paper we proposed subword segmental language modelling (sslm), an approach that unifies language modelling and subword segmentation. we showed that sslm improves intrinsic lm performance for low-resource agglutinative languages, while yielding subwords that approximate morphemes better than previous approaches. as opposed to most neural model architectures in nlp research, which are either language-agnostic or overfit to high-resource languages, our model is designed to suit agglutinative languages like the nguni languages of south africa. our results show that learning subword segmentation in training overcomes some of the limitations of existing subword segmenters. for future work, the sslm could be applied to downstream nlp tasks suited to its autoregressive architecture, such as text generation. more generally, the idea of learning subword segmentation during training could be adapted to other nlp models and tasks.