A Masked Segmental Language Model for Unsupervised Natural Language Segmentation (2022)

Abstract: We introduce a Masked Segmental Language Model (MSLM) for joint language modeling and unsupervised segmentation. While near-perfect supervised methods have been developed for segmenting human-like linguistic units in resource-rich languages such as Chinese, many of the world’s languages are both morphologically complex, and have no large dataset of “gold” segmentations for supervised training. Segmental Language Models offer a unique approach by conducting unsupervised segmentation as the byproduct of a neural language modeling objective. However, current SLMs are limited in their scalability due to their recurrent architecture. We propose a new type of SLM for use in both unsupervised and lightly supervised segmentation tasks. The MSLM is built on a span-masking transformer architecture, harnessing a masked bidirectional modeling context and attention, as well as adding the potential for model scalability. In a series of experiments, our model outperforms the segmentation quality of recurrent SLMs on Chinese, and performs similarly to the recurrent model on English.

## introduction outside of the orthography of english and languages with similar writing systems, natural language is rarely overtly segmented into meaningful units. languages such as chinese, are written with no spaces in between characters, and chinese word segmentation remains an active field of study (e.g. @xcite . running speech is also highly fluent with no meaningful pauses existing between "words" like in orthography. tokenization schemes for large modern language models are now largely passed off to greedy information-theoretic algorithms like byte-pair encoding @xcite and the subsequent sentencepiece @xcite , which create subword vocabularies of a desired size by iteratively joining commonly co-occuring units. however, these segmentations are usually not sensical to human readers @xcite . given the current performance of models using bpe-type tokenization, the nonsensical nature of these segmentations does not necessarily seem to inhibit the success of neural models. nevertheless, bpe does not necessarily help in situations where knowing a sensical segmentation of linguistic-like units is important, such as attempting to model the ways in which children acquire language @xcite , segmenting free-flowing speech @xcite , creating linguistic tools for morphologically complex languages @xcite , or studying the structure of an endangered language with few or no current speakers @xcite . while near-perfect supervised models have been developed for resource-rich languages like chinese, most of the world's languages do not have large corpora of training data @xcite . especially for morphologically complex languages, large datasets containing "gold" segmentations into units like morphemes are very rare. to help mitigate this problem, we propose a novel variant of the unsupervised segmental language model @xcite . segmental language models (slms) function as neural lms that can also be used for unsupervised segmentation correlating with units like words and morphemes @xcite . traditional (recurrent) slms provide a good tradeoff between language-modeling performance and segmentation quality. however, in order to embrace a fully bidirectional modeling context, attention, and the scalability afforded by parallelization, we present a masked segmental language model (mslm), built on a span-masking transformer architecture @xcite . as far as we are aware, we are the first to introduce a non-recurrent architecture for segmental modeling. in this paper, we seek to compare our model to recurrent baselines across two standard wordsegmentation datasets in chinese and english, with the hope of expanding to more languages and domains (such as speech) in future work. we constrain the scope of our work to comparison with recurrent slms both because standard bayesian models have been compared to slms elsewhere @xcite , section 2), and because slms have different use cases from bayesian algorithms, which tend to be weaker language models and lack continuous character representations that are invaluable in settings such as transfer learning. in what follows, we overview baselines in unsupervised segmentation as well as other precursors to slms (section 2), provide a formal characterization of slms in general, as well as the architecture and modeling assumptions that make the mslm distinct (section 3), present our experimental method comparing recurrent and masked slms (section 4), and finally show that the mslm outperforms its recurrent counterpart on chinese segmentation, and performs similarly to the recurrent model on english (sections 5-6). section 7 lays out directions for future work. ## conclusion this study provides strong proof-of-concept for the viability of transfomer-based masked segmental language models as an alternative to recurrent slms in their ability to perform joint language modeling and unsupervised segmentation. mslms the sets used here are also relatively formal orthographic datasets. an eventual use of slms may be in speech segmentation, but a smaller step in that direction could be using phonemic transcript datasets like the brent corpus, also used in @xcite . this set consists of phonemic transcripts of child-directed english speech @xcite . slms could also be applied to the orthographies of more typologically diverse languages, especially ones with complicated systems of morphology (e.g. swahili, turkish, hungarian, finnish). further, though we only test shallow models here, one of the main advantages of transformers is their ability to scale to deep architectures due to their short derivational chains. thus, extending segmental models to "deep" settings would be more feasible using mslms than rslms. lastly, @xcite propose regularization techniques for slms due to low segmentation quality from their "vanilla" models. they report good findings using a character n-gram "lexicon" jointly with expected segment length regularization based on @xcite and @xcite . both techniques are implemented in our codebase, and we have tested them in pilot settings. oddly, neither has given us any gain in performance over our "vanilla" models. a more exhaustive hyperparameter search with these methods may produce a future benefits as well. in conclusion, the present study shows strong potential for the use of mslms. they show particular promise for writing systems with a large inventory of semantic characters (e.g. chinese), and we believe that they could be stable competitors of recurrent models in phonemic-type writing systems given some mitigation of the relative weakness of the positional information available in transformers.