  Segmentation remains an important preprocessing step both in languages where
"words" or other important syntactic/semantic units (like morphemes) are not
clearly delineated by white space, as well as when dealing with continuous
speech data, where there is often no meaningful pause between words.
Near-perfect supervised methods have been developed for use in resource-rich
languages such as Chinese, but many of the world's languages are both
morphologically complex, and have no large dataset of "gold" segmentations into
meaningful units. To solve this problem, we propose a new type of Segmental
Language Model (Sun and Deng, 2018; Kawakami et al., 2019; Wang et al., 2021)
for use in both unsupervised and lightly supervised segmentation tasks. We
introduce a Masked Segmental Language Model (MSLM) built on a span-masking
transformer architecture, harnessing the power of a bi-directional masked
modeling context and attention. In a series of experiments, our model
consistently outperforms Recurrent SLMs on Chinese (PKU Corpus) in segmentation
quality, and performs similarly to the Recurrent model on English (PTB). We
conclude by discussing the different challenges posed in segmenting
phonemic-type writing systems.
