Weakly Supervised Word Segmentation for Computational Language Documentation (2022)

Abstract: Word and morpheme segmentation are fundamental steps of language documentation as they allow to discover lexical units in a language for which the lexicon is unknown. However, in most language documentation scenarios, linguists do not start from a blank page: they may already have a pre-existing dictionary or have initiated manual segmentation of a small part of their data. This paper studies how such a weak supervision can be taken advantage of in Bayesian non-parametric models of segmentation. Our experiments on two very low resource languages (Mboshi and Japhug), whose documentation is still in progress, show that weak supervision can be beneficial to the segmentation quality. In addition, we investigate an incremental learning scenario where manual segmentations are provided in a sequential manner. This work opens the way for interactive annotation tools for documentary linguists.

## introduction recent years have witnessed a blooming of research aimed at applying language technologies (lts) to "under-resourced languages". foot_0 such studies have been mostly motivated on three main grounds (not necessarily mutually exclusive): (a) to develop tools that could speed up the work of field linguists collecting and annotating recordings for these languages; (b) to provide linguistic communities with lts that are necessary in an increasingly digitalised world, e.g. to interact with smartphones or computers in their own language and communicate with speakers of other languages; (c) to challenge existing machine-learning techniques in very low resource settings, where hardly any resource (dictionary, corpus, grammar) is available. those objectives are thoroughly discussed in a recent position paper @xcite who notices, among other things, that objective (c) (training language processing tools with zero resource) is questionable in the context of language documentation works which can often rely on some pre-existing knowledge, such as a word list, or information from related languages. accordingly, this paper explores ways to make the best of prior resources and improve the effectiveness of unsupervised language analysis techniques for the purpose of linguistic documentation. our main objective is to develop tools that will effectively assist field linguists in their documentary tasks (objective (a)). we focus on segmentation tasks, which aim to automatically identify meaningful units in an unsegmented phonetic or orthographic string @xcite @xcite @xcite . following these authors, we experiment with bayesian non-parametric segmentation models, derived in our case from @xcite and subsequent work, which we recap in section 2. our first contribution is in section 3 which studies multiple semi-supervised learning regimes aimed to take advantage of pre-existing linguistic material such as incomplete segmentations and word lists. in sections 4 and 5, we experimentally assess the pros and cons of these weakly supervised approaches in batch and online learning, for two extremely low-resource languages currently in the process of being documented: mboshi, a bantu language used in former studies @xcite ; and japhug, a language from the sino-tibetan family spoken in the western part of china thoroughly documented by @xcite . these two languages were selected because they illustrate actual documentation processes, for which high-quality linguistic resources have been derived from fieldwork, at the end of a long and difficult procedure @xcite . a complementary analysis follows, where we use the japhug corpus to take a closer look at the units identified automatically, contrasting morpheme-based and word-based supervision. ## background going from audio recordings to fully annotated transcripts implies two successive segmentation steps: the first segments words and happens during the production of phonemic or orthographic transcripts; the second further splits words into morphs, which are then annotated with syntactic information and glosses. we mostly focus on the former task, assuming a two-step process: first, the computation of a phonemic transcript that we assume is given; then the segmentation into words for which we consider two settings: batch and online learning. the word and morpheme segmentation tasks are closely related and rely on similar tools: using the japhug corpus, which contains both levels of segmentations, we also study the implications of using lists of words vs morphemes as weak supervision. in its baseline form, the word segmentation process is fully unsupervised, and the only training material is a set of transcribed sentences (see fig. 1 ). we rely on bayesian non-parametric approaches to word segmentation (see @xcite for a thorough exposition), and our baselines are the unigram version of the dpseg model @xcite and a variant where the underlying dirichlet process is replaced by a pitman-yor process as in @xcite . we selected unigram models for their simplicity, which (a) makes them amenable to the processing of very small sets of sentences; (b) makes the online learning setting tractable. while using higher-order models or more sophisticated models of the same family @xcite may improve the performance (see @xcite for an experimental comparison), we believe that in our low-resource conditions, these variations would be small foot_1 and would not change our main conclusions. word segmentation models fundamentally rely on probabilistic models for word sequences defining p (@xmath0. . . w t ); word sequences can also be viewed as segmented sequences of characters @xmath1. . . y l , so that the same model can be used for the joint probability of (y, b), with @xmath2. . . b l representing the vector of boundary locations where value b @xmath3. b @xmath4. no boundary) after symbol y t . in an unsupervised setting, these boundaries are hidden and are latent variables in the model. such models lend themselves well to gibbs sampling, which repeatedly produces samples of each boundary given all the other boundaries in the corpus. in dpseg, the underlying sequence model is a unigram model: p (w 1 . . . w t ) = t @xmath5. the probability of individual words corresponds to a dirichlet process with parameters α, the concentration parameter, and p 0 , the base distribution, and yields the following formulation for the conditional probability of w t given the past words @xmath6: an extension of dpseg, denoted pypseg, uses a pitman-yor process (pyp) instead of the dirichlet process and generalises equation (1) with an additional discount parameter, which enables to better control the generation of new words. pyps are introduced in @xcite ; a fast implementation is in @xcite . for our experiments, both models have figure 1 : the sentence segmentation task illustrated with a sentence from the mboshi corpus: 'báa ámikúndá poo yá kala' ('they found the old village'). the two possible segmentations only differ in one boundary at position @xmath7, one (w 6=0 ) where 'ámikúndá' is one single unit and one (w 6=1 ) where it is split in two. ## supervising word segmentation in this section, we discuss realistic sources of weak supervision for segmentation tasks and how they can be included in bayesian models. ## results this section presents the results for the models presented above. we also report the performance of sentencepiece, another word segmentation tool based on a unigram language model (kudo, 2018): 7 to boost this baseline, the vocabulary size has been set to the reference number of n type (cf. ## related work unsupervised segmentation is a generic nlp task that can be performed at multiple levels of analysis: a document segmented in sections, a speech segmented in utterances, an utterance segmented in words, a word segmented in morphemes, syllables or phonemes. it has been studied in multiple ways, and we report here recent work related to word discovery for language documentation, noting that the same methods also apply to the unsupervised segmentation of continuous speech into 'words' (de @xcite which has given rise to a vast literature on language acquisition. recently, this task has become central in preprocessing pipelines, with new implementations of simple models @xcite . linear segmentation models in the bayesian realm can be traced back to @xcite . they were extended with nesting in @xcite , where the base distribution of the dirichlet process is a char-based nonparametric model; and in @xcite , who consider hidden state variables in the word generation process. this extension enables, for instance, to jointly learn segmentation and pos tagging or to introduce some morphotactics in the model. other sources of weak supervisions along these lines concern the use of higher-order n-grams and of prosodic cues @xcite . finally, @xcite (with particle filtering techniques) and @xcite (with block sampling) study ways to speed up inference. the unsupervised techniques exposed in section 2 only depend on the design of a probabilistic word generation process. this means that they are also readily applicable when this process is conditioned to some input, for instance, when a translation is available as an additional information source. this setup is notably studied in @xcite , and also considered, with radically different tools, in @xcite . a somewhat richer trend of works aimed at informing word segmentation relies on the model of adaptor grammars (ag) of @xcite , applied to the segmentation task as early as @xcite . ags generalise finite-state models such as dpseg and pypseg by modelling trees and subtrees, rather than mere strings. their use necessitates a context-free description of the language, which enables to integrate information regarding word and syllable structures. even generic descriptions can be useful, but finding the most appropriate and effective one is challenging @xcite . this formalism has also been used to introduce syntactic information @xcite , prosodic information @xcite , and partial annotations @xcite . recent software packages for ags are presented in @xcite and @xcite . using ags comes, however, with a high computational price, as the gibbs sampling process typically requires repeated parses of the corpus, even though cheaper estimation techniques may also be considered @xcite . as our goal is to integrate learning techniques in interactive annotation tools, ags were not deemed appropriate, and we explored simpler alternatives. similar arguments apply to the use of neural networks, which have attracted a growing interest even for very low-resource languages, combining supervised segmentation methods @xcite with cross-lingual transfer or data augmentation techniques @xcite @xcite . ## conclusion and outlook in this work, we have studied various ways to use weak supervision for automatic word segmentation. in language documentation scenarios, such supervision is often available, taking the form of a partial annotation or word lists. bayesian non-parametric models lend themselves well to this setting, and our experiments have shown that two variants of a simple unigram model were getting a substantial boost from weak supervision, a result that has been obtained with two languages currently being documented. the most effective approach seems to start with a small set of fully segmented data, which helps learning in two ways: as a training signal for segmentation and as lexical prior for the base distribution. based on this observation, we have further evaluated the longer-term benefits of an incremental training regime and also contrasted the improvement obtained using a word-based vs a morpheme-based vocabulary list. our future work will continue to explore the interplay between word and morpheme segmentations, as both are required in actual documentation settings, possibly extending our analyses on additional languages. we will also consider supervising the annotation process with lists of non-inflected forms, which requires to jointly learn inflectional patterns and segmentation. finally, our main objective remains to integrate these techniques into an annotation platform and evaluate how much they help speed up the annotation process, hence the need to control the run-time of our algorithms. kei uchiumi, hiroshi tsukahara, and daichi mochihashi. 2015. inducing word and part-of-speech with pitman-yor hidden semi-markov models. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1774-1782, beijing, china. association for computational linguistics.