Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages (2022)

Abstract: We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show that this transfer can be achieved by training over a collection of low-resource languages that are typologically similar (but phylogenetically unrelated) to the target language. In our experiments, we transfer from a collection of 10 Indigenous American languages (AmericasNLP, Mager et al., 2021) to K’iche’, a Mayan language. We compare our multilingual model to a monolingual (from-scratch) baseline, as well as a model pre-trained on Quechua only. We show that the multilingual pre-trained approach yields consistent segmentation quality across target dataset sizes, exceeding the monolingual baseline in 6/10 experimental settings. Our model yields especially strong results at small target sizes, including a zero-shot performance of 20.6 F1. These results have promising implications for low-resource NLP pipelines involving human-like linguistic units, such as the sparse transcription framework proposed by Bird (2020).

## introduction unsupervised sequence segmentation (at the word, morpheme, and phone level) has long been an area of interest in languages without whitespacedelimited orthography (e.g. chinese, @xcite , morphologically complex languages without rule-based morphological analyzers @xcite , and automatically phone-transcribed speech data @xcite , respec-tively. it has been particularly important for lowerresource languages in which there is little or no gold-standard data on which to train supervised models @xcite . in modern neural end-to-end systems, unsupervised segmentation is usually performed via information-theoretic algorithms such as bpe @xcite and sentencepiece @xcite . however, the segmentations they produce are largely non-sensical to humans @xcite . the motivating tasks listed above instead require unsupervised approaches that correlate more closely with human judgements of the boundaries of linguistic units. for example, in a human-in-the-loop framework such as the sparse transcription proposed by @xcite , lexical items are automatically proposed to native speakers for confirmation, and it is important that these candidates be (close to) sensical, recognizable pieces of language. in this paper, we investigate the utility of recent models that have been developed to conduct unsupervised surface morpheme segmentation as a byproduct of a language modeling objective (e.g. @xcite , see section 2). the key idea is that recent breakthroughs in crosslingual language modeling and transfer learning @xcite , inter alia) can be leveraged to facilitate transferring unsupervised segmentation performance to a new target language, using these types of language models. specifically, we investigate the effectiveness of multilingual pre-training in a masked segmental language model @xcite when applied to a low-resource target. we pre-train our model on the ten indigenous languages of the 2021 americasnlp shared task dataset @xcite , and apply it to another low-resource, indigenous, and morphologically complex language of central america: k'iche' (quc), which at least phylogenetically is unrelated to the pre-training languages @xcite . we hypothesize that multilingual pre-training on similar, possibly contact-related languages, will outperform both a monolingual baseline trained from scratch and a model pre-trained on a single language (quechua) with the same amount of pretraining data. we also expect that the pre-trained models will perform increasingly better than the monolingual baseline the smaller the target corpus is. indeed, our experiments show that a pre-trained multilingual model provides stable performance across all dataset sizes and far exceeds the monolingual baseline at low-to-medium target sizes. we additionally show that the multilingual model achieves a zero-shot segmentation performance of 20.6 f1 on the k'iche' data, where the monolingual baseline yields a score of zero. these results suggest that transferring from a multilingual model can greatly assist unsupervised segmentation in very low-resource languages, even those that are morphologically rich. the results also provide evidence for the idea that transfer from multilingual models works at a more moderate scale than is typical for recent crosslingual models (3.15 million parameters for our models). in the following section, we overview work relating to unsupervised segmentation, crosslingual pre-training, and transfer-learning (section 2). we then introduce the multilingual data used in our experiments, and the additional pre-processing we performed to prepare the data for pre-training (section 3). next we provide a brief overview of the type of segmental language model used in our experiments, as well as our multilingual pre-training process (section 4). after this, we describe our experimental process applying the pre-trained and from-scratch models to varying target data sizes (section 5). finally, we discuss the results of our experiments and their significance for low-resource pipelines, both within unsupervised segmentation and for other nlp tasks more generally (sections 6 and 7). ## related work work related to the present study largely falls either into the field of (unsupervised) word segmentation, or the field(s) of crosslingual language modeling and transfer learning. to our knowledge, we are the first to propose a multilingual model for unsu-pervised word/morpheme-segmentation. unsupervised segmentation current state-ofthe-art unsupervised segmentation has largely been achieved with bayesian models such as hierarchical dirichlet processes @xcite and nested pitman-yor @xcite . adaptor grammars @xcite have been successful as well. models such as morfessor @xcite , which are based on minimal description length @xcite are also widely used for unsupervised morphology. as @xcite note, most of these models have weak language modeling ability, being unable to take into account much other than the immediate local context of the sequence. another line of techniques has focused on models that are both strong language models and good for sequence segmentation. many are in some way based on connectionist temporal classification @xcite , and include sleep-wake networks @xcite , segmental rnns @xcite , and segmental language models @xcite @xcite . in this work, we conduct experiments using the masked segmental language model of @xcite , due to its good performance and scalability, the latter usually regarded as an obligatory feature of multilingual models @xcite . crosslingual and transfer learning crosslingual modeling and training has been an especially active area of research following the introduction of language-general encoder-decoders in neural machine translation, offering the possibility of zero-shot translation (i.e. translation for language pairs not seen during training; @xcite . the arrival of crosslingual language model pretraining (xlm, conneau and lample, 2019) further demonstrates that large models pre-trained on multiple languages yield state-of-the-art performance across an abundance of multilingual tasks including zero-shot text classification (e.g. xnli, @xcite , and that pre-trained transformer encoders provide great initializations for mt systems and language models in very lowresource languages. since xlm, numerous studies have attempted to single out which components of crosslingual training contribute to transferability from one language to another (e.g. @xcite . others have questioned the importance of multilingual training, and have instead proposed that even monolingual pre-training can provide effective transfer to new languages @xcite . though some like @xcite have tried to systematically study which aspects of pre-training languages/corpora enable effective transfer, in practice the choice is often driven by availability of data and other ad-hoc factors. currently, large crosslingual successors to xlm such as xlm-r @xcite , mass @xcite , mbart @xcite , and mt5 @xcite have achieved major success, and are the starting point for a large portion of multilingual nlp systems. these models all rely on an enormous amount of parameters and pre-training data, the bulk of which comes from very high-resource languages. in contrast, in this paper we assess whether multilingual pre-training on a suite of very low-resource languages, which combine to yield a moderate amount of unlabeled data, can provide good transfer to similar languages which are also very low-resource. ## data and pre-processing we draw data from three main datasets. we use the americasnlp 2021 open task dataset @xcite to pre-train our multilingual models. the multilingual dataset from @xcite serves as segmentation validation data for our pre-training process in these languages. finally, data from tyers and henderson (2021) is used as the training set for our experiments transferring to k'iche', and @xcite provides the validation and test data for these experiments. ## model and pre-training this section gives an overview of the masked segmental language model (mslm), introduced in @xcite , along with a description of our pre-training procedure. mslms an mslm is a variant of a segmental language model (slm) @xcite @xcite , which takes as input a sequence of characters x and outputs a probability distribution for a sequence of segments y such that the concatenation of y is equivalent to x: π(y) = x. an mslm is composed of a segmental transformer encoder and an lstmbased segment decoder @xcite . see figure 3 . the mslm training objective is based on the prediction of masked-out spans. during a forward pass, the encoder generates an encoding for every position in x, for a segment up to k symbols long; the encoding at position i -1 corresponds to every possible segment that starts at position i. therefore, the encoding approximates p(x i:i+1 , x i:i+2 , ..., to ensure that the encodings are generated based only on the portions of x that are outside of the predicted span, the encoder uses a segmental attention mask @xcite to mask out tokens inside the segment. figure 3 shows an example of such a mask with @xmath0. finally, the segment decoder of an slm determines the probability of the j th character of the segment of y that begins at index i, y i j , using the encoded context: the output of the decoder is not conditional on the determination of other segment boundaries. the probability of y is modeled as the marginal probability over all possible segmentations of x. because directly marginalizing is computationally intractable, the marginal is computed using dynamic programming over a forward-pass lattice. the maximum-probability segmentation is determined by viterbi decoding. the training objective optimizes language-modeling performance, which is measured in bits per character (bpc). pre-training procedure in our experiments, we test the transferability of multilingual and monolingual pre-trained mslms. the multilingual models are trained on the americasnlp 2021 data (see section 3). since slms operate on plain text, we can train the model directly on the multilingual concatenation of this data, and evaluate it by its language modeling performance on the concatenated validation data. as mentioned in section 3, we create two versions of the multilingual pre-trained model: one trained on the full americasnlp set (∼172k lines) and the other trained on the downsampled set, which is the same size as the quechua training set (∼120k lines). we designate these models multi-pt f ull and multi-pt down , respectively. our pre-trained monolingual model is trained on the full quechua set (quechua-pt). each model is an mslm with four encoder layers, hidden size 256, feedforward size 512, and four attention heads. character embeddings are initialized using word2vec @xcite over the training data. the maximum segment size is set to 10. the best model is chosen as the one that minimizes the bits per character (bpc) loss on the validation set. for further pre-training details, see appendix b. to evaluate the effect of pre-training on the segmentation quality for languages within the pretraining set, we also log mcc between the model output and gold-segmented secondary validation sets available in nahuatl and wixarika @xcite , see section 3). figure 4 shows the unsupervised segmentation quality for nahuatl and wixarika almost monotonically increases during pre-training (multi-pt f ull ). ## experiments we evaluate whether multilingual pre-training facilitates effective low-resource transfer learning for unsupervised segmentation. to do this, we pre-train slms on one or all of the americasnlp 2021 languages @xcite and transfer it to a new target language: k'iche' @xcite . k'iche' is a morphologically rich mayan language with several classes of inflectional prefixes and suffixes @xcite . an example sentence can be found in as a baseline, we train a monolingual k'iche' model from scratch. we evaluate performance with respect to the size of the target training set, simulating varying degrees of low-resource setting. to do this, we downsample the k'iche' training set to 8 smaller sizes, for 9 total: {256, 512, ... 2 15 , 47.7k (full)}. for each size, we both train a monolingual baseline and fine-tune the pre-trained models we describe in section 4. 4 orthography kinch'aw ruk' le nunan linguistic segmentation k-in-ch'aw r-uk' le nu-nan translation "i speak with my mother" model input kinch'awruk'lenunan target output k in ch'aw r uk' le nu nan architecture and modeling all models are masked segmental language models (mslms) with the architecture described in section 4. the only difference is that the baseline model is initialized with a character vocabulary only covering the particular k'iche' training set (size-specific). the character vocabulary of the k'iche' data is a subset of the americasnlp vocabulary, so we are able to transfer the multilingual models without changing the embedding and output layers. the quechua vocabulary is not a superset of the k'iche', so we add the missing characters to the quechua model's embedding block before pre-training (these are randomly initialized). the character embeddings for the baseline are initialized using word2vec @xcite on the training set (again, size-specific). evaluation metrics slms can be trained in either a fully unsupervised or "lightly" supervised manner @xcite . in the former case, only the language modeling loss (bits per character, bpc) is used to pick parameters and checkpoints. in the latter, the segmentation quality on gold-segmented validation data can be considered. though our validation set is gold-segmented, we pick the best parameters and checkpoints based on bpc only, simulating the unsupervised case. however, to monitor the change in segmentation quality during training, we also use matthews correlation coefficient (mcc). this measure frames segmentation as a character-wise binary classification task (i.e. boundary vs. no boundary), and measures correlation with the gold segmentation. to make our results comparable with the wider word-segmentation literature, we use the scoring script from the sighan segmentation bakeoff @xcite for our final segmentation f1. for each model and target size, we choose the best checkpoint (by bpc), apply the model to the combined validation and test set, and use the sighan script to score the output. for comparison to the chinese word-segmentation and speech literature, any whitespace segmentation in the validation/test data is discarded before it is fed to the model. however, slms can also be trained to treat spaces like any other character, and thus could be able to take advantage of existing segmentation in the input. we leave this for future work. parameters and trials for our training procedure (both training the baseline from scratch and fine-tuning the pre-trained models) we tune hyperparameters on three of the nine dataset sizes (256, 2048, and full) and choose the optimal parameters by bpc. for each of the other sizes, we directly apply the chosen parameters from the tuned dataset of the closest size (on a log scale). we tune over five learning rates and three encoder dropout values. as in pre-training, we set the maximum segment length to 10. for more details on our training procedure, see appendix b. ## results the results of our k'iche' transfer experiments at various target sizes can be found in interestingly, other than in the zero-shot case, quechua-pt and the comparable multi-pt down perform very similarly to each other. however, the zero-shot transferability of multi-pt down is almost twice that of the model trained on quechua only. multi-pt f ull exceeds both multi-pt down and quechua-pt by a wide margin in every setting. finally, all models show increasing performance until about size 4096, after which more target examples don't provide a large increase in segmentation quality. interpretation these results show that multi-pt f ull provides consistent performance across target sizes as small as 512 examples. even for size 256, there is only a 9% (relative) drop in quality from the next-largest size. further, the pre-trained model's zero-shot performance is impressive given the baseline is effectively 0 f1. on the other hand, the performance of the monolingual baseline at larger sizes seems to suggest that given enough target data, it is better to train a model devoted to the target language only. this is consistent with previous results @xcite . however, it should also be noted that multi-pt f ull never trails the baseline by more than 5.2 f1. one less-intuitive result is the dip in the baseline's performance at sizes 8192 and 2 14 . we believe this discrepancy may be partly explainable by sensitivity to hyperparameters in the baseline. though the best baseline trial at size 2048 exceeds multi-pt f ull by a small margin, the baseline shows large variation in performance across the top-four hyperparameter settings at this size, where multi-pt f ull actually performs better on average and much more consistently ( ## analysis and discussion standing of hypotheses within the framework of unsupervised segmentation, these results provide strong evidence that relevant linguistic patterns can be learned over a collection of low-resource languages, and then transferred to a new language without much (or any) target training data. further, it is shown that the target language need not be (phylogenetically) related to any of the pre-training languages, even though details of morphological structure are ultimately language-specific. the hypothesis that multilingual pre-training yields increasing advantage over a from-scratch baseline at smaller target sizes is also strongly supported. this result is consistent with related work showing this to be a key advantage of the multilingual approach @xcite . the hypothesis that multilingual pre-training also yields better performance than monolingual pre-training given the same amount of data seems to receive mixed support from our experiments. on one hand, the comparable multilingual model has a clear advantage over the quechua model in the zero-shot setting, and outperforms the latter in 5/10 settings more generally. however, because the quechua data lacks several frequent k'iche' characters (and these embeddings remain randomly initialized), it is unclear how much of this advantage comes from the multilingual training per-se. instead, the advantage may be due to the multilingual model's full coverage of the target vocabularyan advantage which may disappear at larger target sizes. further analysis of this hypothesis will require additional investigation. significance the above results, especially the strong zero-shot transferability of segmentation performance, suggest that the type of language model used here learns some abstract linguistic pattern(s) that are generalizable across languages, and even to new ones. it is possible that these generalizations could take the form of abstract stem/affix or word-order patterns, corresponding roughly to the lengths and order of morphosyntactic units. because mslms operate on the character level (and in these languages orthographic characters mostly correspond to phones), it is also possible the model could recognize syllable structure in the data (the ordering of consonants and vowels in human languages is relatively constrained), and learn to segment on syllable boundaries. it is also helpful to remember that we select the training suite and target language to have some characteristics in common that may help facilitate transfer. the americasnlp languages are almost all morphologically rich, with many considered polysynthetic @xcite , a feature that k'iche' shares @xcite . further, all of the languages, including k'iche', are spoken in countries where either spanish or portuguese is the official language, and have very likely had close contact with these iberian languages and borrowed lexical items. finally, the target language family (mayan) has also been shown to have close historical contact with the families of several of the 25.7 ± 0.6 (2.3%) 30.5 ± 2.3 (7.5%) 31.7 ± 0.6 (1.9%) quechua-pt 30.1 ± 0.2 (0.7%) 31.4 ± 0.6 (1.9%) 32.7 ± 0.7 (2.1%) monolingual 4.2 ± 0.5 (11.9%) 36.5 ± 6.8 (18.6%) 44.7 ± 2.0 (4.5%) it is possible that one or several of these shared characteristics facilitates the strong transfer shown here, in both our multilingual and monolingual pretrained models. however, our current study does not conclusively show this to be the case. @xcite show that factors like linguistic similarity and geographic contact are often not as important for transfer success as non-linguistic features such as the raw size of the source dataset. indeed, the fact that our quechua pre-trained model performs similarly to the comparable multilingual model (at least at larger target sizes) suggests that the benefit to using multi-pt f ull could be interpreted as a combined advantage of pre-training data size and target vocabulary coverage. the nuanced question of whether multilingual pre-training itself enables better transfer than monolingual pre-training requires more study. however, taking a more pragmatic point of view, multilingual training can be seen as a methodology to 1) acquire more data than is available from any one language and 2) ensure broader vocabulary overlap with the target language. our characterbased model is of course different from more common word-or subword-based approaches, but with these too, attaining pre-trained embeddings that cover a novel target language is an important step in cross-lingual transfer @xcite @xcite future work we believe some future studies would shed light on the nuances of segmentation transfer-learning. first, pre-training either multilingually or monolingually on languages that are not linguistically similar to the target language could help isolate the advantage given by pre-training on any language data (vs. similar language data). second, we have noted that monolingual pretraining on a language that does not have near-full vocabulary coverage of the target language leaves some embeddings randomly initialized, yielding worse performance at small target sizes. pretraining a model on a single language that happens to have near-complete vocabulary coverage of the target could give a better view of whether multilingual training intrinsically yields advantages, or whether monolingual training is disadvantaged mainly due to this lack of vocabulary coverage. finally, because none of the present authors have any training in the k'iche' language, we are unable to perform a linguistically-informed error analysis of our model's output (e.g. examining the types of words and morphemes which are erroneously (un)segmented, rather than calculating an overall precision and recall for the predicted and true morpheme boundaries, as we do in this study). however, we make all of our model outputs available in our public repository, so that future work may provide a more nuanced analysis of the types of errors unsupervised segmentation models are prone to make. ## conclusion this study has shown that unsupervised sequence segmentation ability can be transferred via multilingual pre-training to a novel target language with little or no target data. the target language also need not be from the same family as a pre-training language for successful transfer. while training a monolingual model from scratch on large amounts of target data results in good segmentation quality, our experiments show that pre-trained models, especially multilingual ones, far exceed the baseline at small target sizes (≤1024), and seem to be much more robust to hyperparameter variation at medium sizes (2048, 8192, 2 14 ). one finding that may have broader implications is that pre-training can be conducted over a set of low-resource languages with some typological or geographic connection to the target, rather than over a crosslingual suite centered around highresource languages like english and other european languages. most modern crosslingual models have huge numbers of parameters (xlm has 570 million, mt5 has up to 13 billion, @xcite , and are trained on enormous amounts of data, usually bolstered by hundreds of gigabytes in the highest-resource languages @xcite . in contrast, our results suggest that effective transfer may be possible at smaller scales, by combining the data of low-resource languages and training moderately-sized, more targeted pre-trained multilingual models (our model has 3.15 million parameters). of course, this study can only support this possibility within the unsupervised segmentation task, so future work will be needed to investigate whether transfer to and from low-resource languages can be extended to other tasks. sizes 256 and 512, and every 128 steps for every other size.