M orfessor-enriched features and multilingual training for canonical morphological segmentation (2022)

Abstract: In our submission to the SIGMORPHON 2022 Shared Task on Morpheme Segmentation, we study whether an unsupervised morphological segmentation method, Morfessor, can help in a supervised setting. Previous research has shown the effectiveness of the approach in semisupervised settings with small amounts of labeled data. The current tasks vary in data size: the amount of word-level annotated training data is much larger, but the amount of sentencelevel annotated training data remains small. Our approach is to pre-segment the input data for a neural sequence-to-sequence model with the unsupervised method. As the unsupervised method can be trained with raw text data, we use Wikipedia to increase the amount of training data. In addition, we train multilingual models for the sentence-level task. The results for the Morfessor-enriched features are mixed, showing benefit for all three sentencelevel tasks but only some of the word-level tasks. The multilingual training yields considerable improvements over the monolingual sentence-level models, but it negates the effect of the enriched features.

## introduction current use of subword segmentation in neural natural language processing (nlp) with unsupervised segmentation methods such as bpe @xcite , sentencepiece @xcite , and morfessor @xcite mainly focuses on finding short and frequent subwords that give good performance in the nlp application, while putting less weight on linguistic correctness. the level of segmentation varies by the frequency of the word: frequent words retain their affixes, while rare words, such as rare proper names, are heavily segmented into syllablelike units or even characters. these methods typically perform surface segmentation, meaning that the subwords can be concatenated back into the surface form of the word without any transformation to account for phonological processes e.g. prof @xmath0. however, when linguistic fidelity is of importance-for example because the segments are analyzed statistically as opposed to using a neural model-a supervised segmentation method may be more suitable. the goal is to output morphemes, the smallest meaning-bearing linguistic units. in canonical morphological segmentation @xcite , instead of segmenting into surface forms of morphemes, the different allomorphs are mapped into a single canonical form, reversing any phonological changes. e.g. prof @xmath1. it is not always possible to give a single correct analysis for any particular surface form. a surface form may be homonymous, with inflections or derivations from two or more lemmas. in order to disambiguate the meanings to choose a single analysis from several alternatives, it is necessary to use the surrounding sentence context. in task 2 of this shared task, such sentence level segmentation is performed. e.g. she rose @xmath2. word-level morpheme segmentation is more widely studied than sentence-level morpheme segmentation. in part, the focus on word level segmentation is due to the historically limited ability of models to exploit all of the available context. with neural sequence to sequence (seq2seq) models, this limitation can easily be lifted. limited availability of labeled data for the sentence level task provides a second reason for the popularity of word-level segmentation. this work presents the auuh (aalto university -university of helsinki) team submission to the sigmorphon 2022 shared task on morpheme segmentation @xcite . in this shared task, the imbalance of training data persists. for the word-level task 1, there is ample training data, ranging from 15 000 labeled words for the lowest resourced language, mongolian, to hundreds of thousands of words for the higher resourced languages. task 1 has between 3 and 30 times as much data as in sentence-level task 2. in addition to the labeled data, an order of magnitude more unlabeled data can easily be sourced. considering that these types of data are available in very different amounts, there is an opportunity to improve especially the sentence-level performance by exploiting the other types of data. in this work, we use large amounts of unlabeled data to enrich the input with features from an unsupervised segmentation model. this feature set augmentation approach, which combines the strengths of generative and discriminative models, has previously been applied for word-level surface segmentation @xcite . additionally, we use the word-level labeled data through multi-task and multi-lingual training. our systems are fully data-driven and languageindependent, requiring no linguistic resources beyond the training data. all the software used in the systems has open-source implementations. ## methods our approach for the shared tasks consists of a neural seq2seq model, enrichment of data with features learned in an unsupervised manner, and multi-task and multilingual training. we submitted six different configurations, which we refer to as systems a-f in the following. ## results tables 3 and 4 list the results of tasks 1 and 2 respectively. systems a and b, c and d, and e and f each form comparable pairs, where the former (e.g. system a) uses morfessor-enriched features, and the latter (e.g. system b) is the same system without enriched features. in the result tables, these comparable pairs are separated with horizontal divider lines. some of our systems have the highest score of all shared task participants in specific subcategories of the evaluation. our system b has the highest f 1score (96.31%) and lowest levenshtein distance (1.39) for the english sentence-level task. our system a has the highest f 1 -score (93.23%) for the english word-level evaluation category 001, i.e. compound words without inflectional or derivational affixes. tables 5 and 6 show task 1 results by morphological category, for systems a-b and e-f respectively. for english, russian, and hungarian, the system using the morfessor-enriched features performs better for most categories involving compounding, in particular the 001 category (only compounding). of the languages in this shared task, only hungarian and english vocabularies contain a substantial portion of compound words (17.32% and 6.79% respectively). ## discussion the multilingual model without morfessorenriched features (system b) gives the best results in both tasks for the three languages (ces, eng, mon) for which we trained such a system. when using multilingual training, the morfessor-enriched features are not beneficial. the unsupervised features may be less useful with the increased amount of training data in the multilingual setup, and varying granularities of the unsupervised segmentations for the different languages could confuse the multilingual model. without multilingual training, the results for enriched features are inconclusive for the word-level task, but clearly beneficial for the sentence-level task. the enriched features give better results for 5 languages (ces, eng, rus, mon, hun) in task 1 and all three languages (ces, eng, mon) in task 2. consistent with previous work @xcite , we find that morfessor-features are useful for modeling the boundary between compound parts, which is challenging for supervised discriminative models on their own. except for the corpus weight hyper-parameter of the morfessor model, we did not tune many parameters of the setup, such as thresholds for the words in the wikipedia dumps, different weightings for the corpora, or use of the word frequencies in morfessor training. more extensive optimization could lead to some improvements for the unsupervised features. it would also be possible to use the part of the data, for which the canonical morphemes correspond to surface morphemes as annotations for training semi-supervised morfessor variants @xcite . it is possible that using a different β for the f βscore may result in better tuning. finding the optimal value for β is left for future work. while computationally more burdensome, instead of searching for the best f β -score of emma-2 for morfessor's output, some parameters could also be optimized on the results of the final seq2seq model. ## conclusions we find that morfessor-enriched features are beneficial for the sentence-level tasks, but see mixed results for the word-level tasks. the multilingual training yields considerable improvements for both tasks, but it negates the effect of the enriched features.