Word-level Morpheme segmentation using Transformer neural network (2022)

Abstract: This paper presents the submission of team NUM DI to the SIGMORPHON 2022 Task on Morpheme Segmentation Part 1, word-level morpheme segmentation. We explore the transformer neural network approach to the shared task. We develop monolingual models for world-level morpheme segmentation and focus on improving the model by using various training strategies to improve accuracy and generalization across languages.

## introduction morphological analysis is the heart of nearly all natural language processing tasks, such as sentiment analysis, machine translation, information retrieval, etc. such natural language processing tasks become infeasible without any morphological analysis. one reason is the sparsity resulting from a high number of word forms that introduce out-of-vocabulary (oov). morphological segmentation is a way to deal with language sparsity by introducing the standard segments within the words rather than dealing with word forms (having multiple morphemes). morpheme segmentation is a type of morphological analysis in which words are divided into surface forms of morphemes, for example, @xmath0. automated morpheme segmentation was studied in the early years of natural language development (nlp). however, significant progress has been made in recent years in using various machine learning techniques. since morphemes are the smallest meaningful language units, information about the morphemic structure of words is already used in various nlp applications and additional tasks, including machine translation and recognition of semantically related words (cognates). in this paper, we proposes a supervised method for word-level morphological segmentation using a transformer neural network. the task of machine translation has seen significant progress in recent times with the advent of transformer-based models @xcite for this year's sigmor-phon 2022 shared task on morpheme segemntation @xcite which at the word level, participants will be asked to segment a given word into a sequence of morphemes. input words contains all types of word forms: root words, derived words, inflected words, and compound words. however, to the best of our knowledge, there has not been work that applies such morpheme segmentation transformer-based models. the paper is organized as follows: section 2 addresses the related work on supervised morpheme segmentation, section 3 describes the data used in training, section 5 describes the model architecture, and section 6 presents the experiment results. ## data a dataset for this task, the organizer integrated all basic types of morphological databases (including unimorph @xcite @xcite -inflectional morphology; morphynet @xcite -derivational morphology; universal dependencies @xcite and ten editions of wiktionary -compound morphology and root words) cover 9 languages. 8 of these languages were available initially, while 1 surprise language, mongolia, was released one week before the submission deadline. each language had split a train and a development sample. the amount of data for the different languages vary in size, from 18966 (mongolian) to 926098 (hungarian). each sample occupies a single line and consists of input word, the corresponding morpheme sequence, and the corresponding morphological category. except for spanish, eight languages have morphological word categories shown in ## model architecture we use the character level transformer implementation of fairseq @xcite . our model is composed of one encoder input word, and one decoder output segmentation of the word. we train a monolingual word segmentation model for each given language with identical parameters, 50 epochs, 1 encoder layer, 1 decoder layer, 0.0001 learning rate, using the adam optimizer (kingma and ba, 2014) and the cross-entropy loss. various hyperparameters of our transformer model were experimentally tested in several experiments. the resulted model has the encoder and decoder layer with 128 hidden units, and the batch size is 32. encoder and decoder more layers slightly improve the quality (less than 0.5%), but the model became too heavy both for training and evaluation. we also use created checkpoints to save the checkpoint the latest and the best ones. it is also a safe guard in case the training gets disrupted due to some unforeseen issue. ## conclusion we have presented the monolingual models for morpheme segmentation in 9 languages. our model run outperforms the baseline. even though our models as implemented prior to submission failed to attain reasonable evaluations scores on the word-level morpheme segmentation task, our results indicate that our model has the potential to have a better performance after fine-tuning and the good performance of our model under varying morphological complexity languages. in future work, we plan on exploring multilingual word-level morpheme segmentation a model.