The SIGMORPHON 2022 Shared Task on Morpheme Segmentation (2022)

Abstract: The SIGMORPHON 2022 shared task on morpheme segmentation challenged systems to decompose a word into a sequence of morphemes and covered most types of morphology: compounds, derivations, and inflections. Subtask 1, word-level morpheme segmentation, covered 5 million words in 9 languages (Czech, English, Spanish, Hungarian, French, Italian, Russian, Latin, Mongolian) and received 13 system submissions from 7 teams and the best system averaged 97.29% F1 score across all languages, ranging English (93.84%) to Latin (99.38%). Subtask 2, sentence-level morpheme segmentation, covered 18,735 sentences in 3 languages (Czech, English, Mongolian), received 10 system submissions from 3 teams, and the best systems outperformed all three state-of-the-art subword tokenization methods (BPE, ULM, Morfessor2) by 30.71% absolute. To facilitate error analysis and support any type of future studies, we released all system predictions, the evaluation script, and all gold standard datasets.

## introduction many nlp applications, such as machine translation or question answering, require subword tokenization, i.e. splitting words into a sequence of substrings @xcite . such tokenizers are trained by an unsupervised algorithm, usually either byte-pair encoding (bpe; @xcite or unigram language modeling (ulm; @xcite . to give a few examples, contemporary language models roberta @xcite and gpt-3 @xcite use a byte-level bpe @xcite while xlnet @xcite relies on ulm. these subword tokenization algorithms are not linguistically motivated but are rather based on statistical co-occurrences. therefore, unsupervised and semisupervised methods for morphological segmenta- system type motivation segmentation bpe surface sta. in | val | uable morfessor2 surface sta. & lin. in | valuable deepspin-3 canonical sta. & lin. in | value | able however, state-of-the-art studies in canonical segmentations have been limited to very low num-lang word segmentation category eng sheepiness sheep @@y @@ness 010 pokers poke @@er @@s 110 hun időpontod idő @@pont @@od 101 szőttetek sző @@tt @@etek 100 mon харах харах 000 гэмтлийг гэмтэх @@л @@ийг 110 ## data we collected our morphological data from various sources to account for all types of morphology: derivational, inflectional, compounding. we also collected base forms. for derivational and inflectional morphology, we have used the segmentation data from unimorph 4.0 @xcite and morphynet @xcite . unimorph contains inflectional paradigms collected from linguistic sources as well as wiktionary, while mor-phynet represents derivations scraped from various editions of wiktionary. compounds and base forms were also extracted from wiktionary (see section 3.2 for more details on the data extraction). we then used the data to produce morpheme segmentations for seven high-resource languages. for czech and mongolian, as low-resource languages, we asked native speakers and linguists to develop the resources (section 3.3 provides more details). for english sentence data, we have used the universal dependency treebank of english @xcite . ## baseline systems the shared task provided predictions and results of baseline systems to participants that covered all languages and both subtasks. we chose three baseline systems: first is wordpiece, one of the state-of-the-art subword tokenization algorithms used in bert @xcite , which is based on @xcite and somewhat resembles bpe @xcite . second is ulm (unigram language model @xcite ), another popular subword tokenization, used in xlnet @xcite . third is morfessor2, one of the state-of-the-art unsupervised morphological segmentations @xcite . in future shared tasks, we aim to include more state-of-the-art tokenization tools including other morfessor variants @xcite @xcite , bpe-dropout @xcite , dynamic programming encoding (dpe) @xcite or its variant @xcite , multi-view subword regularization @xcite , charformer @xcite , space-treatment variants of bpe and ulm @xcite . ## system descriptions the sigmorphon 2022 shared task on morpheme segmentation received submissions from 7 teams with members from 10 universities and institutes. many teams submitted more than one system while some focused on a specific set of languages like romance. in total, we had 24 unique systems over two subtasks, including the baseline system. more system details can be seen in auuh researchers at the aalto university and the university of helsinki produced six submission systems: two were transformer models and four were bidirectional gru models created with several innovations of morfessor feature enrichment, multi-task learning, and multilingual learning. morfessor @xcite is the famous language-independent unsupervised and semi-supervised segmentation tool and has a big family of morfessor variants @xcite @xcite . they have used the first variant of morfessor @xcite for enriching input words along with their morfessor subword segmentations. auuh_a, auuh_c, aauh_e systems used this morfessor-based feature enrichment. the key innovation of auuh cluzh researchers at the university of zurich ensembled four submissions @xcite by extending their previous neural hard-attention transducer models @xcite . for subtask 1, they submit the following strong ensemble cluzh composed of 3 models without encoder dropout and 2 models with encoder dropout of 0.15. in the sentence-level subtask 2, they submitted three ensembles, and treated this problem as the word-level problem by tokenizing sentences into words. they have also used pos tags as additional features to provide a light for the context of words. all individual models have an encoder dropout probability of 0.25 and vary only in their use of features: cluzh-1 with 3 models without pos features, cluzh-2 with 3 models with pos tag features, and cluzh-3 with combined all the models from cluzh-1 and cluzh-2. in overall, the cluzh-3 system was the subtask 2 winning system (by winning two out of three languages) and in subtask 1 cluzh was the only system, outranked one (deepspin-1) of three deepspin systems. deepspin researchers submitted three neural seq2seq models: (1) deepspin-1, a characterlevel lstm with soft attention @xcite with softmax trained with cross-entropy loss; gu one team from georgetown university produced two submissions for three romance languages of the word-level subtask, based on the gru-based encoder-decoder model @xcite . in initial attempts, they tried to use additional features from the wiktionary lists of prefixes and suf- in french, the training data were augmented with four morphological category data from italian and spanish training and development datasets. these categories include non-inflection categories of 000, 001, 010, 011. with these experiments, they made minor improvements to these three languages. for these results, more research is needed to understand that transfer learning is useful. num di a single submission from the national university of mongolia @xcite is a transformer-based neural model. their model architecture is simple as single-layered encoder-decoder classic architecture. all the hyperparameter settings are same as fairseq's standard tutorial tool. their submission is also limited by four languages of subtask 1 due to human error. ## the system results all system results can be found and downloaded from the shared task github page. foot_4 ## future directions the submitted systems achieved unexpectedly high accuracy across nine languages. this result suggests that the neural systems may have more capabilities beyond segmenting morphemes. for the next year, we plan to modify the task design and enrich the dataset with more fine-grained analysis. for example, @xmath1, @@ is derivation, and ## is inflection. in another direction, we will explore possibilities of adapting other morphological resources including word-formation resources @xcite @xcite or segmentation resources, unisegments @xcite . our shared task team welcomes continued contributions from the community. ## conclusion the sigmorphon 2022 shared task on morpheme segmentation significantly expanded the problem of morphological segmentation, making it more linguistically plausible. in this task, seven teams submitted 23 systems for two subtasks in total of nine languages, achieving at minimum f1 30.71 improvement over the three baselines of the state-of-the-art subword tokenization and morphological segmentation tools, being used to train large language models, e.g., xlnet @xcite . the results suggest many directions for improving morpheme segmentation shared task.