Beyond Characters: Subword-level Morpheme Segmentation (2022)

Abstract: This paper presents DeepSPIN’s submissions to the SIGMORPHON 2022 Shared Task on Morpheme Segmentation. We make three submissions, all to the word-level subtask. First, we show that entmax-based sparse sequence-tosequence models deliver large improvements over conventional softmax-based models, echoing results from other tasks. Then, we challenge the assumption that models for morphological tasks should be trained at the character level by building a transformer that generates morphemes as sequences of unigram language model-induced subwords. This subword transformer outperforms all of our character-level models and wins the word-level subtask. Although we do not submit an official submission to the sentence-level subtask, we show that this subword-based approach is highly effective there as well.

## introduction nearly all neural models for morphological and phonological nlp tasks operate at the character level. this is a natural design choice because there is usually a monotonic alignment between source and target characters. although often successful, character-level models do not leverage the fact that words contain longer substrings, such as roots and affixes, that can often be copied all at once. they also go against the grain of modern nlp, in which most systems for other tasks are trained on sequences of subword units induced by an unsupervised algorithm, usually either byte-pair encoding (bpe; @xcite or unigram language modeling (ulm; @xcite . although subword units should not be adopted just because they are widespread, they should not be ignored either, especially given the great amount of effort that has gone into integrating morphological inductive biases into subword tokenization @xcite @xcite @xcite . we demonstrate that subword-level modeling does work for morpheme segmentation through our submissions to the sigmorphon 2022 shared task on morpheme segmentation @xcite . our subword-level model, an entmax transformer with sampled ulm tokenizations, outperforms our character-level submissions and wins the word-level subtask. because it generates morphemes as subword sequences, it also offers a way to combine the advantages of subword tokenization (a fixed-size vocabulary, compression) with the advantages of conventional morpheme segmentation (segments do not cross morpheme boundaries). in all, we submit three models to the task: we then retrain deepspin-3 on the combined word-and sentence-level data. although this model is unofficial, it outperforms the winners of the sentence-level subtask for all three languages. ## model in our experiments, we use both attentional lstm @xcite and transformer @xcite sequence-to-sequence models. regardless of those internal details, at time step i the model predicts a next-target-token distribution p θ (• | x, @xmath0. in most sequence-tosequence systems, p θ (• | x, @xmath1, and x and y consist of sequences of characters. in this work, we depart from these defaults by replacing softmax with 1.5entmax @xcite , and by tokenizing into subwords instead of characters. entmax and its loss. language models, including sequence-to-sequence models, produce a normalized probability distribution at teach time step. to do this, they need a function r @xmath2: that is, a function that maps an arbitrary vector of real numbers to a vector in the n-dimensional probability simplex △ n := {@xmath3: @xmath4, 1 ⊤ @xmath5. the standard choice of function is softmax, which is dense: it assigns strictly positive probabilities to all outcomes. however, there is another option, the α-entmax transformation @xcite . entmax, parameterized by a scalar α ≥ 1, computes α-entmax(z) := argmax ## word-level subtask our three submissions to the word-level subtask can be divided into two parts. first, we present character-level lstm-based models trained with cross entropy loss (deepspin-1) and 1.5-entmax loss (deepspin-2). these models are similar to models that performed well at past shared tasks and serve as strong supervised baselines for morpheme segmentation. second, we implement subwordlevel transformer foot_2 models (deepspin-3). ## unofficial sentence-level subtask model although we did not submit to the sentence-level subtask due to time and computation restraints, we were able to train subword-level models similar to deepspin-3 after the conclusion of the task. this system, which we dub deepspin-sent, uses the same hyperparameter grid as deepspin-3. it is trained on the concatenation of data from the wordlevel and sentence-level subtasks. our model does not make use of sentence context: each word in a sentence is presented as a separate example. our results are shown in ## related work given the widely-observed shortcomings of unsupervised subword units for handling morphology @xcite @xcite , several works have attempted to replace these units with a more morphologically-principled representation for downstream tasks. although this sometimes means completely replacing the unsupervised subwords @xcite , other works have adopted a pipeline approach in which unsupervised subwords are applied to a morphological analysis @xcite @xcite @xcite . these techniques are attractive because unsupervised subword techniques are empirically very effective, and removing them entirely risks losing benefits such as their compressive capacity @xcite . although deepspin-3 is similar to these combined approaches, it is not a pipeline: a single neural model predicts both the subword sequence and the location of the morpheme boundaries. ## conclusion we implemented several sequence-to-sequence models for morpheme segmentation, showing that sparse entmax losses outperform cross entropy. our strongest model, which won the word-level subtask, is a transformer that generates morphemes as sequences of subword units, unlike traditional character-level segmentation models.