TAMS : Translation-Assisted Morphological Segmentation (2024)

Abstract: Canonical morphological segmentation is the process of analyzing words into the standard ( aka underlying) forms of their constituent morphemes.This is a core task in endangered language documentation, and NLP systems have the potential to dramatically speed up this process. In typical language documentation settings, training data for canonical morpheme segmentation is scarce, making it difficult to train high quality models. However, translation data is often much more abundant, and, in this work, we present a method that attempts to leverage translation data in the canonical segmentation task. We propose a character-level sequence-to-sequence model that incorporates representations of translations obtained from pretrained high-resource monolingual language models as an additional signal. Our model outperforms the baseline in a super-low resource setting but yields mixed results on training splits with more data. Additionally, we find that we can achieve strong performance even without needing difficult-to-obtain word level alignments. While further work is needed to make translations useful in higher-resource settings, our model shows promise in severely resource-constrained settings.

## introduction morphological segmentation is the task of breaking words into morphemes, the smallest semantic units of a language. morphemes can merge and change during word formation, and the precise morphological composition of a word is often obfuscated in its surface form. segmentation can thus take two forms: surface/linear segmentation and canonical segmentation, which divides a word into the "canonical" forms of its morphemes (cf. figure 1 ). one important motivation for automated canonical segmentation is to expedite the process of linguistic analysis, including the creation of interlinear glossed text (igt). igt is a form of morphological annotation that typically adheres to the leipzig glossing format @xcite , a linguistic representation wherein each line of the target text is broken up into a transcription line, a gloss line (morphological annotation), and a translation line. igt is a crucial resource in endangered language documentation work, but it is costly and time-consuming to generate. the task of morphological segmentation is a key component in glossing, and automated canonical segmentation could aid in this process. prior work has shown that automated methods have the potential to assist language documentation and revitalization @xcite @xcite @xcite . neural models have been shown to perform well on the task of canonical segmentation @xcite , but the success of these models has been restricted by the availability of annotated segmentation data. igt, though a limited resource, is one important source of training data for canonical segmentation. until now, primarily the transcription and gloss lines of igt have been used as input to segmentation models, while the translations have been overlooked. moreover, in real-world language documentation settings, translated data is often much more available than morphologically analyzed data, making it practically attractive as an additional input. a typical language documentation pipeline begins with transcription, followed by translation, followed only then by morphological analysis. this com-monly results in a setting where a linguist will have a wealth of translation data but comparatively little morphological segmentation data. here we consider how supervised canonical segmentation methods can be improved by leveraging this underutilized supplemental data source. also, because they are typically into more widely-spoken languages, translations provide the opportunity to make use of pretrained models that likely have much higherquality representations than any model available for the low-resource target language. our work is inspired in part by @xcite who experiment with leveraging translations for the task of automatic interlinear glossing (i.e., generating the gloss line of igt). they use a multi-source word-level transformer to jointly model the transcription and translation sequences, and outperform previous baselines. their work shows promise for the utility of translation data in morphological analysis tasks. however, they assume the presence of well-segmented data and state that "proper segmentation remains a challenge and that the creation of segmentation tools is a valuable endeavor." our work endeavors to address the segmentation issue. we treat canonical segmentation as a sequenceto-sequence problem and use a character-level pointer-generator lstm @xcite to map each surface form word to its segmented, canonicalized form. we experiment with tsez and lezgi, two northeast caucasian languages, and arapaho, a plains algonquian language. we leverage existing sentence-level english translations present in the igt data from the sigmorphon 2023 shared task on interlinear glossing @xcite and create two datasets of word-level transcription-translation alignments: automatically with awesome-align foot_0 @xcite , and manually according to conventions described in §4.3.2. we then embed these translations with bert @xcite and experiment with incorporating them into our baseline model's encoder and decoder. we also analyze the impact of training set size on the efficacy of our approach by limiting our training split to simulate varied levels of resourcedness. finally, we analyze the effect of automatic vs. manual word-alignments on a subset of the tsez data. 2 based on poor automatic alignment performance, we introduce an additional model variant (tams-cls) which incorporates the translation only at the sentence level. we find that although gold alignment does lead to better performance with our approach, we still see improvement over the baseline with automatic alignments in some cases. in the extremely low data setting (@xmath0, our approach outperforms the baseline by an average of 1.99 percentage points and as high as 2.87 points, even with poor quality automatic alignments. in many cases, across training set sizes, tams-cls is the highest performing model configuration, which suggests that we may not even need word-level alignments to achieve performance improvements with tams. these results are promising in the context of the extremely resource constrained language documentation setting. however, in higher data settings, incorporating translations may or may not be beneficial. our results also suggest that aligning translations may not be necessary to see improvements over the baseline. ## related work modeling morphological segmentation recent work on neural methods for canonical segmentation primarily focuses on lstms. @xcite use a bidirectional rnn encoder-decoder with a neural reranker. @xcite adapt this work to the low-resource setting and find that the pointer-generator network vastly improves over the performance of the lstm canonical segmentation model for this setting. recent work has also used the transformer for canonical segmentation: @xcite test several sequence-to-sequence models for the task and find that the transformer performs the best on 4 nguni languages. ## incorporating translations into morphological segmentation models here, we present our method for using translations as a source of additional signal for morphological analysis. we first align the word forms in our target language to relevant word forms in the translation. we then obtain embeddings of these word-forms from a high-resource language model. we describe several approaches for turning these embeddings into a fixed-length representation and for incorporating them as input to the segmentation model. ## data to perform our experiments, we use igt data from the sigmorphon 2023 shared task on interlinear glossing @xcite in three languages: lezgi, tsez, and arapaho. all of the data is licensed under cc by-nc 4.0. each word in this igt format has a surface form, a canonical segmentation, morpheme-level glosses, and an english translation for the sentence the word appears in. an example is shown in figure 2 , which also provides a sample of our manual alignment process. ## experiment we experiment with the strategies described in §3.2 for incorporating information from translations into our morphological segmentation model. we proceed by first tuning our exact approach on the development split of a single language, tsez, by exhaustively considering every combination of encoder, decoder, and cls token translation incorporation strategies. we then apply our topperforming model to the test splits of all three languages. all experiments are performed on nvidia a100 gpus, and all model implementations were based on those provided by yoyodyne. foot_2 ## results and discussion we treat tsez as our development language and perform a search over all translation incorporation strategies using the tsez development set to determine the overall highest performing configuration, which we call tams. this configuration consists of init-state in the encoder, concat-half in the decoder, and cls-concat as the cls strategy. we additionally consider a variation we call tams-cls, which is identical to tams except that it employs the cls-only strategy described in §5.3.1. further details of our translation incorporation strategy search are included in §6.3. ## conclusion we present a novel method for incorporating information from translations into a morphological segmentation model to support low-resource canonical segmentation. using tsez as a development language, we determine our best-performing model (tams), which uses a fixed-length representation of the translation in two ways: to initialize the hidden state in the encoder (init-state) and to concatenate to the input at each time step in the decoder (concat-half). our model is most beneficial in the super low-resource setting (@xmath1, where it outperforms the baseline by 1.99 percentage points on average across three morphologically complex languages. and although we only tune our model on the tsez development set, we also see performance gains for arapaho, a typologically and morphologically distinct polysynthetic language. this promising result suggests that tams could be beneficial for a wide range of languages. we believe tams will be the most beneficial in the language documentation setting, where extreme resource constraints are realistic and often expected. the findings of our tams-cls experiment are especially promising because they indicate that we may be able to see benefits from incorporating translations in canonical segmentation even without word-level alignments. this opens up tams to many more languages which may not have highquality word-level alignments available. however, our results are more mixed in higherresource settings, which indicates that there is still more work to be done to determine whether translations are a valuable addition to canonical segmentation models in cases where more data is available. overall, canonical segmentation for morphologically complex languages remains a challenging task, but we believe that this work indicates that