Automatic Interlinear Glossing for Under-Resourced Languages Leveraging Translations (2020)

Abstract: Interlinear Glossed Text (IGT) is a widely used format for encoding linguistic information in language documentation projects and scholarly papers. Manual production of IGT takes time and requires linguistic expertise. We attempt to address this issue by creating automatic glossing models, using modern multi-source neural models that additionally leverage easy-to-collect translations. We further explore cross-lingual transfer and a simple output length control mechanism, further refining our models. Evaluated on three challenging low-resource scenarios, our approach significantly outperforms a recent, state-of-the-art baseline, particularly improving on overall accuracy as well as lemma and tag recall.

## introduction the under-documentation of endangered languages is an imminent problem for the linguistic community. of the 7,000 languages in the world, an estimated 50% will face extinction in the coming decades, while around 35-42% still remain substantially undocumented @xcite . perhaps these numbers are no surprise when one acknowledges the difficulty of language documentation -fieldwork demands time, cultural understanding, linguistic expertise, and financial support among a myriad of other factors. consequently, an important task for the linguistic community is to facilitate the otherwise daunting documentation process as much as possible. documentation is not a cure-all for language loss, but it is an important part of language preservation; even in an unfortunate worst-case scenario where a language does disappear, a permanent record of the language is saved for posterity, and could hopefully be useful for revitalization purposes. documentation written for a language a in a language b necessarily involves careful selection of example data from language a that illustrate various aspects of language a's grammar. these data are often provided in the form of a typical semi-structured format known as interlinear glossed text (igt). igt consists of three lines: the first line being the original data from language a, the third line its translation into language b, and the line in between the other lines being a morpheme-by-morpheme gloss, i.e. annotation of the language a data. the provision of this second line illustrates the morphological structure of language a to the reader, who is not necessarily informed on the grammar of language a. an example is outlined in figure 1 . igt helps linguists better understand how other languages work without prior knowledge of those languages or their grammars. however, manual segmentation and igt annotation is still a time-and labor-consuming work, as it requires linguistic training. therefore, language documentation projects typically manifest a yawning gap between the amount of material recorded and archived and the amount of data that is thoroughly analyzed with morphological segmentation and gloss @xcite . this gap can be filled using automatic approaches, which could at least accelerate the annotation process by providing high-quality first-pass annotations. previous approaches to automatic gloss generation include manual rule crafting and deep rule-based analysis @xcite , treating the glossing task as a classification problem focusing only on the morphological tags @xcite and requiring a lexicon for stems @xcite , and using models based on conditional random fields (crf) integrated with translation and pos-tagging information (mcmillan-major, 2020). in contrast, our approach is, to our knowledge, the first one to show that modern neural systems are a viable solution for the automatic glossing task, without requiring any additional components or making unrealistic assumptions regarding data or nlp tool availability for low-resource languages. we rely on the observation that parallel corpora with transcription and translation are likely to be available for many low-resource languages, since the knowledge of the two languages is sufficient for translating the corpus without the need of linguistic training. documentation approaches relying on parallel audio collection @xcite are in fact already underway in the americas @xcite and africa @xcite , among other places. an additional advantage of parallel corpora is that they contain rich information that can be beneficial for gloss generation. as figure 1 outlines, the stems/lemmas in the analysis are often hiding in the translation, while the grammatical tags could be derived from the segments in the transcription. we hypothesize that the information from the translation can further ground the gloss generation, and especially allow a system that properly takes into account to generalize to produce lemmas or stems unseen during training. in this work we propose an automated system which creates the hard-to-obtain gloss from an easyto-obtain parallel corpus. we use deep neural models which have driven recent impressive advances in all facets of modern natural language processing (nlp). our model for automatic gloss generation uses multi-source transformer models, combining information from the transcription and the translation, significantly outperforming previous state-of-the-art results on three challenging datasets in lezgian, tsez, and arapaho @xcite @xcite . importantly, our approach does not rely on any additional annotations other than plain transcription and translation, also making no assumptions about the gloss tag space. we further extend our training recipes to include necessary improvements that deal with data paucity (utilizing cross-lingual transfer from similar languages) and with the specific characteristics of the glossing task (presenting solutions for output length control). our contributions are three-fold: ## interlinear glossed text interlinear glossed text (igt) is the name for a format commonly used by linguists in presenting linguistic data. in addition to providing the original sentence in one language (called the "object language") and a free translation into another language (which renders the utterance interpretable by a broader audience), a morpheme-by-morpheme annotation (called the "gloss") of the original sentence is presented between the object language line and the translation line -hence the name "interlinear gloss". ex. 1 shows an example of igt for a sentence from our tsez dataset. lexical morphemes such as open-class words and their stems are simply glossed as their translations. on the other hand, functional morphemes such as inflectional affixes and closed-class words are glossed with the grammatical categories or function that they encode. glosses for lexical morphemes are referred to as stems and those for functional morphemes are referred to as grams. in ex. 1, the verb bok'ek'asi consists of the class i plural prefix b-, the stem ok'ek' and the resultative participle suffix -asi. the stem, being a lexical morpheme, is glossed as its english translation steal. on the other hand, the prefix and the suffix are functional morphemes. thus they are glossed as the collections of grammatical categories they encode. the prefix bis glossed as the combination of i (for class i -tsez has four noun classes i-iv) and pl (for plural), and the suffix is glossed as res.prt (for resultative participle). when a morpheme is glossed as a collection of multiple grams, they are separated by periods or some other delimiting punctuation. ## evaluating gloss generation the characteristics of gloss generation require special care rather than blindly using metrics established for other tasks like machine translation. previous work uses: comparing all these metrics on our results, we find that unsurprisingly they heavily correlate with each other. we measure correlation with spearman's rank coefficient, and all correlations end up with coefficients higher than 0.85 (@xmath0.0001). ## experimental settings languages we use three languages as the testbed for evaluating our approach: arapaho, lezgian and tsez. they provide challenging scenarios with different amounts of training data available overall (details are listed in arapaho is an algonquian language spoken in and around wyoming and western oklahoma, u.s. with only around 250 fluent speakers, it is very much an endangered language that requires immediate attention and effort for documentation and preservation. being highly polysynthetic, arapaho makes heavy use of verb incorporation with morphemes for tense, aspect, modality, evidentiality, and adverbial information, whose (de)incorporation is determined by pragmatic factors such as saliency and emphasis @xcite . we use the corpus compiled by @xcite , which contains more than 20,000 glossed sentences with english translations. lezgian is a member of the lezgic branch of the nakho-daghestanian language family, also known as the (north)east caucasian family. it is spoken by about 400,000 speakers in southern dagestan and northern azerbaijan in the eastern caucasus. it has a rich consonant inventory with 54 members and an agglutinative morphology. the language has 18 nominal cases, features case-stacking, head-final syntax, ergative-absolutive alignment and no noun-verb agreement @xcite . we use the corpus compiled by arkhangelskiy (2012), comprised of slightly over 1,000 igt examples, with free translations in english. tsez (dido) is a member of the tsezic group of the nakho-daghestanian language family -it is a close relative of lezgian.spoken by 12,467 speakers in dagestan according to a 2010 russian census, tsez also features head-final syntax, ergative-absolutive alignment, rich suffixing morphology, an impressive inventory of cases and a variety of strategies for converb formation @xcite . we use the tsez annotated corpus compiled by abdulaev and abdullaev (2010) which includes almost 1,800 glossed utterances with translations in both russian and english. we report results using the english translations, since the analysis is also using english stems, and as a result the performance using the russian translations was worse in preliminary experiments. word-level vs. morpheme-level settings for each of the language datasets, we have access to gold (hand-created) morpheme-level gloss annotations mapped to morphologically segmented transcriptions. this allows us to evaluate models under two distinct settings, for which we report results separately: transliteration cross-lingual training between typologically related languages has shown promising results in several nlp tasks especially in low-resource settings @xcite . two of our evaluation languages, namely lezgian and tsez are fairly similar as they are both members of the nakho-daghestanian language family, and as such are ideal for crosslingual transfer. however, @xcite pointed out that cross-lingual learning can be inversely impeded if the languages do not use the same script even if they are closely genealogically related languages. lezgian is written in cyrillic script while tsez is written in latin script. to maximally exploit the power of cross-lingual training, we transliterated lezgian from cyrillic script to latin script, and transliterated tsez from latin script to cyrillic script. 5 with the original and the transliterated versions of the training data at hand, we combine them during training into a single training set for the language transfer model. the evaluation is of course performed on the original test sets with the original corresponding scripts. ## results and analysis a summary of our main results on the three languages in the realistic word-level scenario (without gold segmentation) is outlined in our models significantly outperform the baseline in every evaluation metric in all datasets. the results in the subsampled arapaho dataset are less conclusive (the baseline achieves slightly higher bleu and lower wer) but both models in this very challenging setting are quite bad to begin with; with an order of magnitude more data, our model's outputs are exceptional, while the baseline was unable to be trained. in both tsez and arapaho, the multi-source approach yields significantly better performance on both lemma and grammatical tag generation than a single-source model. in lezgian, however, we don't observe any substantial differences between the two models, which we suspect is because the very limited training data may not be sufficient for the model to learn a good gloss-translation cross-attention. 10 nevertheless, the improvements in tsez and arapaho indicate that our model is indeed able to leverage the information provided by the translation to further aid in automatic glossing. ## related work several works have studied the automated igt generation task @xcite @xcite @xcite . they mainly used machine learning methods such as crf and svm to generate gloss and proposed a series of heuristic post-editing algorithms to improve the performance. among them, @xcite , @xcite combined machine labeling and active learning for creating igt. @xcite tested lstms to predict the morphological labels within glosses, but underperformed against crf models in that task. mcmillan-major (2020) exploited parallel information in gloss generation. these models either depended on an assumption of only a finite set of possible morphological tags, or required additional linguistic information and feature engineering such as pos tags and word alignment. ## conclusion in this work, we make the initial attempts to leverage neural-based models with dual sources -source transcription and its translation -for the automatic gloss generation task. we further extend our model with cross-lingual transfer to overcome data paucity, and with simple techniques to adapt to the characteristics of the glossing task. our multi-source transformer-based model significantly outperforms previous work on various evaluation metrics in three low-resource languages. we also hold both qualitative and quantitative error analysis. future research directions include (i) combining a multi-source model and a multi-task model together by taking both word-level transcription and translation as input, to generate morphologically segmented transcription and morpheme-level gloss, (ii) adding additional structural biases for attention, and (iii) data augmentation algorithms and heuristic post-editing methods.