Glossy Bytes: Neural Glossing using Subword Encoding (2023)

Abstract: This paper presents several different neural subword modelling based approaches to interlinear glossing for seven under-resourced languages as a part of the 2023 SIGMORPHON shared task on interlinear glossing. We experiment with various augmentation and tokenization strategies for both the open and closed tracks of data. We found that while byte-level models may perform well for greater amounts of data, character based approaches remain competitive in their performance in lower resource settings.

## introduction and motivation subword 1 representations can leverage the compositional nature of input words to model the morphology of a language. approaches that treat words as atomic units have limitations when handling morphologically rich languages @xcite , where words may be composed of several meaningful morphemes (which, in turn, are composed of characters). another limitation of the word-level approach is its inability to handle out-of-vocabulary (oov) words. when data is scarce and many test words are absent from the training set, generic oov handling (i.e. <@xmath0. recent strategies for oov handling in neural machine translation include using pre-trained contextualized word embeddings @xcite or exploiting external data @xcite . however, these methods are often domain-specific and may be unrealistic in a truly low-resource setting. in such scenarios, models capable of learning relationships between orthographically similar sequences @xcite may be especially valuable for disambiguating rare and unseen words, as there is often overlap between an oov word (e.g. desktop) and those present in the vocabulary (e.g. desk, top). a drawback @xcite of character-level representations lies in the non-trivial relationship between word forms and their meanings. subword models may represent a compromise between characters, which are semantically void, and word-level representations. indeed, byte-pair encoding (bpe) @xcite can effectively handle rare and unknown words in neural machine translation, particularly when a word-level translation may be derived from the translation of word-internal units. throughout this paper, we examine several approaches to neural interlinear glossing, and our contributions are as follows: additionally, we propose that sequence-tosequence (seq2seq) models are a viable approach for automated gloss generation in low-resource scenarios even for the closed-track task, where systems are trained exclusively on unsegmented input sentences and glosses. ## related work given the data-hungry nature of neural systems, many approaches for automating low-resource igt generation @xcite mcmillan-major, 2020) have been statistical, treating gloss generation as a token classification task where each morphologically segmented source line is mapped to its corresponding morphosyntactic descriptor (msd). as crfs cannot encode variable-length sequences, they do not extend to the closed-task setting. the baseline @xcite for this task uses a bert-based model to label each whitespaceseparated sequence with its corresponding glossed unit. this choice in architecture is motivated by the scarcity of training data and fails to exploit orthographic regularities which lend consistent clues to the internal structures of morphologically rich grammars. in a recent neural approach to automated gloss generation for lezgi, tsez, and arapaho, zhao et al. (2020) experimented with both word and byte-pair tokenization. while they noted that the subword model outperformed the word-level model for all languages but lezgi, they did not systematically analyze each approach. ## data the data for this shared task comes from seven lowresource languages from various language families. some languages in the set include a large number of training examples, while others contain very few. 2 all languages have original texts (orthographic representations) and gold-standard glosses. some languages also have translated lines of text (in either english or spanish). for the open track, all languages except nyangbo have morphologically segmented lines, and uspanteko has pos annotations. the format of the data was as follows for the closed track: ## conclusion in this paper, we explored the potential of using subword representations in grammatical glossgenerating models to 'learn' the morphological patterns of a low-resource language. at a byte-pair level, we found this strategy to be competitive but dependent on the amount of training data available. as such, the byte-pair tokenized model performed the best for arapaho (the dataset with the most tokens). we recognized that there might be more robust ways to implement this tokenization for languages with fewer tokens, and attributed some of the underperformance of the model in other languages to our generic tokenization strategy. we found that at a character level, even with no preaugmentation and fewer tokens, the model delivered impressive results. we propose that the character level modelling approach excels in terms of both accessibility and performance in this setting.