An Ensembled Encoder-Decoder System for Interlinear Glossed Text (2023)

Abstract: This paper presents my submission to Track 1 of the 2023 SIGMORPHON shared task on interlinear glossed text (IGT). There are a wide amount of techniques for building and training IGT models (see Moeller and Hulden, 2018; McMillan-Major, 2020; Zhao et al., 2020). I describe my ensembled sequence-to-sequence approach, perform experiments, and share my submission’s test-set accuracy. I also discuss future areas of research in low-resource token classification methods for IGT.

## introduction this paper is a system demonstration for our submission to the 2023 sigmorphon shared task on interlinear glossed text @xcite . we focused on the closed track of the task, where only the input sentence, output gloss, and translation are provided in training data. this was more restrictive than the open track, in which more information was available, such as morphological segmentations or part-of-speech tags. ## methods the system was based on an encoder-decoder model using the lstm architecture @xcite . it used ensembling and data augmentation as a method to counteract the relatively lower performance of encoder-decoder models as highlighted in the previous section. the system was implemented with fairseq @xcite and trained on a single nvidia geforce mx350. fairseq has built-in support for transformers as well as lstms, but the former requires more resources to train. the gpu used for this project did not have sufficient memory for training a convergent transformer model, and so the lstm architecture was chosen instead. ## results the final system consisted of two word-level sequence-to-sequence models, trained with a word window size of one and two, respectively. the input alphabet consisted of characters and the output alphabet was token-level. the models were trained with an inverse square root learning rate scheduler, early stopping, and the adam optimizer. models trained on all languages except for gitksan used a batch size of 128. since the gitksan training dataset was just 31 examples long, it used a batch size of 64 instead. see ## conclusion this was a system demonstration of our submission to the 2023 sigmorphon shared task on interlinear glossed text. while the system was not the best-performing of all the submissions, it nonetheless performed consistently better than the baseline model in terms of word-and morpheme-level testset accuracy. the system was relatively inexpensive to train, as it was built on a single cuda-enabled laptop. this could be an advantage of the lstm-based architecture: when @xcite introduced the transformer architecture to character-level transduction tasks, the authors noted that transformer-based performance depended on finely-tuned hyperparameters and longer training times. however, non-encoder-decoder systems for the ilg task still show lots of promise, especially for small datasets. further research can be done to examine the effect of ensembling and data augmentation on crf or lstm-based token classification systems. more work can be done on the stem generation system as well: a linguist-created inflectional database like the one described in @xcite could algorithmically recognize stems and look up translations. also, an upstream word alignment model, such as one of the ibm models @xcite , could help with the construction of a stemmer system. we hope this demonstration will lead to future work on low-resource systems for automatic ilg, in terms of both computation and dataset size.