A llen Institute for AI @ SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages (2024)

Abstract: In this paper, we describe Allen AI’s submission to the constrained track of the SIGTYP 2024 Shared Task. Using only the data provided by the organizers, we pretrained a transformer-based multilingual model, then finetuned it on the Universal Dependencies (UD) annotations of a given language for a downstream task. Our systems achieved decent performance on the test set, beating the baseline in most language-task pairs, yet struggles with subtoken tags in multiword expressions as seen in Coptic and Ancient Hebrew. On the validation set, we obtained ≥70% F1- score on most language-task pairs. In addition, we also explored the cross-lingual capability of our trained models. This paper highlights our pretraining and finetuning process, and our findings from our internal evaluations.

## introduction this paper describes allen ai's submission to the constrained track of the sigtyp 2024 shared task on word embedding evaluation for ancient and historical languages. the constrained track requires participants to build a system for three linguistic tasks-parts-of-speech (pos) tagging, morphological annotation, and lemmatisation-using only the corpora provided by the organizers @xcite . the dataset contains universal dependencies v2.12 data @xcite in eleven languages with five old hungarian codices (has research institute for linguistics, 2018). the texts were from before 1700 ce, containing four language families (indo-european, afro-asiatic, sino-tibetan, and finno-ugric), and six scripts @xcite . finally, the dataset has over 2.6m tokens for training, and around 330k tokens for validation and testing. chu old church slavonic cop coptic fro old french got gothic grc ancient greek hbo ancient hebrew isl medieval icelandic lat classical and late latin latm medieval latin lzh classical chinese ohu old hungarian orv old east slavic san vedic sanskrit our general approach involves pretraining a transformer-based multilingual language model (lm) on the shared task dataset, and then finetuning the pretrained model using the universal dependencies (ud) annotations of each language. throughout this paper, we will refer to the pretrained model as libertus. we also explored data sampling and augmentation techniques during the pretraining step to ensure better generalization performance. on the validation set, we @xmath0. moreover, our systems achieved decent performance on the test set, yet struggled with subtoken tags in multiword expressions as seen in coptic (cop) and ancient hebrew (hbo). ## results in the following sections, we will outline our internal evaluations and benchmarking experiments. ## conclusion this paper describes team allen ai's system: a pretrained multilingual model (libertus) finetuned on different languages for each downstream task. our system obtained decent performance for the majority of language-task pairs. however, due to our training paradigm, it struggles annotating subtokens of multiword expressions. nevertheless, to ensure that the number of their unique tokens is greater than or equal to the most dominant language. because any form of sampling resulted to unstable pretraining and higher validation loss, we decided to stick with the dataset's original data distribution. we highly recommend exploring alternative data mixes to ensure that all languages will be represented while keeping the training process stable. a.2 finetuning a model per language vs.