UDP arse @ SIGTYP 2024 Shared Task : Modern Language Models for Historical Languages (2024)

Abstract: SIGTYP’s Shared Task on Word Embedding Evaluation for Ancient and Historical Languages was proposed in two variants, constrained or unconstrained. Whereas the constrained variant disallowed any other data to train embeddings or models than the data provided, the unconstrained variant did not have these limits. We participated in the five tasks of the unconstrained variant and came out first. The tasks were the prediction of part-of-speech, lemmas and morphological features and filling masked words and masked characters on 16 historical languages. We decided to use a dependency parser and train the data using an underlying pretrained transformer model to predict part-of-speech tags, lemmas, and morphological features. For predicting masked words, we used multilingual distilBERT (with rather bad results). In order to predict masked characters, our language model is extremely small: it is a model of 5-gram frequencies, obtained by reading the available training data.

## introduction since word embeddings and the transformer architecture @xcite found their way into natural language processing (nlp), results for all nlp tasks improved to unseen levels. multilingual pretrained language models like multilingual bert @xcite and xlm-roberta @xcite include word embeddings for up to 100 languages. however, historical languages are most unlikely to be covered by these models. since corpora of historical languages are limited in size and will most likely not grow anymore (unless an archaeological miracle unearths corpora yet unheard of) it will be difficult to include these languages to existing or new language models. in the sigtyp shared task on word embedding evaluation for ancient and historical languages 2024 (st 2024) 1 , it is proposed to present word 1 https://sigtyp.github.io/st2024.html embeddings/models for 16 historical languages (cf. the data used for the shared task @xcite has been compiled from various sources. old, middle, and early modern irish is taken from @xcite al. (2017), doyle (2018 both the training and the test data for the tasks 1, 2, and 3 is in conll-u foot_1 format, i.e. the documents are segmented into tokenised sentences. the values for pos and the morphological features in tasks 1 and 3 are the upos and ufeats sets of the ud project. however not all languages use all possible features, e.g., the old french data does not use the features number or person. the evaluation of the shared task is carried out by the codalab platform @xcite and uses the metrics shown in we participated in all five tasks of the unconstrained variant of the shared task, even though our approach for filling mask characters does not use any other data than the data provided by the organizers. apart from task 4 (filling masked words) we got the best results of all participants. ## related work even though this shared task is not about dependency parsing, pos tagging and lemmatisation are often present in dependency parsing. the shared task in dependency parsing 2018 @xcite processed three historical languages, ancient greek, latin, and old church slavonic, for which annotated data was present in the universal dependencies project at the time. in many of the approaches word embeddings were used (calculated on corpora of these languages using word2vec @xcite , glove @xcite , or fasttext @xcite , the latter already provides word embeddings for latin. the best results of the participants of the 2018 shared tasks for historical languages are above 98% for latin, above 97% for ancient greek, and above 96% for old church slavonic for pos tagging. lemmatisation for these languages also performs similarly well as modern languages. @xcite also studied the creation and evaluation word embeddings on latin for the analysis of language change. more recently, several large language models for classical greek and latin have been provided by @xcite who evaluated this models on pos-tagging and lemmatisation (as this shared task), and dependency parsing. brigada villa and giarda (2023) exploited models trained on modern english to parse old english, similar to our approach. evidently, word embeddings can be used for other tasks as well. e.g., @xcite use word embeddings of earlier version of english (but not going beyond the 1800s) to detect semantic shifts in english.