T artu NLP @ SIGTYP 2024 Shared Task: Adapting XLM - R o BERT a for Ancient and Historical Languages (2024)

Abstract: We present our submission to the unconstrained subtask of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages for morphological annotation, POS-tagging, lemmatization, characterand word-level gap-filling. We developed a simple, uniform, and computationally lightweight approach based on the adapters framework using parameter-efficient fine-tuning. We applied the same adapter-based approach uniformly to all tasks and 16 languages by fine-tuning stacked language- and task-specific adapters. Our submission obtained an overall second place out of three submissions, with the first place in word-level gap-filling. Our results show the feasibility of adapting language models pre-trained on modern languages to historical and ancient languages via adapter training.

## introduction the application of natural language processing techniques and pre-trained language models to analysis of ancient and historical languages is a compelling subject of research that has been so far overlooked. while there exist a number of benchmarks, such as glue @xcite , superglue @xcite , or xglue @xcite , for evaluating the quality of embeddings and language models for modern languages, such benchmarks are lacking for ancient and historical languages. thus, the sigtype 2024 shared task on word embedding evaluation for ancient and historical languages contributes to filling this gap. in the current transformer-based language models paradigm, one of the common approaches to solving the tasks present in such benchmarks is to use the task data to fine-tune an encoder transformer model. the approach was first in introduced in @xcite and was shown to yield superior results compared to the alternatives on the glue benchmark. additionally, the pro-posed pre-training method is very similar to the word-level gap-filling problem in the shared task. consequently, the model can be applied to solving the problem directly. this motivates our choice to use a transformer model in the shared task. large pre-trained language models, however, are predominantly trained on corpora of modern languages, with few exceptions such as latin-bert @xcite . ancient and historical languages generally lack sufficient data to perform full pre-training of large language models or to continue training from a checkpoint trained on some different language. full fine-tuning of modern language models on a relatively small amount data in an ancient/historical language might lead to overfitting and catastrophic forgetting. these considerations are also common in context of other low resource tasks or domains. several approaches have been proposed to alleviate these issues. for instance, the supplementary training approach proposed by @xcite involves first fine-tuning a pre-trained model on an intermediary labeled task with abundant data, and then on the target task which may have limited data. this approach showed gains over simply finetuning on the target task. @xcite developed a cross-lingual transfer-learning approach based on the adapters framework for parameter efficient fine-tuning of language models @xcite . their approach involves fine-tuning a language adapter and a task adapter stacked on top of each other. this adapter-based method is expanded by @xcite by adopting a custom tokenizer and an embedding layer. several ways of initializing the new embedding layers are compared on underrepresented modern languages. since ancient and historical languages are underrepresented, the same techniques should be applicable in this context as well. a useful feature of both the supplementary training approach of phang et al. (2018) and the adapterbased training of pfeiffer et al. ( 2020b ) is that they provide a uniform framework that can be applied to different languages and tasks in a similar manner. while previous related works mainly focused on modern languages, we aim to assess the feasibility of this unified approach for ancient and historical languages. our submission to the sigtyp 2024 shared task on ancient on word embedding evaluation for ancient and historical languages adopts the methods described by @xcite by stacking fine-tuned language and task adapters, and customizing the tokenizer and the embedding layers. our system is implemented as a unified framework, where various pre-trained models, languages, and tasks can be plugged in. the system was evaluated on pos tagging, morphological annotation, lemmatization, and filling-in both wordlevel and character-level gaps for 16 ancient and historical languages. we participated in the unconstrained subtask, which allowed using any additional resources, such as pre-trained language models. out of 3 participants on the leaderboard we took a close second place, with the first place on the word gap-filling task. our main contribution is showing that by adopting the parameter-efficient adapter training methodology, the large language models pre-trained on modern languages are applicable also to low-resource ancient and historical languages. ## adapters there exist a number of approaches to parameter efficient fine-tuning. in our submission to the shared task the focus is on adapters exclusively. @xcite propose a parameter efficient fine-tuning strategy that involves injection of a number of additional trainable layers into the original architecture. the architecture of the proposed strategy is illustrated on figure 1 . in each transformer layer an adapter layer is added twice: after the attention sub-layer and after the feedforward sub-layer. to limit the number of trainable parameters, the authors propose a bottleneck architecture of adapter layers: the adapter first projects the input into a smaller dimension, applies non-linearity, then projects back to the original dimension. this is known as the bottleneck adapter. pfeiffer et al. (2020b) extend on the strategy in context of cross-lingual transfer learning in two ways. first, the adapter stack technique is introduced (illustrated on figure 2 ), which is primarily used to separate language adaptation training and task-specific training. in the technique a language adapter is first trained, then a task-specific adapter is stacked on top of it and trained (while the language adapter is frozen). secondly, the bottleneck adapter is expanded upon to include the embedding layer as well-the invertible adapter is introduced that transforms both input and output token representations to further improve language adaptation. ## methodology our approach in this work is based on training language adapters and task specific adapters for xlm-roberta @xcite )-the multilingual variant of roberta (zhuang et al., 2021) trained on 100 different languages. since many of the languages in the shared task have related modern languages in the training data of xlm-roberta, or are even included themselves (such as latin), we expect to benefit from knowledge transfer. ## results according to the results on we note a surprisingly strong performance of our algorithmic approach to character-level gap-filling with the exception of classical chinese where the approach is simply not applicable. we expect the extensions to the approach outlined in section 3.4 could improve the results further. however, in context of the shared task, we do not believe that the possible improvements would be worth the effort. due to the whitespaces appearing as masked characters there is no way to know in advance how many individual words in the sentence in total have masked characters in them. in addition to that, a single word may contain several masked characters. the combination of these two factors makes improvements based on language model rescoring quite computationally expensive. for classical chinese, however, we hypothesise that the score would be considerably higher if the ground truth was also in the decomposed form, because to the best of our understanding, reconstructing the original characters from individual strokes is not a straightforward task. despite taking the first place in the word-level gap filling task, the absolute scores are still very low. this is to be expected, because the difficulty of the task is compounded by the presence of multiple masked words per sentence, as well the reliance on subword tokenization of our approach. additionally, the implementation may not be suitable at all for certain languages due to differences in the writing systems. finally, we surmise that due to the amount of training being quite small for the masked language modeling problem, we cannot expect to be able to correctly predict completely new words, however we can, in theory, predict previously unseen word forms due to subword tokenization. this means that experimenting with the k parameter used to limit the number of times the look ahead for word continuation is performed in the word-level gap filling problem could improve the results somewhat. the pattern-based approach to lemmatization also performs reasonably well in most languages. the approach is restricted by the set of patterns derived based on the training set. if the evaluation set contains many words for which a suitable pattern is missing, the system simply cannot make a correct prediction. however, we believe that this is not the problem here. in particular, we note that those languages which have lower scores on lemmatization, also perform consistently lower on other tasks. thus, we suggest that the lower performance is rather related to insufficient data for training meaningful representations. in addition to underpeforming on languages with underrepresented writing systems, we observe somewhat low performance on vedic sanskrit as well. one explanation could be that the usage of diacritics may negatively affect the character-based transformation rule generation; however this hypothesis requires further investigation. finally, our combined approach to pos-tagging and morphological analysis performs quite well on most of the languages. this is somewhat unexpected, because the model is limited in predicting only the combinations of part-of-speech and morphological features that are present in training data. ## conclusion this paper described our solution to the sigtyp 2024 shared task developed based on the adapters framework. the system is simple and uniform, and can be easily extended to other tasks and languages. for each language we trained a language adapter, and two task adapters, one for pos and morphological tagging and one for lemmatization. for languages with scripts underrepresented in the xlm-roberta vocabulary, we additionally created custom tokenizers and embeddings. one notable advantage of the adopted approach is its resource efficiency-adapting the model to a new language and task can be done in less than an hour. as a future work, the system could be improved with more advanced adapter-based techniques such as adapter fusion @xcite to leverage typological relatedness of languages.