Findings of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages (2024)

Abstract: This paper discusses the organisation and findings of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages. The shared task was split into the constrained and unconstrained tracks and involved solving either 3 or 5 problems for either 13 or 16 ancient and historical languages belonging to 4 language families, and making use of 6 different scripts. There were 14 registrations in total, of which 3 teams submitted to each track. Out of these 6 submissions, 2 systems were successful in the constrained setting and another 2 in the uncon- strained setting, and 4 system description papers were submitted by different teams. The best average result for morphological feature prediction was about 96%, while the best average results for POS-tagging and lemmatisation were 96% and 94% respectively. At the word level, the winning team could not achieve a higher average accuracy across all 16 languages than 5.95%, which demonstrates the difficulty of this problem. At the character level, the best average result over 16 languages 55.62%

## introduction the importance of nlp for studies in the classics is growing, as can be seen by the variety of technologies, digital text resources, and applications being developed to support research tasks in this field in recent years @xcite @xcite . as the value of machine learning for historical linguistics is becoming more apparent, academic interest in word embedding models for use in these contexts is also increasing @xcite @xcite @xcite . since the rise of word embeddings, their evaluation has been considered a challenging task that sparked considerable debate regarding the optimal approach. the two major strategies that researchers have developed over the years are intrinsic and extrinsic evaluation. the first amounts to solving specially designed problems like semantic proportions, or comparing the similarity of machine-generated words or sentences against human-generated examples. the second one focuses on solving downstream nlp tasks, such as sentiment analysis or question answering, probing word or sentence representations in real-world applications. in recent years, sets of downstream tasks called benchmarks have become a very popular, if not default, method to evaluate general-purpose word and sentence embeddings. despite the general trend towards multilinguality and ever-growing attention to under-resourced languages, ancient and historical languages remain under-served by embedding evaluation benchmarks, and the goal of this shared task is to bridge this gap. we argue that there is a need for a universal multilingual evaluation benchmark for embeddings learned from ancient and historical language data and view this shared task as a proving ground for it. ## related work starting with decanlp @xcite and senteval @xcite , generalpurpose multitask benchmarks for natural language understanding (nlu) have become increasingly common in the literature, and new ones are reported regularly @xcite @xcite @xcite @xcite . however, even the largest multilingual benchmarks, such as xglue, xtreme, xtreme-r or xtreme-up @xcite @xcite , only include modern languages. the evalatin evaluation campaign @xcite attracted some embeddingbased solutions for pos-tagging, lemmatisation, and morphological feature prediction challenges @xcite , but it did not specifically focus on embedding evaluation. moreover, it was confined to latin, which is the english of the ancient world in terms of language resources and technologies available. individual scholars focusing on latin and ancient greek mostly adopt large language models (llms) together with their evaluation techniques through downstream tasks @xcite @xcite @xcite , while those working with lessresourced languages tend to translate intrinsic evaluation datasets from modern languages or create their own diagnostic tests @xcite @xcite . however, this is not a universal rule: the latest paper on distributional semantic models of ancient greek proposes a new dataset for intrinsic evaluation, agree @xcite , while some recent papers featuring medieval french and spanish adopt transformer models and test them on named entity recognition @xcite . ## setup and schedule for the purposes of our evaluation, languages are distinguished in accordance with iso 639-3 codes foot_0 except for latin, which was manually separated as discussed in section 4. as a result, different historical stages of irish and latin are treated as distinct 'languages' in this paper. such a distinction may be linguistically arbitrary, at least in the case of certain texts. however, as universal dependencies (ud) @xcite corpora are separated in accordance with iso 639 codes, and the majority of data used in this evaluation was drawn from this resource, the same system for distinguishing languages was utilised here. the shared task involved three problems (hereafter also referred as 'challenges' and 'downstream tasks') for 13 languages in the constrained setting and five problems for 16 languages in the unconstrained setting. these languages belong to four language families and use six different scripts (see ## data for problems 1-3, data from universal dependencies v.2.12 (zeman et al., 2023) was used for 11 ancient and historical languages, omitting corpora which contained fewer than 1,000 tokens or for which only a test set was available. old hungarian texts, annotated to the same standard as ud corpora, were added to the dataset from the mgtsz website foot_1 (has research institute for linguistics, 2018; @xcite . old hungarian data was edited to simplify complex punctuation marks ## evaluation the shared task was hosted on codalab foot_11 and will remain available for post-competition submissions for anyone who would be interested in testing their approach on our data. our evaluation script calculates a score for each problem in the task (pos-tagging, lemmatisation etc.) per language with the metrics listed in ## baseline models baselines were provided for the three challenges which are shared by both the constrained and unconstrained tracks. as the aim of this shared task was to provide a benchmark for embedding models, multi-layer perceptron network models were developed to classify token data for each of the three challenges. for the sake of ensuring simplicity across the baseline models and results, model design and input data format was kept as similar as possible across all challenges. slight variation was tolerated, however, depending on the requirements of each specific challenge. specific models were trained for each of the 13 languages for both the pos-tagging and lemmatisation challenges. by contrast, the approach taken for the morphological annotation challenge was to train a language-agnostic model for each of the 44 morphological features used across all languages in the dataset. this was found to produce better results than using language specific models, particularly for morphological features which were not common across all languages in the dataset. it also reduced model training time, as the alternative would have been to create a discrete model for each feature in use by each individual language, resulting in significantly more models. early stopping was applied during training of all models to avoid overfitting. validation loss was used as a metric to determine when early stopping should be applied for pos-tagger and morphological feature analysis models. however, tracking validation accuracy instead was found to produce better results when training lemmatiser models. all pos-tagger and morphological feature analysis models used 64 neurons per hidden layer, as did lemmatiser models for smaller datasets, however, for languages with larger datasets this was found to be insufficient. to avoid hampering performance, lemmatiser models were created with up to 1024 neurons per hidden layer, depending on the size of the dataset. aside from the areas of divergence just mentioned, the design aspects common to all models are as follows: ## submitted systems there were 14 registrations in total, of which three teams submitted to each track. out of these six submissions, two systems were successful in the constrained setting and another two in the unconstrained setting, and four system description papers were submitted by different teams. we expected that participants would use the same pre-training technique for every problem, as is common in benchmarking, but the winning teams applied different pre-training approaches to different problems. at the same time, all participants leveraged various transformer architectures, with roberta @xcite and its modifications being the most popular one. while all participants outperformed our baselines for morphological feature prediction with the best average result about 96% across 13 languages, only the winning teams beat the baselines for postagging and lemmatisation, achieving average results of 95.25% and 93.67% respectively in the constrained setting, and 96.09% and 94.88% in the unconstrained setting. baselines were not provided for the mask filling problems which formed a part of the unconstrained track only. at the word level, the winning team could not achieve a higher average accuracy across all 16 languages than 5.95%, with the best result for an individual language being 16.9% for medieval icelandic. this outlines the particular difficulty of this specific problem. at the character level, the best average result over 16 languages was 55.62% and the best result for an individual language was 74.59% for gothic. the combined results of the constrained and unconstrained settings for problems 1-3 are provided in ## discussion analysing results of the competition, we made a few interesting observations. first of all, data scarcity does have an effect on sequence labelling tasks, such as pos-tagging and morphological feature prediction, but this effect is not as dramatic as one might expect. thus, the difference between the smallest corpus of 21k tokens (vedic sanskrit) and the biggest corpus of 599k tokens (medieval latin) is only 9.5% on average for pos-tagging and 11.3% for morphological feature prediction. the same is true for lemmatisation; however, models trained for this task seem to be more susceptible to orthographic variation and lexical variety in the data, as well as to the morphological complexity of a language. thus, we see poorer results for lemmatisation across all languages despite the milder metrics. cross-lingual and cross-temporal (i.e. from modern languages to their ancestors) transfer could have played an important role in the systems that used xlm-roberta. however, riemenschneider and @xcite showed that similar results can be achieved with pre-training on modestly sized monolingual data without any transfer. mask filling tasks appeared to be much harder than we expected even for sota models. the problem could be attributable to the following reasons, or to some combination thereof: ## conclusion the shared task on word embedding evaluation for ancient and historical languages attracted participants from five major research institutions and was an important step towards creating a universal multilingual evaluation benchmark for embeddings learned from ancient and historical language data. the best average results across 13 languages for pos-tagging, lemmatisation and morphological feature prediction were 96.09%, 94.88% and 96.68% respectively. however, participants only managed to achieve an average of 5.95% at wordlevel and 55.62% at character-level across 16 languages in more challenging mask filling tasks. the dataset and evaluation scripts are available on our github, foot_17 and the post-competition phase on codalab will remain open for anyone interested in testing their approach on our data. we are planning to further expand the dataset with more languages and add more downstream tasks in the next release of the benchmark. we would appreciate any suggestions and collaboration from both computer scientists and historical linguists.