Sentence Embedding Models for A ncient G reek Using Multilingual Knowledge Distillation (2023)

Abstract: Contextual language models have been trained on Classical languages, including Ancient Greek and Latin, for tasks such as lemmatization, morphological tagging, part of speech tagging, authorship attribution, and detection of scribal errors. However, high-quality sentence embedding models for these historical languages are significantly more difficult to achieve due to the lack of training data. In this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text. The state-of-the-art sentence embedding approaches for high-resource languages use massive datasets, but our distillation approach allows our Ancient Greek models to inherit the properties of these models while using a relatively small amount of translated sentence data. We build a parallel sentence dataset using a sentence-embedding alignment method to align Ancient Greek documents with English translations, and use this dataset to train our models. We evaluate our models on translation search, semantic similarity, and semantic retrieval tasks and investigate translation bias. We make our training and evaluation datasets freely available.

## introduction sentence embedding models, which map sentences or other sequences of text to a dense vector space, such that semantically similar sentences are close together in the vector space, have many applications in nlp. current state-of-the-art sentence embedding models, however, are trained on modern, high-resource languages such as english and use massive datasets consisting of billions of sentence pairs @xcite . a different approach is needed for historical languages, which have much less data available. in this work, we train several sentence embedding models for ancient greek. many more ancient greek texts have survived compared to texts from most other ancient languages, which makes sentence embedding models both more feasible and useful. several previous works have trained language models for ancient greek. @xcite introduced the classical language toolkit (cltk) which includes several tools for ancient greek processing, including static word embeddings. @xcite fine-tuned a modern greek bert model @xcite on ancient greek text for pos tagging, morphological tagging, and lemmatization tasks. @xcite trained a bert model for authorship classification of pseudo-plutarch texts. @xcite trained another bert model for the purpose of identifying errors in scribal transmission. @xcite produced the most comprehensive work on classical language models to date, training multiple models on a large multilingual corpus of ancient greek, latin, and english texts and comprehensively evaluating and comparing their new models to previous models on a variety of tasks. none of these works, however, produce sentence embedding models for ancient greek. although there are many digitized ancient greek texts available, there is a lack of suitable training data for training sentence embedding models from scratch. the best approaches for highresource languages involve large human-annotated datasets, such as the natural language inference (nli) datasets used by sentence-bert @xcite . needless to say, such datasets are not available for ancient greek. following @xcite , we use multilingual knowledge distillation to train sentence embedding models with an aligned vector space for ancient greek and english. given a teacher model m for a language s, and a dataset of translated sentences ((s 1 , t 1 )..(s n , t n )) where s i and t i are parallel sentences, we train a new student model m to mimic the sentence embeddings of the teacher m using mean squared loss, such that m (s i ) ≈ m (t i ) and m (t i ) ≈ m (s i ). in our case, the teacher model is english and the student model learns both greek foot_0 and english embeddings. this approach has numerous advantages: 1) it requires a relatively small amount of training data, 2) the student model inherits the vector space properties of a state-of-the-art english sentence embedding model, 3) the student model is multilingual, and 4) the vector spaces are aligned across languages. the cross-lingual nature of this approach is especially useful for ancient greek semantic retrieval, since it is much easier to formulate search queries in english than in ancient greek. although it is possible to operate on the english translations of greek texts, translations are not readily available for all greek texts, and the available translations are usually not aligned at the sentence level, making it difficult to quickly find the corresponding greek text. furthermore, english translations can suffer from various kinds of translator bias, whereas a language model that operates directly on the greek text can offer an "average" of multiple translators' interpretations of the text (see section 4.4). we produce a training dataset of parallel sentences using a two-step translation alignment process: an initial, smaller dataset was produced using a sentence-length heuristic and dictionary-based alignment technique @xcite , and this initial dataset was used to train an intermediate multilingual sentence embedding model, which was used to align a larger dataset using the approach introduced by @xcite , which uses sentence embeddings for state-of-the-art alignment quality. we create new evaluation datasets for ancient greek translation search, semantic textual similarity (sts), and semantic retrieval (sr) and we evaluate our models on these datasets. in summary, our contributions are as follows: ## training 2.1 base models to train a sentence embedding model, we first need a base language model trained on ancient greek text. the existing ancient greek language models were unsuitable for our purposes; most of them are monolingual, but we are training a multilingual model. the models trained by riemenschneider and frank (2023) would be the best candidates because they include english, but one of their goals was to avoid contamination from modern languages, such as modern concepts and technology like cellphones which were unknown in antiquity. however, for us this is not a concern, since one of our goals is to train a model to facilitate semantic search with modern language and terminology. instead, we fine-tune multilingual bert-base (mbert) @xcite and xlm-roberta-base (xlm-r) @xcite for our base models. @xcite shows that low-resource languages can benefit from multilingual pre-training. we use masked language modeling (mlm) to fine-tune mbert, (denoted as negatively impact performance @xcite . we use a similar approach to @xcite to compare the mbert and xlm-r tokenizers. we take a random sample of 20k ancient greek sentences from the pre-training corpus and compute the average token length and average words per token for a rough estimation of tokenization quality (see ## conclusion in this paper, we have shown that multilingual knowledge distillation is an effective approach for training sentence embedding models for ancient greek, in spite of the lack of available training data compared to modern, high-resource languages. in addition, we have produced a new dataset of parallel ancient greek and english sentences as well as evaluation datasets for translation search, semantic textual similarity, and semantic retrieval, which we make publicly available.