A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval G reek (2021)

Abstract: This paper presents a pilot study to automatic linguistic preprocessing of Ancient and Byzantine Greek, and morphological analysis more specifically. To this end, a novel subword-based BERT language model was trained on the basis of a varied corpus of Modern, Ancient and Post-classical Greek texts. Consequently, the obtained BERT embeddings were incorporated to train a fine-grained Part-of-Speech tagger for Ancient and Byzantine Greek. In addition, a corpus of Greek Epigrams was manually annotated and the resulting gold standard was used to evaluate the performance of the morphological analyser on Byzantine Greek. The experimental results show very good perplexity scores (4.9) for the BERT language model and state-of-the-art performance for the fine-grained Part-of-Speech tagger for in-domain data (treebanks containing a mixture of Classical and Medieval Greek), as well as for the newly created Byzantine Greek gold standard data set. The language models and associated code are made available for use at https://github.com/pranaydeeps/Ancient-Greek-BERT

## introduction during the last decades, large collections of digital texts have become available for ancient greek and latin. as a result, classicists are becoming more and more interested to apply natural language processing (nlp) techniques to extract information from these texts in an automatic and structured way. although there has been a boost in nlp research for greek and latin thanks to the introduction of the classical language tool kit foot_0 and the development of dependency treebanks, such as the the ancient greek and latin dependency treebank (agldt) foot_1 , there is still a lack of nlp tools that perform well on different historical varieties, such as for instance byzantine greek 3 . the presented research is to be situated in the overarching dbbe project foot_3 , where byzantine epigrams are studied as nodes between textual transmission and cultural and linguistic developments @xcite . this multidisciplinary project aims to reveal the connections between linguistic patterns and text-historical developments in a corpus of metrical paratexts in byzantine greek manuscripts and will develop new digital tools designed to perform linguistic analysis and to detect patterns and variations in this fragmented corpus of byzantine text. the linguistic analysis of @xcite greek has some challenges. the free word order is expected to create difficulties for automatic linguistic preprocessing, but both @xcite and @xcite conclude that the word order does not cause specific problems for the task of part-of-speech (pos) tagging. considering we are interested in byzantine greek in particular, it is also important to mention there are major differences between classical greek and byzantine greek, which can be observed on the phonetic, phonological, morphological, syntactical and lexical level. in some cases, byzantine greek is reminiscent of modern greek, rather than ancient greek. as a consequence, many of the systems that are widely used for the analysis of ancient greek are expected to struggle with byzantine greek. the very popular morphological analysis tool morpheus (cf. infra) in particular, is expected to not operate well for medieval greek. the remainder of this paper is organized as follows. in section 2, we briefly discuss some of the relevant research related to linguistic preprocessing of ancient greek. in section 3, we describe the data set used to train and evaluate the morphological analyser and provide details on the creation of a novel byzantine gold standard data set. section 4 gives an overview of the bert language model creation for ancient and medieval greek, whereas section 5 describes the fine-tuning of the resulting model on the task of morphological analysis. in section 6, we end with concluding remarks as well as indications for future research. ## related research previous research has resulted in various tools and resources for the linguistic analysis of ancient greek. the first systems to perform nlp tasks on ancient greek relied on morphological analysis tools. in the early 1970s, @xcite already wrote a program to analyze greek words with a 95% accuracy. great strides forward were achieved in the 1990s, when the morphological analysis tool morpheus was developed in the framework of the perseus project @xcite . from this point onward, morpheus has been the standard tool for ancient greek morphological analysis. in his paper on generating and parsing classical greek, @xcite describes how morpheus operates by combining information from three databases: one database containing stems, a second one containing possible inflections and a third one containing irregular forms. although this system is accurate in optimal circumstances, it has two distinct weaknesses: (1) all possible morphological analyses are provided, but morpheus does not decide which analysis is correct in casu, and (2) the rule-based nature does not deal well with unknown words or non-classical greek forms. the system can deal with some historical and dialectal variation to a certain extent, but byzantine greek is not within the scope of morpheus. a possible remedy consists in manually adding words to morpheus' vocabulary as shown by @xcite . this approach, however, does not scale very well nor would it remedy the issue of the conjugation and declension of existing words changing. nevertheless, we can observe that in many of the approaches presented for ancient greek, the morphological information provided by morpheus is fundamental in the analysis. ## language modeling with bert a shallow linguistic analysis showed that the actual nature of the "types" diverges from the expected byzantine greek. many of the byzantine irregularities have been normalized, and the text of the types contains a lot of classical structures. consequently, this will cause many of the existing systems for ancient greek to work remarkably well. therefore, we decided to train our final bert language model on a combination of ancient greek and modern greek data, as the types are definitely more similar to ancient greek than to modern greek. 4.1 ancient greek data used to train the model bert models are very data greedy. consequently, we did not only rely on the treebank databases to train the language model, but also used the (much larger) full text databases available for ancient greek. the text databases used were the following: (1) the (ancient greek part of the) perseus digital library and (2) the first1kgreek project database. the perseus digital library currently contains 13,407,448 words of classical greek texts. the first1kgreek collects one edition of every greek work composed between homer (9th century bc) and 250 ad, that does not occur in the perseus digital library. the combination of these two databases results in a relatively complete representation of classical greek. ## fine-tuning of the bert model for morphological analysis in a last step, we incorporated the novel bert language model for ancient and byzantine greek into our morphological analysis system. ## conclusions and byzantine greek, the so-called expanded ancient greek bert model, obtaining very good perplexity results. the model starts from a pre-trained modern greek language model, to which ancient and byzantine greek data (perseus digital library, first1kgreek database and agdt, proiel and gorman treebank data) is added. the performance of the language model was further improved by using special lowercasing and de-accentuation for greek. second, we fine-tuned a novel morphological analysis model for ancient and byzantine greek, that does not rely on separate morphological input from morpheus, as is the case for many existing taggers. as a result, this model is more flexible to analyse irregular forms in ancient greek and will perform better at analysing greek that does not originate from the classical period. the model achieves state-of-the-art performance on a validation set extracted from the proiel, agdt and gorman treebank data sets. the model also obtains very promising results on a newly created gold standard consisting of byzantine greek epigrams. in future research, we will build on the insights from this pilot study to improve the performance of the fine-grained part-of-speech tagger for byzantine greek. a first line of research will consist of establishing detailed annotation guidelines and conducting inter-annotator experiments to further improve and expand the gold standard data set. once a considerable amount of the dbbe will be annotated, this data can be used to further fine-tune the morphological analyser for byzantine greek data. another venue for future research will consist of adding more diverse greek data, e.g. byzatine data from the dbbe, to update the language model, and evaluate whether this impacts the performance of the tagger on byzantine greek data. finally, we will also experiment with a cascaded approach predicting the various parts of the morphological analysis consecutively, starting with the coarse-grained part-of-speech category (e.g., noun, verb). this allows to separate the "classical" part-of-speech from the fine-grained morphological analysis, providing more evidence for each part of the analysis. this approach might, however, introduce error percolation from the various consecutive steps. to conclude, we are confident that the newly developed bert-based language model will be a valuable contribution to nlp for ancient and medieval greek, as it can be fine-tuned for a variety of downstream tasks, such as linguistic analysis.