Medieval Social Media: Manual and Automatic Annotation of Byzantine G reek Marginal Writing (2023)

Abstract: In this paper, we present the interim results of a transformer-based annotation pipeline for Ancient and Medieval Greek. As the texts in the Database of Byzantine Book Epigrams have not been normalised, they pose more challenges for manual and automatic annotation than Ancient Greek, normalised texts do. As a result, the existing annotation tools perform poorly. We compiled three data sets for the development of an automatic annotation tool and carried out an inter-annotator agreement study, with a promising agreement score. The experimental results show that our part-of-speech tagger yields accuracy scores that are almost 50 percentage points higher than the widely used rule-based system Morpheus. In addition, error analysis revealed problems related to phenomena also occurring in current social media language.

## introduction despite the nonexistence of the world wide web in the middle ages, byzantine book epigrams bear some resemblance to current social media, such as twitter. 1 just like a tweet, a book epigram is usually a rather short, personal statement of an author, who expresses themselves on their daily occupation, i.e. copying manuscripts. furthermore, the typeface of both tweets and book epigrams displays a lot of orthographic inconsistencies as the content is often written phonetically. however, the big difference between social media and byzantine book epigrams is the amount of text available for nlp: 575,000 tweets are sent every minute, while the database of byzantine book epigrams (dbbe) @xcite counts 12,000 epigrams in total. the byzantine book epigrams that make up the dbbe, can be defined as metrical paratexts, i.e. poems standing next to (para, from the greek word παρά) another text or figure. they often appear in the margins of manuscripts or as scribblings between two paragraphs. concerning content, these epigrams, among other things, comment on the main text of the manuscript, give some insight in the life of the scribe or show off the scribe's knowledge. dbbe occurrence 32143 serves as an example, provided with the authors' translation: the scribe of the book epigram -who may or may not have authored it -did not always know (or care?) which of the five [i]'s needed to be written. the disyllabic word ἰδεῖν (to look), for example, is present in 19 different forms in dbbe. exactly that is the added value of dbbe compared to other pre-modern greek corpora: these corpora generally @xcite greek that is normalised to an ancient greek model, while dbbe provides both the original transcription of the manuscript and an edited, normalised version. the former is called occurrence, the latter type. the texts of the dbbe will be subject of further linguistic and literary research, for which these texts are ideally all annotated. since manual annotation is not feasible for all words, we opted for an automatic way to do so. preliminary tests showed that existing systems for morphological analysis do not perform well on the text of the occurrences. to overcome the shortcomings of current systems for morphological analysis, we developed a novel transformerbased part-of-speech tagger for ancient and medieval greek. 3 to evaluate the performance of the tagger, a novel gold standard for byzantine greek was developed, where all tokens were provided with a coarse-grained part-ofspeech tag and full morphological analysis. in addition, we also performed an error analysis, which revealed several problems that are very typical to this kind of texts, i.e. texts where the material context (the manuscript) strongly affects the language. ## related research the interest in nlp for pre-modern greek has increased over the last few years, thanks toamong other things -the availability of opensource corpora. the first corpus initiative for greek texts was the thesaurus linguae graecae (tlg) @xcite , a comprehensive digital library of greek texts written between 800 bc and 1453 ad (viz. the fall of byzantium), that sums up to more than 110m tokens, covering 10,000 works and 4,000 authors. the tlg, however, is not freely available. an open-source alternative is the open greek and latin project foot_3 , that consists of the perseus digital library @xcite , a collection of more than 13,5m tokens of mostly classical greek prose and poetry, on the one hand, and the first1k project, a complementary part to perseus summing up to 25,5m tokens of classical and post-classical greek prose and poetry 5 . in addition to these two text corpora, several treebanks were developed. the ancient greek dependency treebank (agdt) @xcite stores 560,000 tokens from both classical prose and poetry, that were manually provided with a part-of-speech tag, morphological analysis, lemma and syntactic relation. proiel @xcite has a more specific content: the treebank stores the new testament in greek and four other languages, counting 277,000 tokens. the gorman treebank (gorman, 2020) is a treebank of around 550,000 tokens of exclusively classical greek prose. as a last example, the pedalion trees @xcite are almost completely complementary to the agdt (apart from some texts) and count some 320,000 tokens. the pedalion trees contain annotated texts from trismegistos @xcite , a database of papyrus texts, that displays the original text with all its idiosyncrasies and even errors, just like the occurrences in dbbe. all of the above mentioned treebanks make use of or have extended the universal dependencies @xcite . since the development of morpheus @xcite , a rule-and dictionary-based system to perform part-of-speech tagging (or morphological analysis) of greek tokens, multiple partof-speech taggers have been developed to cope with morpheus' two main pitfalls: it does not disambiguate ambiguous forms and it cannot deal with out-of-vocabulary words. @xcite did a comparative study, which showed that matetagger @xcite outperformed hunpos tagger @xcite , rftagger @xcite , the opennlp part-of-speech tagger foot_5 and nltk unigram tagger @xcite on ancient, normalised greek data. when keersmaekers (2019) repeated that experiment with mate tagger, rftagger and marmot tagger @xcite to find out which is best suited for papyrological data, rftagger outperformed the other two. @xcite also developed rnn tagger, the neural counterpart of rftagger. @xcite explored the possibilities of a transformer-based part-of-speech tagger on dbbe types, the normalised text of the book epigrams, which yielded promising results. ## data compilation and annotation our aim is not to annotate the dbbe types, the normalised poems, but the dbbe occurrences. to achieve this, we trained a transformer-based language model, of which the embeddings are used to train a part-of-speech tagger. section 3.1 describes the data sets used for training the language model and fine-tuning it for partof-speech tagging, while section 3.2 describes the manual annotation and validation of the byzantine greek evaluation set. ## error analysis and discussion as mentioned in section 1, the book epigrams bear some resemblance to modern social media posts. exactly those similarities are an interesting starting point for our error analysis. let us begin with the appearance of a social media comment, which can accompany a picture, an opinion on someone's message, or just a retweet of another tweet. those social media comments could be categorised as paratexts: a text standing next to (para, from the greek παρά) another text, just as our book epigrams. they are mostly to be found in the margins around the main text of a manuscript, a material property of this corpus that determines the first category of errors. to illustrate this error type, we will discuss the following verse (english translation in italics): ## conclusion and future research the database of byzantine book epigrams stores a very challenging corpus with its own peculiarities and problems for automatic processing. this automatic processing is necessary since manual annotation is not feasible for the complete dbbe corpus. to develop a more flexible approach that is able to cope with lots of orthographic variety and out-ofvocabulary words, we trained a novel language model for greek, the dbbert, and fine-tuned it for part-of-speech tagging. to evaluate this part-of-speech-tagger on byzantine greek, we developed a novel gold standard, which was manually annotated using the agdt annotation guidelines. this label set was first subject of an iaa study, that showed very high agreement scores. although the evaluation showed promising results, the error analysis exposed once more the inherent problems of the book epigrams, which philologists still agonise over. an important next step in our research is the development of a lemmatizer, which will make the annotation of our corpus complete. once this annotation is done, we will research how similarity can be measured between hemistichs, verses and epigrams in the dbbe, in order to link similar texts copied (and sometimes altered) by different scribes.