Restoring M ycenaean L inear B ‘A& B ’ series tablets using supervised and transfer learning (2024)

Abstract: We investigate the problem of restoring Mycenaean linear B clay tablets, dating from about 1400 B.C. to roughly 1200 B.C., by using text infilling methods based on machine learning models. Our goals here are: first to try to improve the results of the methods used in the related literature by focusing on the characteristics of the Mycenaean Linear B writing system (series D), second to examine the same problem for the first time on series A&B and finally to investigate transfer learning using series D as source and the smaller series A&B as target. Our results show promising results in the supervised learning tasks, while further investigation is needed to better exploit the merits of transfer learning.

## introduction for many years the language attributed by the linear b script (c. 1400-1200 b.c.) was a point of contention among scientists, who argued over the origin of the mycenaean syllabary. the answer was given in 1952 by michael ventris @xcite who, together with the philologist john chadwick, proved that the syllables of the linear b script form words of the greek language and that the mycenaean world was both linguistically and culturally linked to ancient greece. mycenaean linear b is a syllabic script. it includes syllables as well as logograms or ideograms. in summary, linear b is structured by groups of phonetic symbols, which are accompanied by ideograms. the surviving tablets typically refer to human names, place names, agricultural production, land ownership, religious offerings, or military equipment. the mycenaean inscriptions have been classified based on their place of origin, but also based on the category to which they belong. the place of origin is indicated by the following abbreviations: kn (knossos), py (pylos), my (mycenae), th (thebes), ti (tiryns), kh (khania), mi (midea), etc. the classification into categories was based on the ideograms of the tablets: series a&b, (lists of personnel), series c (animal records), series d (sheep records), series e (grain records), series f&g (records of oil, agricultural products and their offerings), series l (textile records), etc. @xcite , @xcite , @xcite , @xcite . the main challenge faced by those dealing with the study and restoration of the mycenaean linear b texts, either manually @xcite @xcite @xcite @xcite @xcite @xcite @xcite or computationally @xcite , is the scarcity of data. furthermore, we have to take into account the particularities presented by the mycenaean inscriptions: a) their eminently administrative content, b) their subject as they deal extensively with people and places and c) their state of preservation since most of them are broken, worn out or burnt. these make the infilling task very challenging. this article contributes by investigating transfer learning (tl) techniques to alleviate the above mentioned data scarcity. tl is the process of taking a model that has been trained to do one task (pre-trained model) and fine-tuning it to work on a related (different or similar) task. in natural language processing (nlp), pre-trained models are often used as the starting point for a wide range of nlp tasks, such as language translation, sentiment analysis, and text summarization. by using a pretrained model, we can save time and resources, as they don't have to train a model from scratch on a large dataset @xcite @xcite . in this work we deal with series a&b and series d to find out the effect of tl from a series to another one. to this end we have chosen to investigate the effect of tl from the series d, since it is the largest with the most tablets and resulting sequences, to the series a&b, as opposed to training the series as a whole. we also contribute by investigating whether there is an improvement in the predictions of models learned from series d @xcite if we remove the ideograms from the sequences. this choice is based on the observation that in linear b, typically there is consistent separation of phonographic (syllabograms) and non-phonographic (ideograms as well as signs for measurement units and numbers) graphemes @xcite . finally we apply generative methods for the infilling problem of series a&b. following the same tactic with series d, we create a dataset of sequences derived from the mycenaean tablets of series a&b, excluding ideograms, and use it to learn a generative model in order to predict damaged parts of this series' tablets. the rest of the paper is organized as follows: in section 2 we provide an overview of previous research and studies on various methods of restoring ancient inscriptions. in section 3 we present the dataset for our experiments. section 4 demonstrates supervised learning for language modelling and for missing symbol recovery. section 5 presents transfer learning techniques for infilling series a&b. finally, in section 6 we present our conclusions and the future work. ## related work the problem of text restoration through infilling is attracting more and more attention from researchers community; however most recent results from the natural language processing (nlp) community have been only partially applied, obviously due to the lack of sufficient data. some of the simplest models are the n-grams. these are probabilistic models for predicting the next item in a sequence of n elements and can be used to model almost any type of sequential data. they have been used for machine translation @xcite , but also for textual restoration. @xcite and @xcite use n-gram markov chains for texts in the indus script. the benefits of n-gram models are their simplicity and scalability. with larger n, a model can store more context, enabling small experiments to scale up. however, when n increases, the number of possible n-grams increases exponentially and therefore the out-of-vocabulary n-grams increase as well and actually undermine the performance of the model. obviously, the n-grams are not appropriate for long sequences. similar to our work is the pythia system (assael et al., 2019) and its follow-up system ithaca @xcite . it aims to fill the missing symbols (characters) in ancient greek inscriptions. the authors use a sequence-to-sequence framework @xcite with long short-term memory (lstm) networks in the encoder and the decoder. the encoder involves the input character embeddings sequence with missing characters, and a separate stream is also modelled using the word sequence as embeddings as well; an attention layer is also used. the decoder is trained to output the missing characters. they use a dataset that results from processing the epigraphical corpora of the packard humanities institute (packard human-ities institute, 2005), the phi-ml. as @xcite argue in their ancient text restoration experiment, @xcite perform restoration at the character-level where the number of characters to recover is assumed to be known and indicated by a corresponding number of '?' symbols. in reality, when epigraphists restore a deteriorated document, the length of the lost fragment is unknown and needs to be guessed as a first step. blm, in essence a variant of blm, the l-blm, can bypass this limitation and flexibly generate completions without this additional knowledge. a single token, sized equal to the number of '?' symbols, is defined and the l-blm is trained to predict a character to fill in and the length of the new blank to its left. compared to our work, the problem presented by the authors of these articles @xcite is similar in the sense that it concerns a known script and known language and uses a machine learning architecture. however, our task is more challenging, due to the fact that the corpus is of much smaller size (over 40000 inscriptions available in the aforementioned articles versus 1100 inscriptions in ours); that impedes training. @xcite use recurrent neural networks (lstm) to restore fragmentary babylonian texts. these involve ancient texts in the akkadian language, which belong to the semitic language family. comparisons to simple 2-gram baseline approach (considering the previous and the next word) are made, resulting in better performance. the experiments use a dataset of 3000 transliterated archival documents belonging to economic, juridical and administrative genres. similarly to this work, @xcite also introduce bertbased models aiming to solve the task of predicting missing signs in akkadian texts. the difference with the previous article @xcite is that the completion of missing signs is done by combining large-scale multilingual pretraining with akkadian language finetuning. although @xcite have small-scale data at their disposal to train the learning algorithm (c. 3000 babylonian transliterated texts, 539-331 b.c.e.), what is emphasized by the authors is that the late babylonian texts are structured official bureaucratic documents, e.g., legal proceedings, receipts, promissory notes, contracts and so on. this is in stark contrast to linear b tablets, which are significantly impeded by syntactic inconsistencies. this is proved by the fact that the pre-processing of each mycenaean tablet requires special handling, so as to extract valid sequences of mycenaean words in accordance with the principles of mycenaean language. another bert-based model, latin bert, is proposed by @xcite . they pre-trained bert model on latin texts from perseus, proiel and index thomisticus treebank, targeting restoration and several other downstream tasks. @xcite offer a review on published research using machine learning for the study of ancient texts. they also classify the studies of ancient texts into tasks: digitisation, restoration, attribution, linguistic analysis, textual criticism, translation and decipherment. finally, a similar review task takes place in the article @xcite , but focusing on the computational techniques related to the bronze age aegean and cypriot scripts, namely the archanes script and the archanes formula, cretan hieroglyphic (including the malia altar stone and arkalochori axe), phaistos disk, linear a, linear b, cypro-minoan and cypriot scripts. the work in @xcite is similar to ours and is the only one that we are aware of that does infilling for the linear b tablets. however, that work is limited to series d tablets and considers both phonogrpahic and non-phonographic symbols. furthermore, like all aforementioned methods, it does not investigate transfer learning. ## the mycenaean dataset here we present the modifications we made to the mycenaean dataset of series d, that was initially created as described in @xcite and @xcite . we also present the way to create the new dataset of series a&b. the linear b script uses two basic symbol systems, one phonetic (phonographic component) and one logographic (non-phonographic component). the symbols of the phonetic system are called syllabograms-syllables. the phonetic system is usually represented transcribed, i.e., the syllable is rendered in letters, and in most cases by a combination of consonant and vowel. the system of the phonetic symbols, includes at least 87 different syllables. for the symbols of the logographic system, the term 'ideograms' or 'logograms' is used, sometimes modified by ligatured signs or 'adjuncts' (mostly acrophonic abbreviations) @xcite . the ideograms are 143. for their representation a transcription is used, based on the abbreviation of the latin name of the represented object or being, e.g., vir 'man', mul(ier) 'woman'. additionally there are numbers that follow the decimal system and measurements units of weight and capacity @xcite . the assumption for the creation of both datasets was based on the clear separation of the signs of the linear b writing system into phonograms and nonphonograms. the phonographic part is made up of the syllables, while the non-phonographic component includes the ideograms sometimes modified by ligatured signs or 'adjuncts' (mostly acrophonic abbreviations), as well as signs for measurement units, numbers, and other marks, such as signgroup dividers or 'check-marks' @xcite . therefore, we decided to exclude the ideograms. ## supervised learning for infilling mycenaean tablets in the first place, we employ a symbol-level bidirectional recurrent neural network (brnn), (as has been employed in @xcite ) to fill in the gaps in the mycenaean tablets of series d and a&b. the goal in this case is to check if the predictions of the model improve by removing the ideograms from the mycenaean sequences. ## transfer learning for infilling series a&b in the following we investigate different transfer learning approaches for enhancing the model for series a&b (target) using the model learnt from series d (source). series d data involves more tablets and is syntactically richer while the data in series a&b is scarce. however, for the nlp standards both series are considered very scarce. we have essentially experimented with 2 simple tl techniques: