A Transformer-based parser for S yriac morphology (2023)

Abstract: In this project we train a Transformer-based model from scratch, with the goal of parsing the morphology of Ancient Syriac texts as accurately as possible. Syriac is still a low resource language, only a relatively small training set was available. Therefore, the training set was expanded by adding Biblical Hebrew data to it. Five different experiments were done: the model was trained on Syriac data only, it was trained with mixed Syriac and (un)vocalized Hebrew data, and it was pretrained on (un)vocalized Hebrew data and then finetuned on Syriac data. The models trained on Hebrew and Syriac data consistently outperform the models trained on Syriac data only. This shows, that the differences between Syriac and Hebrew are small enough that it is worth adding Hebrew data to train the model for parsing Syriac morphology. Training models on different languages is an important trend in NLP, we show that this works well for relatively small datasets of Syriac and Hebrew.

## introduction in this paper we develop a morphological parser for the syriac language. the trained model is able to segment graphical units into distinct words, it segments the morphemes within a word, and disambiguates morphemes and lexemes, all at the same time. syriac is a semitic language with a rich morphology. therefore, to add linguistic annotations to a text, it is better to encode the smaller parts of a word (morphemes) rather than the complete words. a complication is that the syriac language is written without vowels, which leads to the problem that a word can be parsed in different ways. furthermore, we only have a small syriac training set. therefore, we try to improve the model's prediction accuracy by adding biblical hebrew data to the training process. biblical hebrew is a semitic language that is closely related to syriac, and the training set that we have for this language is substantially bigger. since the late 1970s, the eep talstra center for bible and computer (etcbc) of the vrije universiteit amsterdam has developed and maintained a richly annotated dataset of the masoretic text of the hebrew bible. this dataset contains a wealth of linguistic features on the levels of words, phrases, clauses and larger text units. more recently, ancient texts in syriac have been prepared in a similar way. however, a vast corpus of syriac texts is available, and we hope to develop a faster approach to annotate these texts, because annotating them manually is a labor-intensive task. we have trained the transformer model in five different ways, to see which approach gives the highest accuracy on the syriac test set: a model trained on syriac data only, a model trained on a mix of (vocalized or unvocalized) hebrew and syriac data, and a model which is trained on (vocalized or unvocalized) hebrew data first and trained further on syriac data. a trained model can make predictions on "new" syriac texts, resulting in morphologically segmented texts. these results need to be corrected manually, and these corrected results can be processed further in a rule-driven way to produce the linguistic annotations. therefore, training the models is the first step in a longer pipeline. ## state of the art between 2000 and 2020 a number of studies were published in which natural language processing (nlp) tasks for semitic languages are described, often dealing with part of speech tagging (e.g., other studies deal with morphological analysis @xcite and segmentation @xcite . with the larger availability of digital (annotated) semitic texts and the advent of large, transformerbased language models, there is an acceleration in the development of models and tools for nlp tasks for semitic languages. a large language model which focuses on modern hebrew, is alephbert @xcite , which can be used for a number of tasks, including segmentation, part of speech tagging, full morphological tagging, named-entity recognition and sentiment analysis. a similar model for arabic, arabert, was developed by antoun, @xcite . relatively close to our research is a paper on adding diacritics to consonantal hebrew texts @xcite . it uses a combination of a machine learning ("several bi-lstm based modules") and a rule-driven approach ("comprehensive inflection tables and lexicons"). @xcite give an overview of developments in machine learning in relation to the hebrew language and its texts. a list of nlp resources for hebrew can be found here: an important trend in nlp is the development of multilingual models. these are models that can be used for a number of nlp tasks in various languages. some of these models are trained on one language, like english, and they can be trained further on other languages, but there are also models that are trained from scratch on a number of languages @xcite ). ## data our dataset consists of five files foot_1 , which are based on the etcbc database. the hebrew files that can serve as the input data for the model, contain vocalized or unvocalized text of the masoretic text (mt) of the hebrew bible. the hebrew output file contains the morphologically parsed mt. the text of these datasets is based on the fifth edition of the biblica hebraica stuttgartensia foot_2 . the syriac input file contains some books from the peshitta, a translation of the hebrew bible in syriac foot_3 @xcite and some non-biblical texts foot_4 . the syriac input texts are unvocalized, but they contain some diacritics, which can be found in the syriac manuscripts. each line in a data file contains one verse, and the text is represented in the etcbc transcription. the first line of the vocalized hebrew dataset, which is the first sentence of the hebrew bible, looks as follows: gen 1 1 b.:r;>cijt b.@r@> >:elohijm >;t hac.@majim w:>;t h@>@rey this line contains four tab-separated fields, with the following data: book, chapter, verse, and text. in hebrew script, the text, which means "in the beginning god created the heaven and the earth", looks as follows: all consonants, vowel signs and diacritics have a value in the transcription, e.g., ‫ב‬ is transcribed with b, ‫א‬ with >, qametz is transcribed with @, shewa with ":", and dagesh with ".". the transcription is read from left to right, unlike the text in hebrew script. the same line, but taken from the unvocalized dataset looks as follows: gen 1 1 @xmath0, but it misses the vowel signs. finally, the corresponding verse in the morphologically parsed output file looks as follows: gen 1 1 b-@xmath1:a the output dataset contains the same consonantal text as the input data, with a number of extra signs which indicate the morphological structure of the words: the dash (-) separates different words within a graphical unit. a word can have different morphemes, which are marked with special signs: after "[" follow verbal endings, and after "/" follow nominal endings. "+" initializes a pronominal suffix. between exclamation marks, one finds the verbal preformative, e.g., !j! in a 3 rd person masculine singular yiqtol, !t! in a 2nd person masculine singular yiqtol or !! in a qal infinitive or imperative. between closing square brackets one finds the prefix that is characteristic for a verbal stem, e.g. ]ht] for hitpael, ]n] for niphal, etc. "~" initializes a univalent final, for example, a ~h is a locative he. the etcbc approach of encoding morphology distinguishes between a paradigmatic form and a realized form of the morphemes. e.g., the paradigmatic form of the masculine plural marker is jm ( ‫-ים‬ in hebrew script). in several places in the mt, it is spelled as m ( ‫.)ם‬ here the j ( ‫,)י‬ which is part of the paradigmatic form, is not written. this is indicated in the encoding with an opening parenthesis. e.g., in genesis 17:20, one finds ‫נשיאם‬ ("princes"), which has the morphological encoding @xmath2, indicating that the j occurs in the paradigmatic plural form, but it is not realized. the opposite can also occur. if a character occurs in the text, but not in the paradigmatic form, it is preceded by "&". in the morphological encoding, there are some latin letters preceded by a colon: :a marks that a word is in absolute state. :c marks that a word is in construct state. :n marks the narrative vocalization of the waw. :d marks the d-stem. :u marks the u-a pattern of the passive. the "=" sign is used to disambiguate consonantal homographs, e.g., one distinguishes between kbd/ ( ‫ד‬ ‫,כָּבֵ‬ "heavy"), @xmath3,כָּבֵ‬ "liver"), and @xmath4,כֹּ‬ "heaviness"). the alphabets of syriac and hebrew are identical, also in the etcbc transcription, except that the sin ( ‫)שׂ‬ is lacking in syriac. the syriac dataset contains three different syriac diacritics: dots below and above the text, and seyame. a limitation of the present dataset is that for every word in the input, there is only one correct parsing in the output. in some cases, the text is ambiguous, and a word could be parsed correctly in different ways. a possible improvement of the dataset is to include alternative parsing options. ## data preparation we start with texts that do not have any parsing, which means that a text has not been segmented in phrases, clauses, or sentences. all verses of a book in the dataset are concatenated and split separately in shorter sequences of n graphical units. n is one of the required hyperparameters for training a model. these shorter sequences are partly overlapping and form a moving window. e.g., if the text is: @xmath5, the text will be split in the following three training inputs: @xmath6, a problem is that part of the sequences in the test set can also be found in the training set, which means that training and test set are not independent of each other. a possible solution is to select a few complete books as test set, but that leads to the problem that the language of these books may not be representative of syriac in general. therefore, we have used a different solution. if n is 5, the texts of 5 consecutive verses are grouped, and from all these groups of 5 verses, the validation and test set are selected. with this approach, it is guaranteed that the texts are long enough to extract at least one sequence of 5 graphical units, and they are short enough to split a book in many sequences, with the result that parts of the book can be found in the training, validation and test set, without overlap between these datasets. after this split, each sequence of 5 verses is split further in the partly overlapping shorter sequences of 5 graphical units. all short sequences that contain a case of ketiv/qere in the hebrew datasets are removed, because the consonantal text that is written (the ketiv) and the morphological analysis generally do not match. these words are indicated with a "*" in the data files. ## the model the morphological analysis is approached here as a sequence to sequence (seq2seq) problem, for which we use a transformer model foot_5 . the transformer is the state-of-the-art model for numerous nlp tasks @xcite and is also the basis of large language models like chatgpt and gpt4. the transformer seq2seq model has an encoder/decoder architecture. the encoder consists of a stack of encoder layers, in which the output of one layer serves as the input of the next one. each layer consists of two components: multi-head attention and a feedforward network. fundamental for the transformer model is the concept of self-attention, with which a word is related to all other words in a text sequence. in the self-attention mechanism, the embedding matrix of a sentence is multiplied with three randomly initialized matrices w q , w k , and w v , thus forming three new matrices q (query), k (key) and v (value). from these matrices, the attention matrix z1 is calculated as follows: z1 has the index 1, because this is the first attention head. there can be an arbitrary number of heads that are concatenated: after that, information of the word order in a sentence is added using positional encoding. the resulting matrix is fed to a feedforward network consisting of two dense layers with relu activation. just like the encoder, the decoder consists of a number of layers, one layer giving its output to the next one. the decoder of the transformer model starts with a start symbol and the representation of the sentence produced by the encoder, and from that the first word of the output after the start symbol is generated. then, the representation, the start symbol and the first word together are fed to the encoder, after which the second word is generated. this is done until a stop symbol is generated. in the present implementation, various hyperparameters can be tweaked, which can be found in the readme of the github repo. the only thing that we vary in the experiments described here are the number of epochs and the training datasets. the model is trained from scratch, which makes it possible to get a good impression of what the difference is between a model trained on syriac data alone, and a model that is trained on hebrew and syriac data. in all our experiments, the number of heads in the encoder is 8, and the number of encoder layers and decoder layers is 3. the feedforward hidden dimension is 512. during decoding we used beam search, with a beam size of 3. the length of the partly overlapping text sequences is 7 graphical units. ## results the model was trained with five different training strategies: the approach of two experiments is called transfer learning. in transfer learning, a model is trained first on a large dataset, after which the model is trained further on a smaller dataset for a specialized task. this is generally beneficial if there is only a small training dataset available for the specialized task, like in our case. in all the experiments, we varied the number of epochs in the main training loop (20, 25, 30, 35, and 40 epochs). we checked the accuracy of the predictions on the syriac test set, which is identical for each experiment. the accuracy is defined as the percentage of graphical units that is predicted fully correctly at a specific index of the test sequences. these test sequences are partly overlapping, just like the training sequences. therefore, for most words in the test set multiple predictions are made. the results can be found in figure 1 , which shows the results of the index with the highest accuracy. the accuracy of the model trained on syriac data increases with the number of epochs from 87.3% for 20 epochs to 89.3% for 30 or more epochs. the accuracy of the predictions of the models trained on hebrew and syriac data vary somewhat between 89.8% (20 epochs) and 90.8% (35 epochs), both achieved by the model trained simultaneously on unvocalized hebrew and syriac data. the models trained on hebrew and syriac data perform consistently better than the models trained on syriac data only. even though the accuracy of the latter models is only 1-2% higher, this is quite substantial, and it is hard to achieve this result by tuning hyperparameters. the hebrew datasets consisting of 22946 verses are substantially bigger than the syriac datasets (5596 verses) we used. therefore, training a model with hebrew data takes substantially longer, which may be a disadvantage for including this dataset, especially if one wants to optimize the model further by tuning the hyperparameters. so, as is often the case, there is a tradeoff between speed and performance. ## error analysis in the predictions on the test set, the model can make different kinds of mistakes. we provide a notebook in the github repository foot_6 , with which each mistaken prediction is classified as one of six error categories, with the goal of further improving the model. the following kinds of mistakes are distinguished: 0. parse errors in the encoding. in this case, the prediction is ungrammatical according to the parsing conventions. 1. the consonantal form of the prediction and the true surface form differ. 2. ungrammatical morpheme type combinations. this is the case if there is, e.g., a combination of verbal and nominal morphemes that do not match. 3. unparadigmatic morphemes. in this case the model predicts a morpheme that falls outside of the etcbc inventory of paradigmatic syriac morphemes. 4. difference in number of analytical words with the true form. in this case, the number of "-" signs in the graphical unit is incorrect. 5. difference in morphemes with the true form. in this case, the analysis of the word is grammatically correct, but not within the given context, there could for instance be an incorrect number of "=" signs at the end of the lexeme. the results are shown in figure 2 . it shows the results for the number of epochs with the highest accuracy. in general, the models show similar patterns. for every model, the most frequent type of error is 5, which means that the parsing is grammatically correct, but not in the given context. the error types 0 and 2 hardly occur. in most error categories, the model which was trained on syriac only has more errors than the other models. an important difference between this model and the other models is found in error type 1, indicating errors in the surface text, where the model trained on syriac has 2-3 times more errors than the models trained on both hebrew and syriac data. the consonantal text of the input and output should be identical, and this is language independent. this is a clear sign that adding the hebrew data helps here, simply because the volume increases. the same may be true for the error categories 3, 4, and 5. here and there, the hebrew may help because a morpheme is the same as in syriac, but it is likely that it helps mostly because it adds volume to the dataset, which helps to make the model more consistent in analyzing morphemes. ## conclusions in this paper we trained a transformer model from scratch with the goal of analyzing syriac morphology. an important part of the research was to see if adding hebrew to the training set would improve the accuracy of the predictions on the syriac test set. we compared results of the models that were trained on syriac data alone, models that were trained on (un)vocalized hebrew and then trained on syriac, and models that were trained on (un)vocalized hebrew and syriac simultaneously. the highest accuracy of the model trained on syriac data was 89.3%. the best model overall was trained on unvocalized hebrew and syriac simultaneously with an accuracy of 90.8%, which outperforms the best "syriac only" model with 1.5%. further improvements can possibly be achieved by optimizing the hyperparameters of the models, but it is clear that adding hebrew data to the training set helps with improving the performance on the syriac test set. the same effect may be expected with a larger syriac dataset, but as long as that dataset is relatively small, adding hebrew data is a good solution. another way to expand the dataset is to use data augmentation, which we are considering for future experiments. it has been shown in other tasks that a model trained on a variety of data can be very useful to be trained further for specialized tasks. in our project we see the same phenomenon. the experiment could be broadened in various ways. one could for instance use one of our models and train it further on data from other languages than hebrew and syriac, such as akkadian or arabic, or train models to parse syriac texts syntactically.