L atin Morphology through the Centuries: Ensuring Consistency for Better Language Processing (2023)

Abstract: This paper focuses on the process of harmonising the five Latin treebanks available in Universal Dependencies with respect to morphological annotation. We propose a workflow that allows to first spot inconsistencies and missing information, in order to detect to what extent the annotations differ, and then correct the retrieved bugs, with the goal of equalising the annotation of morphological features in the treebanks and producing more consistent linguistic data. Subsequently, we present some experiments carried out with UDPipe and Stanza in order to assess the impact of such harmonisation on parsing accuracy.

## introduction in universal dependencies @xcite five treebanks are available for latin: 1 index thomisticus treebank (ittb; @xcite , late latin charter treebank (llct; @xcite , perseus @xcite , proiel @xcite , udante @xcite . these treebanks differ on multiple levels. first, they cover different domains: a shallow distinction can be made between poetry (found in perseus and, less, in udante) and prose (all treebanks), but it can be further specified in terms of specific genre included. for instance, ittb encompasses philosophical texts, while llct consists of charters, representing an instance of documentary genre. additionally, the history of latin, spoken for over two millennia, entails a substantial diachronic variation, as the language gradually evolved over time. indeed, the five latin treebanks include data that differ substantially in this respect. already considering the medieval treebanks alone, we can observe how wide the covered time range is: ittb encompasses medieval texts dating back to xiii century, llct features early medieval charters (viii-ix century), 1 see https://universaldependencies.org/ . while dante alighieri's work available in udante belongs to xiv century. in addition to that, perseus and proiel include classical texts (i bc -iv ad), as well as the vulgate (iv century). a level of spatial variability can be observed too; for instance, llct includes texts written in tuscany, italy, and some features typical of the romance languages are already emerging. in addition to the aforementioned levels of variability, and besides variation in size, latin treebanks also differ in terms of annotation choices, in spite of the ud work towards consistency. this issue can be doubly problematic: first with respect to ud itself, as the annotation is expected to be consistent across and within languages; secondly, in light of the fact that the quality of data may affect the results of any experiment or linguistic investigation carried out on those data. @xcite , investigating parsing performances, already observe this as regards the syntactic layer of these data. nevertheless, what has been observed with respect to syntax does not necessarily apply to morphological features as well, and the extent to which inconsistent morphological annotation affects parsing performances thus remains unclear. for this purpose, we first propose a harmonisation of the morphological features of the five treebanks, and thereafter assess its impact on models predicting morphology, as well as syntactic parsers. section 2 presents some related work and the motivation behind our study. section 3 features an overview of the harmonisation process, while in section 4 we describe the strategy designed to spot inconsistent or missing annotations. section 5 highlights the main harmonising interventions, whose impact on parsing accuracy is assessed in section 6. finally, section 7 concludes the paper and suggests future research directions. ## related work and motivation any nlp task is likely to show degraded performance when a model is applied to data that differ from training data. it has been observed several times that this issue is particularly prominent in (morpho-)syntactic parsing of latin texts. the issue is strongly intertwined with latin intralinguistic variability, as the language has undergone a number of significant changes by spreading over a period of more than two millennia and across europe. in order to investigate genuine linguistic diversity, first and foremost the impact of divergent annotation styles has to be ruled out. to perform any experiment that exploits data, we need those data to be consistent. harmonising such discrepancies would allow for the isolation of the impact that annotation choices have, so that actual intralinguistic variability can emerge and be examined. the issue of latin variability has been addressed in the two evalatin campaigns @xcite , aiming to evaluate nlp tools for latin. in particular, evalatin has been focusing on lemmatisation, morphological analysis and pos tagging. however, latin diversity has been observed several times already before, in light of the behaviour of parsing accuracy, which was far from being homogeneous. see, for instance, @xcite , @xcite , @xcite . several studies have also been addressing the issue of inconsistent annotations. @xcite , @xcite @xcite , aggarwal and zeman (2020), and @xcite are only some of the methods that have been proposed to detect inconsistencies in treebanks. @xcite present a harmonisation of dependency relations in latin treebanks, yet without intervening at the level of morphological features. their harmonisation highlighted several levels of inconsistencies and proved to lead to substantial improvements in terms of parsing accuracy. we investigate whether similar improvements can be achieved by also addressing inconsistencies in morphological annotation. the output of the present study is two-fold: ## overview of the harmonisation process the focus of the harmonisation process is exclusively on morphological features. we define the workflow to detect inconsistencies and missing features as follows. first, we run the udapi block on the input data, with the goal of spotting features which are either required but missing, or not allowed. as output, the trees that feature either of these two kinds of inconsistencies are stored in a html file, where those bugs are prominently highlighted (see figure 1 ). in light of the output html file, we build python scripts that address and fix the observed bugs. we employ the harmonised version of the five treebanks, as made available by @xcite , as input. nevertheless, differently from what was done for syntactic harmonisation, we do not strictly follow udante annotation. this choice is justified by the fact that we observe a considerable difference in the set of morphological features employed in udante -predominantly -and the other treebanks (ittb and llct) maintained by the same developers, i.e. the team at università cattolica del sacro cuore in milan, italy, as opposed to the two remaining treebanks (perseus and proiel) out of the five available for latin. we thus decide to define two levels of coherence: by default, the block operates at the lower level, but a parameter can be supplied to udapi, which will trigger the more detailed features. morphologically harmonised treebanks and harmonisation scripts are available on github, foot_0 while the block is available in udapi github repository. moreover, we are ready to contribute the harmonised treebanks to the official ud release. ## the markfeatsbugs block the markfeatsbugs block is structured as follows. for each upos tag, a set of required features is first defined. (note that the official ud validator foot_1 has some limited ability to check permitted uposfeature combinations, but not to enforce required features.) additional features that are permitted but not required are listed, and for each permitted feature the set of its permitted values is defined. unlike in the official ud validator, the conditions for a feature-value to be permitted or required are not limited to whole upos categories. for example, the ud validator knows that the person feature is allowed for verbs and auxiliaries; but we further restrict it to finite forms, i.e., the feature verbform must be present and its value must be fin. the set of allowed features is then expanded to include additional feature-value pairs that may be found in udante, ittb or llct (higher level of detail). eventually, the block checks for each node whether its morphological features are permitted and if every node has all the required features. if not, invalid and missing features are explicitly marked with a transparent label allowing to easily distinguish them, and saved in the bug attribute in the misc column of the conll-u file. it can be later used in filters and highlighted in the data. the code snippet in script 1 provides an example, although not exhaustive, of the block section concerning verbs and auxiliaries, in compliance to what has been implemented in the treebanks among all the proposals illustrated in cecchini (2021). script 2 if re.match(r'^(verb|aux)$', node.upos): @xmath0, 'aspect'] @xmath1: ['inf', 'fin', 'part', 'conv'], ↩→ 'aspect': ['imp', 'inch', 'perf', 'prosp']} ↩→ if node.feats['verbform'] not in ['part', 'conv']: ↩→ rf.append('tense') af['tense'] = ['past', 'pqp', 'pres', 'fut'] ↩→ if node.@xmath2.@xmath3.lemma != 'sum'): ↩→ ↩→ rf.append('voice') af['voice'] = ['act', 'pass'] if node.feats['verbform'] == 'fin': rf.extend(['mood', 'person', 'number']) ↩→ af['mood'] = ['ind', 'sub', 'imp'] ↩→ af['person'] = ['1', '2', '3'] af['number'] = ['sing', 'plur'] elif node.feats['verbform'] == 'part': ↩→ rf.extend(['gender', 'number', 'case']) ↩→ af['number'] = ['sing', 'plur'] if node.misc['traditionalmood'] != 'gerundium' else ['sing'] ↩→ ↩→ ↩→ af['gender'] = ['masc', 'fem', 'neut'] if node.misc['traditionalmood'] != 'gerundium' else ['neut'] ↩→ ↩→ ↩→ af['case'] = ['nom', 'gen', 'dat', 'acc', 'voc', 'loc', 'abl'] ↩→ ↩→ af['degree'] = ['abs', 'cmp'] if node.misc['traditionalmood']. startswith('gerundi'): ↩→ af['voice'] = ['pass'] af['aspect'] = 'prosp' elif node.feats['verbform'] == 'conv': ↩→ rf.extend(['case', 'gender', 'number']) ↩→ af['case'] = ['abl', 'acc'] af['gender'] = ['masc'] af['number'] = ['sing'] af['voice'] = ['act'] elif node.feats['verbform'] == 'inf': ↩→ af['tense'].remove('pqp') script 1: portion of the block that partially exemplifies how morphological features are checked for the verbal system: rf stands for 'required features', af stands for 'allowed features'. script 2: richer, more detailed morphological features as allowed by the relevant parameter if set to 1. the most representative example is inflclass, foot_2 which reflects the original endings of the proto-indo-european stems. inflclass has not been added everywhere in udante, therefore -when higher-level validation is turned on -it is only considered as allowed, instead of required. ## harmonisation examples three treebanks have been harmonised to the higher level of detail (as defined in the previous sections): llct, ittb and udante. the remaining two treebanks (perseus and proiel) have been harmonised to the lower level because the highlevel annotation is not available for them. the harmonisation process derives transparently from the feature constraints in the udapi block. it would not be helpful to discuss every constraint in detail here (and if necessary, the reader can refer directly to the source code of the block); nonetheless, we want to discuss some interesting examples regarding verbs and auxiliaries. there is a more general issue raised by cecchini (2021), who proposes a reorganisation of latin non-finite verbal features towards a higher degree of universality. in accordance with their proposal, foot_3 we reannotate all gerund and gerundive forms as participles (@xmath4. traditional terminology used in grammars, i.e. gerund and gerundive, is saved in the misc field as @xmath5. similarly, supine forms are reannotated as @xmath6. the use of traditionalmood and traditionaltense is extended to finite forms as well, for the purpose of consistency and in line with udante. as far as finite forms are concerned, auxiliaries occurring in ittb require some intervention as well. unlike in the other treebanks, such forms (e.g. sum 'they are') do not present aspect, mood, person and tense. for the sake of consistency, we annotate them with respect to those features, assigning the relevant value. overall, the examinations of bugs highlighted by the block confirms what has been already noted in @xcite with respect to perseus and proiel status: their level of annotation detail is remarkably lower in comparison to ittb, llct and udante. an outstanding example is provided by prontype, which is a key feature for pronouns and determiners. often missing in particular in perseus, it is systematically added during the harmonisation process. additionally, the block can also serve as a tool to spot isolated errors. whenever such errors are highlighted, we proceed to correct them. ## impact on parsing to evaluate the significance of the harmonisation process of morphological features, we try to investigate its impact on parsing accuracy. therefore, we train new models for every morphologically harmonised treebank. the models are trained on the same data, but in the first case udpipe 1.2 is used, while for the second one we choose to employ stanza. with both stanza and udpipe we train the parser model on predicted lemmas and tags. indeed, through stanza's prepare_depparse_treebank.py script, 6 the trained pos tagging model is used to retag the training data before training the parser. similarly, for udpipe foot_5 we train a parsing model that relies on lemmas, upos tags and features as generated by the tagger. we use pretrained fasttext embeddings foot_6 @xcite and training hyperparameters as used for syntactic harmonisation in @xcite . for udpipe, these hyperparameters correspond to the optimised ones ittb llct perseus proiel udante notes aspect 26,243 4,596 4,344 35,420 -aspect is added. gender_n 2,655 9,746 1,037 11,756 -gender is added, corrected or deleted (nominal only). gender_v 1,514 1,834 30 3,899 -gender is added, corrected or deleted (verbal only). gerund(ive)s 2,740 1,855 91 1,046 -interventions on gerunds and gerundives. mood 20,269 ----mood is added. number_v 21,783 1,834 30 322 -number is added (verbal only). @xmath7,029 2,415 162 1,671 142 @xmath8. person 20,269 ----person is added to verbs. person_p --1,346 15,887 -person in pronouns is either added, if missing, or deleted, if not relevant. prontype 24,825 21,062 3,105 31,023 21 prontype is either added, if missing, or corrected. tense 51,096 10,988 1,277 9,430 -tense is either added, corrected or deleted. voice 2,591 1,855 216 1,064 -voice is added when missing. voice_no -4,113 369 7,848 -voice is deleted when not relevant. udante: @xmath9.01, @xmath10, @xmath11, @xmath12. 10 in both cases parsing models are trained on predicted tags. raw text to universal dependencies @xcite , which takes into consideration main dependency relations only and not subtypes. first and foremost, a clarification is necessary. as explained earlier, the treebanks are not forced all to the same set of features: llct, ittb and udante have some extra features that are not found in perseus and proiel. it would be possible to remove these extra features for the sake of parsing evaluation but we chose to keep them. one can thus expect somewhat worse results when applying models from one of these treebank groups to test data from the other group. as illustrated in the tables, the results do not show any clear pattern and, overall, the improvements are neither widespread nor substantial. a closer look at the scores reveals that udpipe shows improved accuracy scores in less than half of the cases, and in general performs worse than stanza, with the gap being almost around 10% on average. improvements obtained with models trained on udpipe are never substantial and, in general, very hard to interpret. stanza seems to allow for some additional remark. we first want to examine distinctly the two groups that correspond to the two possible values of the discussed parameter. the ## conclusion and future work the paper presents the harmonisation process that we carried out, with respect to morphology, on the five latin ud treebanks. we first defined an udapi block for latin, listing which morphological features a token should possess. such lists of features are defined based on upos tags. subsequently, we corrected the retrieved inconsistencies -consisting in either missing or not allowed features -via python scripts. as a result, we produced morphologically harmonised versions of the latin treebanks that were previously harmonised syntactically @xcite . we contributed the script to investigate latin features, possibly reusable by anyone working on latin treebanks, and we described a workflow that can be replicated and applied to potentially any other language, provided that language-specific information is supplied within the template. in the second part of the paper, we presented some parsing experiments carried out with udpipe and stanza. by comparing syntactic attachment scores before and after morphological harmonisation, we observed the absence of a clear pattern that would allow to explain results; on the other hand, morphological accuracy clearly improved. the coexistence of a coarsegrained and a fine-grained level of consistency in annotation partially explains the outcome of the parsing experiments, that however must not discourage from pursuing an ever-growing harmonisation of linguistic resources in terms of annotation choices. intra-and inter-resource consistency is a key factor to exploit data, whether it comes to in light of the slight improvement that resulted in parsing accuracy from the harmonisation process, we do not plan on further developing the harmonisation of treebanks. the higher degree of consistency in treebank annotation, i.e. the availability of more homogeneous data, allows now to investigate the actual reasons for variability in parsing. syntactic constructions evolving over time may be inspected, as well as other factors that may affect parsing results on data that differ from training data -as already problematised several times, e.g. by @xcite , @xcite , @xcite . variation in time is most probably expected to be a relevant factor, and it is strongly connected to two other relevant variables, i.e. space and domain. consider, for instance, the late latin charter treebank: while featuring early medieval latin (viii-ix century), not as late as ittb (xiii century) and udante (xiv century) latin varieties, the treebank does not include literary texts yet charters written in tuscany, italy. the gradual development of latin towards romance languages, exemplified by evolving syntactic constructions and changes in word endings, can already be observed in the treebank @xcite . variation in terms of genre appears to be relevant also with respect to the distinction between poetry and prose. with latin treebanks encompassing mostly literary data, such distinction cannot be overlooked. indeed, latin poetry is strongly affected by prosody and metre: the sequence of short and long syllables in words, as defined by prosodic rules, together with the specific structure of the selected metre, rigidly determine possible sequences of words. as a result, the natural word order is unsettled, and the position of a word in the verse (and, hence, in the sentence) is mostly defined by the way its short and long syllables follows one another. this whole mechanism, highly affecting word order, entails a high degree of non-projectivity, and would need to be further inspected.