Universalising L atin U niversal D ependencies: a harmonisation of L atin treebanks in UD (2023)

Abstract: This paper presents the harmonisation process carried out on the five treebanks available for Latin in Universal Dependencies, with the aim of eliminating the discrepancies in their annotation styles. Indeed, this is the first issue to be addressed when parsing Latin, as significant drops in parsing accuracy on different Latin treebanks have been repeatedly observed. Latin syntactic variability surely accounts for this, but parsing results are as well affected by divergent annotation choices. By analysing where annotations differ, we propose a Python-based alignment of the five UD treebanks. Consequently, the impact of annotation choices on accuracy scores is assessed by performing parsing experiments with UDPipe and Stanza.

## introduction a significant number of resources is available for latin. with respect to syntax, notable are the five treebanks in universal dependencies 1 (de @xcite , which represent a remarkable amount of data. here is an overview: the treebanks highly differ in terms of included texts and size (see this study aims to syntactically harmonise the five latin treebanks, as well as to assess the impact of different annotation choices on parsing accuracy. section 2 motivates the present study. section 3 presents an overview of the alignment process, while in section 4 the harmonising interventions are highlighted in more detail. section 5 reports the parsing scores on the aligned treebanks, demonstrating the impact of diverse annotations on parsing. finally, section 6 presents the conclusions and future research directions. ## related work and motivation parsing accuracy scores on latin texts drop significantly when a model is applied to data that differ from those it was trained on. the issue is of course more general and concerns out-of-domain data, but with respect to latin it is strongly intertwined with the issue of its syntactic variability. indeed, spread over a span of more than two millennia and all across an area that corresponds to europe, the latin language has undergone a number of significant changes, which affected the syntactic layer as well. to be able to investigate genuine syntactic diversity, first we have to ask how much the observed drop in parsing performance is due to divergent annotation styles. a deeper understanding, and possibly levelling of such divergences would allow to isolate the impact of annotation choices and highlight intra-linguistic syntactic variability. such syntactic diversity, leading to lower parsing accuracies, has been repeatedly noted. for instance, @xcite and ponti and passarotti (2016) observed how performances drop when a model is employed to parse out-of-domain data, while @xcite dealt with the need of adapting a pre-existing parser to the specific processing of medieval latin. the issue of latin variability has also been addressed in the evalatin campaigns @xcite , devoted to the evaluation of nlp tools for latin. 2 on the other hand, the issue of inconsistent annotations is not unprecedented. methods for inconsistency detection in treebanks have been proposed e.g. by @xcite , @xcite , de @xcite , @xcite , and @xcite . with respect to latin, a huge effort towards harmonisation has been made by the lila project foot_1 @xcite . within the framework of linguistic linked open data, lila seeks to overcome the different lemmatisation criteria through a pivotal use of lemmas and hypolemmas in a knowledge base. ## alignment process for the alignment process we decide to model our interventions on the 2.10 version of the udante treebank, which was released in may 2022. this choice is motivated by several factors: as mentioned in section 2, the interventions mainly focus on dependency relations, yet not exclusively: spotted conversion and random errors are corrected, as well as some inconsistencies in terms of lemmatisation and pos tags. as a starting point of our alignment process, we choose the treebanks' train, dev and test sets as available in their ud github dev branch as of august 30th, 2022. the treebanks are then aligned through python scripts, specifically designed for each treebank. to manipulate data we exploit udapi @xcite , a framework providing an application programming interface for ud data. our scripts are openly available on github, 6 together with the aligned treebanks. moreover, we are ready to contribute the harmonised treebanks to the official ud releases. ## treebank investigation an overview of the current state and our modifications of the treebanks is presented in the following subsections. further information can be retrieved directly from the scripts available in github. ## impact on parsing afterwards, we try to assess the impact that a harmonised annotation of the five treebanks has on parsing accuracy. in order to achieve this, with both udpipe and stanza we retrain a model for every aligned treebank. we then test the obtained models on each of the treebanks; tables 4 and 6 summarise the scores, in terms of labeled attachment score (las) and unlabeled attachment score (uas) @xcite , obtained with models trained with udpipe and stanza respectively. to measure accuracy, we employ the python evaluation script foot_13 designed for the conll 2018 shared task on multilingual parsing from raw text to universal dependencies @xcite . as mentioned earlier, the script takes into account only main dependency types, without considering subtypes. this reflects our current needs; nevertheless, the present treebank alignment is only the first stage of a larger harmonisation effort, and additional evaluation criteria (including relation subtypes) can be introduced in the future. to demonstrate the effect of harmonisation, we also present las and uas scores of models trained on pre-harmonisation data (tables 3 and 5 ), again with udpipe and stanza. such models are trained and tested on master data of universal dependencies 2.10, officially released in may 2022. both series of models, pre-and post-alignment, are trained with the same settings. with respect to udpipe, version 1.2 is used; we employ pretrained fasttext embeddings 16 @xcite and optimised training hyperparameters as described for reproducible training by @xcite , within the publication of ud 2.5 models for udpipe. since optimised hyperparameters are available only for ittb, perseus and proiel, for llct and udante we experiment with different options and select the best ones. 17 as for prealignment models for stanza, we employ the ittb, perseus and proiel models made available foot_16 by the stanza team and pretrained on ud 2.8, since those treebanks did not change afterwards, as reported in their change log. we train pre-alignment models for llct and udante, as well as all postalignment ones, with default parameters and fast-text embeddings. some conclusions can be drawn from the comparison of the tables. with udpipe, the interventions prove effective in most cases, as models trained on harmonised treebanks reach higher scores than the pre-alignment ones. this holds true especially with respect to results on perseus and proiel; indeed, each of the post-alignment models gains higher scores on these two treebanks. the improvement is substantial (up to +9% with more than one model), and confirms once more the absolute relevance of a truly universal annotation style. higher impact on perseus and proiel is expected, given their previous condition (section 4). ittb and llct models show a less consistent behaviour, performing sometimes better (i.e. on perseus and proiel), sometimes marginally worse (e.g. ittb model on ittb and llct test data). a closer analysis of the parser outputs, despite not providing a precise explanation for the parser behaviour, reveals that the harmonisation can be further enhanced. for instance, it emerges that the harmonisation of copular constructions, as discussed in subsection 4.3, 20 did not catch all occurrences and the wrong original annotation 19 llct represents an exception. 20 see the example from ittb: successio autem propter motum aliquem est. the general trend of improved scores can be observed also when models are trained with stanza. yet, the increase is not as considerable as when udpipe is employed. however, tables 3 , 5 , 4 and 6 also highlight how the treebank annotation alignment does not solve the issue discussed in section 2: the drop is still significant when data are parsed with models trained on a different treebank. moreover, the absolute scores presented depend also on the size of training data, which varies substantially across the treebanks (see ## conclusion and future work the annotation alignment proposed in the present paper confirms the relevance of a shared and universal annotation scheme. thus, although the universal dependencies project already represents an outstanding milestone, the effort needed in this direction is still remarkable, and two-fold: on the one hand, treebanks should be constantly updated to the latest ud guidelines, as they keep developing towards a more consistent annotation formalism. on the other hand, different research teams working on the same language should collaboratively define shared guidelines and adopt the same approach in annotation, so that universal dependencies can grow more and more universal. many future directions can be envisaged for this study. the alignment needs to be further inves- once the treebanks follow a more uniform annotation style, it will be possible and appropriate to investigate the actual linguistic differences causing performance drops when models trained on one treebank are applied to another. possible directions for this future work include an analysis of genre diversity, a closer examination of different types of employed embeddings, and exploitation of latin bert @xcite . the results could lead to the definition of strategies to overcome the issue of latin syntactic variability.