Improving L atin Dependency Parsing by Combining Treebanks and Predictions (2024)

Abstract: This paper introduces new models designed to improve the morpho-syntactic parsing of the five largest Latin treebanks in the Universal Dependencies (UD) framework. First, using two state-of-the-art parsers, Trankit and Stanza, along with our custom UD tagger, we train new models on the five treebanks both individually and by combining them into novel merged datasets. We also test the models on the CIRCSE test set. In an additional experiment, we evaluate whether this set can be accurately tagged using the novel LASLA corpus (https://github.com/CIRCSE/LASLA). Second, we aim to improve the results by combining the predictions of different models through an atomic morphological feature voting system. The results of our two main experiments demonstrate significant improvements, particularly for the smaller treebanks, with LAS scores increasing by 16.10 and 11.85%-points for UDante and Perseus, respectively (Gamba and Zeman, 2023a). Additionally, the voting system for morphological features (FEATS) brings improvements, especially for the smaller Latin treebanks: Perseus 3.15% and CIRCSE 2.47%-points. Tagging the CIRCSE set with our custom model using the LASLA model improves POS 6.71 and FEATS 11.04%-points respectively, compared to our best-performing UD PROIEL model. Our results show that larger datasets and ensemble predictions can significantly improve performance.

## introduction in recent years, significant progress has been made in morpho-syntactic dependency parsing for latin, an advancement that greatly benefits a wide range of research in the humanities. linguistically tagged corpora are crucial, as lemmatized corpora, for instance, are valuable also for historians searching for sources within databases. the universal dependencies (ud) framework plays a key role by organizing linguistic analysis into machine-readable databases with columns in tab-separated value tables. these conll-u formatted treebanks provide essential information on lemmas, parts of speech, morphological features, syntactic roles, and dependency relations. in the realm of latin treebanks notable recent developments include the morphological harmonization of the five largest latin treebanks (ittb, llct, perseus, proiel, and udante foot_0 ), a significant milestone reached by @xcite as a continuation of earlier work on syntactic harmonization @xcite . additionally, there have been many efforts to enhance the performance of latin parsing tools. these include the evalatin campaigns @xcite , as well as the application of gpt models for part-of-speech (pos) tagging @xcite . despite these advancements, there remains potential for further improvement, particularly in syntactic parsing. for instance, the highest labeled attachment score (las) reported by @xcite is 64.87% for the udante and 59.43% for perseus. in the present study, we leverage the recently released harmonized treebanks @xcite to further enhance automatic parsing. our focus is on the five largest established treebanks in the ud format, ensuring that our results are reliably comparable to previous studies. our models can also easily be applied to parse new text corpora. to achieve our goal, we employ two approaches: first, we train new parser models using these harmonized treebanks, along with two state-of-theart parsers -stanza @xcite and trankit @xcite -as well as a custom ud tagger by fine-tuning a bert-based latin language model (ströbel, 2022) following the architecture of @xcite . the parsing models are trained using both individual and diverse merged treebanks. second, we investigate whether combining predictions from our newly trained models in a voting system targeting part-of-speech (pos) and morphological features (feats) tags improves performance. our hypothesis is that selecting the most common prediction from the different models enhances the results in a 'majority vote wins' scenario. third, we use the voting setup of the different models to analyze how unanimous the various parser models are in their pos predictions. this provides insight into which tasks are accurately tagged and offers potential for identifying prevailing issues in the annotation guidelines. upon the publication of this paper, 2 all data, code, and results, as well as the models, will be made openly and freely accessible for noncommercial use. these resources include clear instructions, designed to be easily used by scholars who may not be familiar with language technology but wish to experiment with their own texts. ## previous work the first latin bert model by @xcite provided the state-of-the-art pos scores of its time (perseus 94.3%, proiel 98.2%, ittb 98.8%). similarly, @xcite reported very competitive las scores for the previous releases of the treebanks using a biaffine parser on top of a latin bert (ittb 92.99%, proiel 86.34% and perseus 80.16%). there have been some trials with merging existing treebanks into larger training datasets. @xcite combined the ittb, perseus, and proiel treebanks, while @xcite trained a single model for all ancient languages, including three latin treebanks. additionally, @xcite combined all the ud treebanks into a single multilingual dataset and trained a model for all ud languages. while these studies demonstrated the potential for improving performance by merging training data from multiple treebanks, the first reports only a single experiment, and the latter two do not focus specifically on latin, leaving room for further experiments. the challenge of selecting and combining treebanks is also brought to attention in the latest evalatin campaign @xcite . merging treebanks for training models has not 2 https://github.com/hannakoo/latin-parsing been widely explored, likely because the developers of the treebanks have varied interpretations of the ud guidelines since the treebanks have been composed at different points in time (with continuous updates regarding the annotation guidelines). these discrepancies in annotations has complicated combining them into larger merged training datasets. the work of @xcite focuses on the harmonisation of the datasets, and they train models using only the individual treebanks. combining the predictions of several models through voting has been tested in many studies. e.g. early pioneering work by @xcite applied majority voting for four parsers for czech, reporting improvements of 2%-points in dependency relation prediction. combining parser outputs has also been used by @xcite to improve the parsing of the ittb treebank. more recent work by @xcite developed an ensemble classifier by applying a voting model on top of several pos taggers. their voting model was designed to learn which predictions to trust in different contexts. ## data there are five latin ud treebanks used for training: the index thomisticus treebank (ittb) @xcite , the late latin charter treebank (llct) @xcite , perseus @xcite , proiel @xcite , and udante @xcite ). for a concise numerical comparison of these latin ud treebanks and a detailed description of their contents, see foot_1 . for a general overview, see @xcite . the efforts of @xcite are crucial for merging the treebanks and serve as a foundation of our model training. these harmonized treebanks are accessible at a githubrepository foot_2 . for a concise numerical overview and a brief description of the treebanks used in this study, refer to ## methods in our aim to improve morpho-syntactic parsing tools for latin, we use two different methods: training new models and experimenting with a voting system. our first task is the training of new parser as neither trankit nor stanza support the usage of a custom pretrained language model, we also experiment with a custom part-of-speech and morphological tagger trained on top of a monolingual latin language model (ströbel, 2022) following the task-specific fine-tuning of @xcite . ## results the performance of the trained models is summarized in the complete set of metrics is available on the project's github page 9 and the all conll-u formatted treebanks respectively foot_8 . in this paper, we report and discuss the scores for tokenization, pos, morphological features (feats), lemmatization, and syntax, including both the unlabeled attachment score (uas) and labeled attachment score (las). for the custom tagger, only the upos and feats results are relevant. all metrics were generated using the ud evaluation tools, based on the conll 2018 shared task script 11 . our state-of-the-art results demonstrate improvements in pos-tagging of 8.41 %-points for perseus, 7.78 for proiel, and 5.93 for udante compared to the findings of @xcite . additionally, our results show an improvement in las of 11.85%-points for perseus and 16.10%-points for udante compared to @xcite . all numerically highest f1 scores achieved by the models are in the ## conclusion and future studies the task of full morpho-syntactic parsing across the five largest established treebanks comprises 30 subtasks, of which 8 are best performed by the trankit five merged model. this model demonstrates particular strength in part-of-speech labeling. additionally, stanza's lemmatization capabilities are noteworthy, consistently achieving the highest numerical values across all five treebanks. overall it can be stated that merging the available five latin ud datasets is very beneficial especially when it comes to smaller treebanks and out of domain parsing. with our experiments, by using thematically compiled and everything merged datasets, we are able to set a new state of the art for many morpho-syntactic parsing tasks. the average improvement of our final results are reported in future studies should first focus on addressing the issues related to the treatment of multi-word tokens. one approach could involve ensuring that the five established treebanks strictly adhere to current guidelines, such as avoiding the splitting of enclitics (e.g., -que 'and') into separate tokens. additionally, the introduced voting system could be further refined and applied to a gold-standard pretokenized input, followed by a detailed numerical error analysis and close reading. this enables determining the specific morphological annotation tasks that our current models succeed upon. such analysis could also determine whether observed errors on one hand, many tasks are successfully accomplished using a single treebank for training, development, and testing, as demonstrated by the ittb data, which does not require the inclusion of additional treebanks for improving performance. this highlights the importance of incorporating new genres across a broad time span into the ud latin treebank family, ensuring that the training data is sufficiently diverse, comprehensive and large enough. while the development of novel gold-standard annotated datasets offers significant benefits, it is also highly demanding in terms of human resources. we hope that our high-performing models will facilitate the annotation of these datasets by providing accurate predictions that serve as a strong starting point for manual corrections, thereby easing the process. on the other hand, one of the conclusions drawn from our diverse merged training sets is that the notion of "latin is latin" does not hold true. it is well established that medieval latin is distinctly different from classical latin. in practical terms most scholars often identify themselves as experts in one or the other. however, a possible future study could investigate the specific attributes in a treebank's training data that make a parser model particularly adept at classical or medieval latin. another conclusion from our experiments is that the accuracy of parsing latin from the classical period (broadly defined) is diminished when the model is exposed to medieval training data. this warrants further exploration to define the characteristics that distinguish the two and will shed more light into computational historical linguistics. one study could be the evolution of medieval latin and the extent to which medieval treebanks reflect preserving features of classical latin, analyzed by auctor and decade. it might reveal how well and what ways medieval writers were competent in classical latin. another potential research direction is to investigate why parsing the udante treebank appears less selective, with all five merged models performing well. this raises the question of whether users of latin from this late medieval period were equally accustomed and influenced by reading both classical and medieval authors. alternatively, this phenomenon might be explained by the size of the training data, where additional examples contribute to improved results, as our lasla experiment in the circse test set show.