L atin Treebanks in Review: An Evaluation of Morphological Tagging Across Time (2024)

Abstract: Existing Latin treebanks draw from Latin’s long written tradition, spanning 17 centuries and a variety of cultures. Recent efforts have begun to harmonize these treebanks’ annotations to better train and evaluate morphological taggers. However, the heterogeneity of these treebanks must be carefully considered to build effective and reliable data. In this work, we review existing Latin treebanks to identify the texts they draw from, identify their overlap, and document their coverage across time and genre. We additionally design automated conversions of their morphological feature annotations into the conventions of standard Latin grammar. From this, we build new time-period data splits that draw from the existing treebanks which we use to perform a broad cross-time analysis for POS and morphological feature tagging. We find that BERT-based taggers outperform existing taggers while also being more robust to cross-domain shifts.

## introduction large-scale digitized latin archives now document cultures across many centuries in wide a variety of genres from literature to legal documents. with increasingly powerful latin natural language processing tools (e.g. @xcite , morphological feature tagging is a promising method for latin-based computational humanities. the goal of morphological tagging is to identify a set of morphological feature-value pairs for each token of a given sentence. these features can help researchers analyze agency, power, and other morphosyntactically-signalled phenomena which have been fruitfully investigated in english @xcite and other languages @xcite . for example, voice (active, passive verbs) and case (e.g., nominative, accusative ablative nouns) are useful for studying power and agency. although latin taggers have relatively good performance, in our experience they often perform poorly on rarer feature values-such as passive voice-that may prove crucial for downstream analyses. toward this end, we hope to develop a latin morphological tagger whose accuracy is robust across time and genre by leveraging the recent development of five separate latin universal dependencies (ud; de @xcite treebanks and recent efforts to harmonize their morphological tags @xcite . in this work we review these harmonized treebanks foot_0 plus the non-ud treebank lasla @xcite , and conclude that more data curation is required to fully evaluate and improve morphological tagging's cross-domain accuracy. our contributions include: 1) precisely documenting genre and historical context for the 544 texts within the ud treebanks as a machinereadable, cross-treebank resource that will enable future work to examine morphosyntactic association against these variables; 2) harmonizing the ud and lasla treebanks to reduce annotation differences that can affect training; 3) proposing edits to the ud tagset that better align with standard analyses of latin grammar to facilitate work by researchers with standard latin training; and 4) conducting a cross-time analysis with experimental results broken down by historical period that show the promise of our harmonization efforts and bert-based morphological taggers. foot_2 2 latin treebanks revisited 2.1 time and genre metadata detailed metadata on the texts included in the latin ud treebanks is difficult to aggregate or lacking altogether. information on the included works' time period, genre, author, and relative size has not been compiled in one place. our work takes major steps to fill this gap. for all 544 texts across the five ud treebanks, we manually collected the following metadata: the source treebank, time period, century, internal treebank identifiers, cumulative and split-level sentence counts, author, and exhaustive genre labels. ## harmonizing ud and lasla annotations in this section, we describe steps taken to reduce the annotation differences between the harmonized ud treebanks ( ## related work: morphological tagging there is a long history of work analyzing pos and morphological tagging of latin @xcite @xcite . our work follows recent trends of using transformer-based contextual representations. several recent papers have explored morphological tagging for latin. as part of the 2022 evalatin feature identification task @xcite , participants trained and tested on a subset of data from the lasla corpus that had been automatically converted to ud format @xcite . only a subset of ud morphological features were retained, partly to limit the task to morphological features identifiable by the word form, and partly to avoid features affected by annotation differences. participants were then able to train on combined ud and lasla data if they wished, but models were only evaluated on evalatin test sets, not ud test sets. nehrdich and hellwig (2022) used latinbert @xcite to train a morphological tagger predicting the case, gender, number, tense and verbform features. its outputs were then fed into the authors' dependency parser, outperforming prior work using udpipe and static word embeddings @xcite . their training and test data came from three ud treebanks (ittb, proiel, and perseus). burns (2023) developed latincy, a full nlp pipeline for latin which includes morphological feature classification. 14 notably, this pipeline was trained on all five ud treebanks with early attempts made at harmonization, using a smaller tagset than ud that is closer to standard analyses of latin grammar ( part-of-speech (pos) tagging is closely related to morphological tagging. in the 2020 evalatin campaign, participants trained and tested pos taggers on a subset of the lasla corpus @xcite . more recently, riemenschneider and frank pretrained a trilingual roberta @xcite model on english, ancient greek, and latin which surpassed the 2022 evalatin competitors ( researchers have also experimented with using gpt3.5-turbo and gpt4 for pos tagging of 16th century latin texts @xcite . no pos-annotated data exists for 16th century latin, so the authors experimented with zero-shot prompting and finetuning using data from the five ud treebanks. although the ud testsets are not entirely comparable with evalatin's, the accuracy of these gpt-based approaches seems low when compared to the results of evalatin's pos tagging shared task. although substantial progress has been made in latin morphological tagging, gaps still exist. aside from @xcite and burns (2023), prior work has not leveraged all five ud treebanks for training and evaluation. while gamba and zeman (2023a) measure overall tagging accuracy, more detailed analysis of specific morphological features and diachronic trends has been left to future work. moreover, to our knowledge no recent paper has evaluated the currently available taggers on ud test data. ## experiments we use three metrics in our evaluations: whole string morphological accuracy, macro f1 for individual features, and f1 for particular feature-values. see a.8 for more detailed explanations. ## conclusion and future work in this work, we consider the diverse time periods represented in the latin treebanks when training and evaluating morphological taggers. we hope the genre metadata we provide can be used for future cross-genre analysis of latin, similar to the crosstime analysis we present in this paper. we also believe further improvements can be made through the harmonization of remaining annotation differences ( §3.4) and more informed modeling choices. specifically, we hypothesize that (1) conditioning morphological feature prediction on upos, or vice versa; (2) enforcing grammatical constraints through modeling, rather than only through training data; and (3) constructing a morphologically-aware tokenizer may all lead to improved performance.