SEFLAG : Systematic Evaluation Framework for NLP Models and Datasets in L atin and A ncient G reek (2024)

Abstract: Literary scholars of Latin and Ancient Greek increasingly use natural language processing for their work, but many models and datasets are hard to use due to a lack of sustainable research data management. This paper introduces the Systematic Evaluation Framework for natural language processing models and datasets in Latin and Ancient Greek (SEFLAG), which consistently assesses language resources using common criteria, such as specific evaluation metrics, metadata and risk analysis. The framework, a work in progress in its initial phase, currently covers lemmatization and named entity recognition for both languages, with plans for adding dependency parsing and other tasks. For increased transparency and sustainability, a thorough documentation is included as well as an integration into the HuggingFace ecosystem. The combination of these efforts is designed to support researchers in their search for suitable models.

## introduction recent years have seen a surge of publications employing natural language processing (nlp) for the analysis of ancient texts @xcite @xcite . however, as with other historical languages (see @xcite for classical chinese), the communities around latin and ancient greek rarely provide standardized and centralized resources specifically for the training and evaluation of nlp models. the corresponding treebanks in universal dependencies are one notable exception. lemmatization, on the other hand, is notorious for the many different approaches to, e.g., character encoding (tauber, 2019), handling of diacritics @xcite , homographs @xcite , and other challenges. besides, existing nlp models are scattered across many different technical platforms such as spacy @xcite , flair nlp @xcite or google cloud @xcite . as a consequence, every member of the classics community has to collect and evaluate the same resources again. this leads us to the central question of how we can support literary scholars of latin and ancient greek in choosing the right nlp models for their research agenda. to address this challenge, we present seflag foot_0 , the systematic evaluation framework for nlp models and datasets in latin and ancient greek. our work is still in progress, so we share only a small proof of concept with lemmatization and named entity recognition (ner). next up, dependency parsing will follow. our contributions are as follows: ## related work building highly specialized frameworks like se-flag can suffer from various problems. for example, transferring modern developments (large language models, analytical categories) to ancient contexts is non-trivial @xcite @xcite @xcite . in particular, the loose distinction between different seg-ments of a text in vedic sanskrit @xcite and other historical languages necessitates elaborate interpretative efforts to introduce modern syntactic concepts like punctuation or main and subordinate clause. this relatively relaxed notion of syntax correlates with the rather pronounced linguistic variation of latin and ancient greek due to their diachronic, diatopic and diastratic differences @xcite . similar issues arise in the treatment of historical newspapers @xcite and early modern scientific texts @xcite , which indicates a general trend of higher linguistic variation and lesser availability of language resources for historical languages @xcite . for ancient texts, there is the additional burden of manifold textual transmission (including indirect transmission through citations), which prevents us from establishing texts in their original form with certainty @xcite . besides, existing nlp implementations for textual annotation often do not fully adhere to the fair @xcite guiding principles of research data management. earlier evaluations of available resources, on the other hand, were often performed indirectly, e.g., by carrying out surveys in the user community @xcite rather than directly testing the resources. finally, large-scale data processing necessitates automation due to its efficiency, but automation can lead to a loss of data quality in highly heterogeneous datasets @xcite and is often not sufficient for unifying multiple conceptually different resources, e.g., for valency patterns in ancient languages @xcite . fortunately, there has also been some progress in nlp for latin and ancient greek. most researchers involved in nlp evaluation choose one of the two languages and a single nlp task, like word embeddings @xcite or topic modeling @xcite . a few of them even work on both languages, usually for single tasks that can be addressed through inherently crosslingual methods @xcite . some engineering work has been done by the cltk team @xcite , offering a solid basis for data processing in historical languages but still suffering from a lack of evaluation and benchmarking. nevertheless, all of these approaches are an important foundation for our mission of collecting and disseminating such resources in a centralized manner. a more coordinated and comprehensive initiative was the lila project foot_2 , which managed to successfully collect, harmonize and disseminate multiple existing language resources for latin @xcite . in particular, they launched the evalatin evaluation campaign @xcite . unfortunately, their project has officially ended and their platform (which is still running) does not cover ancient greek at all. even for latin, it addresses many nlp tasks, but not all: ner, topic modeling and some others are missing. other platforms have partly solved the problem of long-term availability and funding, such as the perseus digital library. like lila, they use linked data @xcite to make their content findable and interoperable, but do not support evaluation reports. generic language platforms like hf offer such reporting, but cater to a different audience (namely computer scientists and computational linguists), thus neglecting our target group of literary scholars. ## methodology our intermediate goal is to find existing nlp models and apply them to existing datasets (both having been created by others). then, we perform one measurement for each of the two languages: predictions of the nlp model are assessed using the ground truth annotations from the dataset and consistent metrics. for enhanced transparency and reusability, we document conceptual mappings, as most datasets and nlp models were created using rather different annotation schemas. 4 for example, names of fictional characters in ancient literature may count as person names in one ner dataset, but not in others (see section 4). this results in many datasets and nlp models that belong to the same task (namely, ner), but are not easily interoperable. conceptual bridging (e.g., through mappings) is needed to close this gap and enable combinations of those resources. even language resources that do not share the same annotation schema may still profit from the unification of certain annotation labels (such as person and prs, see section 4), depending on their conceptual overlap. for enhanced sustainability, in applicable cases like lemmatization, we merge multiple datasets (i.e., various treebanks from universal dependencies) into a larger benchmark and publish it as a hf dataset foot_4 for the corresponding task. in doing so, we adopt the approach of @xcite by integrating diverse language material with regard to time and genre. additionally, we use datasheets and model cards (see appendix a and b) to describe language resources systematically. ideally, such datasheets and model cards should be provided by the creators themselves. however, the adoption of those standards is still insufficient in the classics community. as the next best option, we create such materials ourselves and try to infer their content from publicly available information about the resources (in scientific publications, source code repositories, etc.). they will be uploaded to the hf space foot_5 of an nlp infrastructure (see section 4) and integrated into their website. this infrastructure allows our users to directly apply the evaluated nlp models to their own datasets and learn more about the various tasks through open educational resources. model cards are provided separately for each nlp model in our evaluation. they include general metadata like license, version or architecture, but also more complex considerations like ethical implications, ecological factors and possible risks of certain use cases. apart from literary scholars of latin and ancient greek, we also take neighboring disciplines into account, such as historians, theologians or archeologists dealing with ancient textual materials. from our point of view, their shared characteristics are limited technical background knowledge @xcite and a high interest in practical applications as well as methodological innovation, though all of these aspects are somewhat disputed in the scientific literature @xcite @xcite . ## implementation for long-term sustainability, we aim to integrate our work into the daidalos research infrastructure foot_6 with institutionalized governance as provided by the datacenter foot_7 of humboldt university berlin foot_8 , which offers a dedicated cloud computing service (as recommended by @xcite ). funding for such an infrastructure has already been secured for an initial period of 3 years, which can be extended to about 10 years depending on periodic evaluation. as a consequence of our integration into that infrastructure, we also build on their community work: their already existing biannual workshops, national research partnerships with classical scholars and open educational resources on historical language processing foot_9 are the backbone of our strategy to interact with our target audience and disseminate the evaluation results as widely as possible. we provide explicit mappings for two cases foot_10 : nlp models that are evaluated internally (on the test split of their original training data), and externally (i.e., a completely new dataset). the internal case is covered by the flair_grc_multi_ner tagger being tested on the data foot_11 that was curated by yousef et al., i.e., a mixture of herodotus, homer and athenaeus of naucratis. under those circumstances, no mapping is needed at all because the model was conceptualized directly with that dataset in mind. mapping the external case is more challenging: we took the latincy model and applied it to the herodotos project foot_12 dataset. each of the two resources uses 4 different entity tags that roughly correspond to the original ones introduced in @xcite . the person and prs classes are arguably most compatible. however, the annotation guidelines for neither of the two language resources have been published anywhere. thus, we cannot say for sure if their rules for assigning labels to named persons match each other, even considering the vague statement in burns (2023) that the label applies to "people, including fictional". as a last resort, since the herodotos dataset was included as training data for the lat-incy model, we may conclude that all 3 entity classes roughly correspond to each other, which allows us to apply mappings and pair the two for evaluation. ## results we report the evaluation results in a qualitative analysis of the lemmatization results empirically reveals some of the problems that were outlined in the research literature (see section 1): variant spellings of the same lemma exist due to flexible orthography (οὕτως or οὕτω(ς), parvulus or paruulus), capitalization (romanus or romanus), diacritics (τίς, τις or τὶς), and separate entries for specific inflected forms (κύκλος or κύκλῳ, diuerto or diuersus). ## limitations and risks our approach of curating datasheets and model cards for resources that we did not create ourselves leads to information gaps in the documentation. nevertheless, our effort of inferring information from other sources and disseminating it in a centralized, systematic fashion is highly beneficial for the targeted research community. currently, we only report rather simple metrics. to enable deeper insights into model bevavior and dataset structure, we plan to add class-wise confusion matrices, detailed qualitative error analyses and task-specific metrics (like the ones introduced by the message understanding conference @xcite ) which allow to distinguish between errors related to entity status and entity type. finally, we are very confident that our framework scales well to other planned tasks like part-ofspeech tagging, sentiment analysis and dependency parsing. 15 as an abstract representation of the different linguistic annotations, for example, we intend to use a graph model like salt @xcite . besides, we have clear interfaces for adding more models, datasets and evaluation metrics. as of now, however, it is unclear to which extent we may need to introduce further metadata (spatial, temporal, stylistic, etc.). ## conclusions our evaluation framework seflag aims to support literary scholars of latin and ancient greek in selecting the right nlp models for their research. we provide quantitative evaluations of existing models on suitable datasets. conceptual mappings between tagsets used for the annotation of different language resources are documented explicitly and in a human-readable way. evaluation results are reported using common metrics (f1, accuracy) and are accompanied by additional documentation for the language resources: datasheets for datasets and model cards for nlp models. using that additional information, we enable researchers to critically assess the value of such resources for their own research, including questions of dataset characteristics, model architecture, annotation guidelines and contact persons. in short, we provide low-level access to the costly and complex task of nlp evaluation for latin and ancient greek, with a proof of concept that focuses on ner and lemmatization. in the near future, we will work on fully integrating our framework into the daidalos research infrastructure. furthermore, we would like to add more nlp tasks, models and datasets. finally, we also want to create open educational resources to educate interested researchers about central essentials of the evaluation, such as specific metrics, task concepts and annotation approaches.