“Gotta catch ‘em all!”: Retrieving people in A ncient G reek texts combining transformer models and domain knowledge (2024)

Abstract: In this paper, we present a study of transformer-based Named Entity Recognition (NER) as applied to Ancient Greek texts, with an emphasis on retrieving personal names. Recent research shows that, while the task remains difficult, the use of transformer models results in significant improvements. We, therefore, compare the performance of four transformer models on the task of NER for the categories of people, locations and groups, and add an out-of-domain test set to the existing datasets. Results on this set highlight the shortcomings of the models when confronted with a random sample of sentences. To be able to more straightforwardly integrate domain and linguistic knowledge to improve performance, we narrow down our approach to the category of people. The task is simplified to a binary PERS/MISC classification on the token level, starting from capitalised words. Next, we test the use of domain and linguistic knowledge to improve the results. We find that including simple gazetteer information as a binary mask has a marginally positive effect on newly annotated data and that treebanks can be used to help identify multi-word individuals if they are scarcely or inconsistently annotated in the available training data. The qualitative error analysis identifies the potential for improvement in both manual annotation and the inclusion of domain and linguistic knowledge in the transformer models.

## introduction identifying the mentions of people in texts is one of the goals of the broader task of named entity recognition (ner). for scholars working on historical texts, accurately finding and identifying people is particularly valuable for studying the representation of individuals, both in qualitative and data-driven studies. the present research, for instance, is embedded in a broader project aiming at performing large-scale analysis on the mentions of individuals in ancient greek and latin texts (nikaw, networks of ideas and knowledge in the ancient world). for classical languages, and ancient greek in particular, the task remains challenging to automate. this study capitalises on recent advancements in transformer models, which have shown promising improvements over previous approaches. after introducing the available methods and data for ner on ancient greek (sections 2 and 3), in section 4, we compare four recent transformer models of ancient greek and their performance for ner, with a focus on identifying mentions of people. this comparison allows the selection of a model for further exploration. since the ancient world has a wealth of domain-specific resources on offer, in sections 5, we focus on the specific task of predicting pers entities by simplifying the ner task, and we explore how integrating gazetteers (section 5.2) and syntactic annotations (section 5.3) can impact the process of pinpointing individuals in texts. in the qualitative error analysis in section 5.4, we identify several shortcomings of the reduced transformer method and discuss how domain knowledge and linguistic information impact the performance. with this, we contribute to advancing ner for ancient greek, identifying the strengths and limitations of currently available models and data and offering concrete suggestions for the way forward. ## related work the task of ner for historical languages presents several challenges, which can be traced back to four main factors @xcite : diversity of sources, noisiness of data, language change, and lack of resources. these challenges are transferable to ancient greek and latin corpora. however, the use of transformer models yields promising results: this is demonstrated for latin by torres aguilar (2022); @xcite , and for ancient greek by @xcite ; @xcite . @xcite present two transformer models finetuned for the task of ancient greek ner. this model was created by training a xlm-roberta-based multilingual model that was previously fine-tuned on the word alignment task for ancient languages, including ancient @xcite and an ancient-greek-bert model @xcite respectively. in this paper, we compare the ner performance of four transformer models for ancient greek, described in detail in section 4 and 5. in addition, recent studies highlight the advantages of incorporating domain knowledge, in particular gazetteers, in the training of ner models, especially for lowresource languages @xcite @xcite . gazetteers are external resources that often take the form of name dictionaries, grouped by a specific entity type (e.g. location or person). to leverage the advantage of domain knowledge, we incorporate the trismegistos gazetteers of names and name variants (tm namvar) (broux and depauw, 2015) foot_0 and of places (tm geovar) foot_1 in two approaches described in section 5.2. this rejoins the efforts of exploiting available knowledge bases for annotating ancient greek texts, as discussed in @xcite . finally, we address the problem of multi-token entities, which are particularly difficult to label automatically given their sparsity in the training data and the potential complexity added by factors such as overlap, nesting, and non-consecutiveness @xcite @xcite . in section 5.3, we explore the effectiveness of expanding single-token entities into multi-token entities using syntactical dependencies. ## models for normal ner in this section, we compare the performance of four transformer-based models for ner. we have a twofold objective: determine the best-performing model for the general ner task, 8 and determine to what extent the inclusion of domain knowledge can improve the results of the best-performing transformer models. ## predicting pers entities (reduced models) because the overarching project in which this research is embedded is primarily interested in the mentions of people, and because, as demonstrated by ## conclusion the goal of our study is to consistently and fully automatically annotate attestations of people using transformer-based ner. we trained several transformer models on available data for ancient greek ner and evaluated performance both on a held out test set and on randomly annotated data representative for greek literary data. while all models performed adequately, we conclude that inconsistency in annotation remains an obstacle in achieving high performance -which is in line with the findings by @xcite and @xcite , especially concerning multi-token entities. the approaches introduced in sections 5.1-5.3 increase the performance for detecting persons specifically, but we recognise that there is still room for improvement (see section 7). in future work, we will consider the integration of other available gazetteers, 10 and incorporate attestation counts as weights. the syntactically informed annotation of multi-token entities could equally benefit from an improvement of the rulebased extraction through a more careful analysis of the structure of i-entities in the dependency tree.