An evaluation of Named Entity Recognition tools for detecting person names in philosophical text (2025)

Abstract: For philosophers, mentions of the names of other philosophers and scientists are an important indicator of relevance and influence. However, they don’t always come in neat citations, especially in older works. We evaluate various approaches to named entity recognition for person names in 20th century, English-language philosophical texts. We use part of a digitized corpus of the works of W.V. Quine, manually annotated for person names, to compare the performance of several systems: the rule-based edhiphy, spaCy’s CNN-based system, FLAIR’s BiLSTM-based system, and SpanBERT, ERNIE-v2 and ModernBERT’s transformer-based approaches. We also experiment with enhancing the smaller models with domain-specific embedding vectors. We find that both spaCy and FLAIR outperform transformer-based models, perhaps due to the small dataset sizes involved.

## introduction named entity recognition (ner) tools have the ability to quickly and accurately extract named entities that can then be used to form connections between people. this has clear applications in digital humanities research @xcite . a user study on the national library of france's portal gallica showed that 80% of search queries contain a proper name @xcite . in philosophy, and particularly in histories of ideas research, tracing names in digitized texts can reveal how concepts have spread and what influence authors had. modern citation practices only emerged around the beginning of the 20th century, which makes citation analysis unsuited to study most of the history of science and philosophy. referencing by mentioning names was the main type of referencing before citation conventions developed, and can thus be used to trace histories of philosophical ideas, as outlined by @xcite . however, in the field of philosophy, remarkably little attention has been paid to even simple computational tools that could help with quantitative analysis @xcite and supplement the traditional close reading of texts. @xcite perform mention detection on late 19th century and early 20th century anglophone philosophical texts using a rule-based gazetteer approach, after finding mistakes in applying spacy's ner model to some of these texts. however, a downside of this approach is a lack of out-of-domain coverage -such a system can only be expected to identify mentions of philosophers, missing out on e.g. other scientists, politicians or family members. to support these interests, we perform a quantitative evaluation of a diverse range of ner approaches for philosophical text, from a rule-based approach to the recent modernbert llm. 1 of course, extensive literature on ner systems and their general performance already exists (e.g. this survey by @xcite , but that does not necessarily translate to equal performance in the domain of philosophical texts. obtaining state of the art results in ner relies heavily on domain-specific knowledge @xcite and large annotated datasets, and many single-domain systems have been developed @xcite @xcite @xcite . philosophy is certainly a specific domain. even for philosophical texts that are in english and from the 20th century, there are significant differences between such text and wikipedia text in terms of lexical semantics and word frequencies @xcite . the frequent mention of low-frequency philosophical terms and capitalized german nouns may throw off ner systems. similarly, the types of names mentioned are likely to be different than those in general-purpose ner training datasets. lastly, textual corpora in this domain are smaller. therefore, in this study, we manually annotated part of the quine corpus @xcite , consisting of the works of w. v. quine, for person names, for the purpose of tuning and evaluating named entity recognition systems in the domain of philosophical text. we use this data to train and evaluate state-of-the-art approaches as well as approaches that are more accessible to humanities researchers by being packaged in text processing tools. for ner tools, we evaluate the cnnbased ner @xcite as implemented in spacy @xcite and bilstm-crfbased ner as implemented in flair @xcite . furthermore, we evaluate the recent mod-ernbert llm @xcite , an updated approach to bidirectional stacked encoders with modern optimizations. we also include ernie-v2 @xcite , a model that incorporates entitylevel and phrase-level masking strategies into its pre-training objectives, and spanbert @xcite , a model with a span-based version of the bert masked language modelling training objective. these bert variants are potentially more suited to performing the ner task compared to base bert @xcite . we also include a gazetteer baseline and a rule-based system based on edhiphy, @xcite database of philosophical names for mention detection. ## background even though some studies regarding the philosophical textual domain have been conducted @xcite , only @xcite cover the task of recognising named entities or person name mentions. nevertheless, names that are frequently referred to in philosophical texts, especially names of philosophers, are often relevant and important to the writer. ner can aid in instantiating a web of relevance in philosophical texts. there is some work on domain adaptation of word embeddings for philosophical text @xcite . these studies evaluate the performance of embedding models with different types of domain adaptation, focusing on the challenge of having a small amount of in-domain data for philosophy. they test concatenation of in-domain and general-domain data using the hyperwords @xcite imple-mentation of a count-based model with svd dimension reduction, in-domain pretraining from scratch with word2vec @xcite , continued pretraining on the target domain with word2vec, tuning on target in-domain terms with nonce2vec @xcite and elmo contextual embeddings @xcite with indomain pretraining as well as general pretraining and in-domain finetuning. these studies show that models benefit from a combination of in-domain pretraining and generaldomain tuning, and that older modeling approaches are competitive with contextual embeddings in small data settings. based on these findings, we experiment with incorporating domain-specific embeddings into the spacy and flair models. ## data we use the quine corpus, version 0.5 @xcite , which consists of 228 documents, philosophical articles, books and letters; all written by the 20th century american philosopher willard van orman quine. topics range from mathematics to formula-heavy logical writing to philosophical theories and concepts. the corpus contains 2,150,356 word tokens in the format for linguistic annotation (folia-xml, van gompel and reynaert, 2013), originating from printed texts written by quine that were digitised using optical character recognition and semi-automatically corrected. we randomly select 6800 sentences from the corpus (8.8% of the corpus) for manual person name annotation. random selection ensures sentences from throughout quine's bibliography are included. sentences containing formulae were not considered for random selection. annotation was performed using a web-based annotation tool foot_3 that yields character-based indices. entities were annotated by a single annotator for maximum coverage. this annotator received domain instructions from a quine expert. names with spelling or ocr errors were also annotated, and in hyphen-linked entities, such as "the einstein-boole theory", both separate entities are labelled. incorrect capitalization was also included in labeling -for example, alonzo church (a 20th century mathematician), often referred to as church, is frequently written in lowercase. the annotated data is split into a 70% training, 20% test and 10% validation split. besides personal names, other named entities that ner typically covers such as organization names and location names were not included in the annotation effort due to resource constraints. we did examine the potential relevance of location mentions in this corpus but found that most of them were related to holidays, travel or unrelated examples rather than e.g. universities, academic events and publisher locations. we also examined instances where names of philosophers sometimes occurred in the text without referring to the actual person, such as plationian, copernican, boolean, referring to plato, copernicus, boole. these instances, in which philosophers' ideas or groups were mentioned, were initially given the norp (nationalities or religious or political groups) label. however, this entity group was too strongly 4 models rule-based baseline as rule-based systems can perform well in narrow domains, we include such a baseline that draws on a gazetteer of 1117 names of philosophers drawn from the britannica list of philosophers foot_4 and the website famousscientists.org foot_5 . we include a partial match baseline that considers last names as a true match, or first names if only a first name occurs, and an exact match baseline that only considers firstname-lastname occurrences as a true match. this baseline has shallower coverage of the philosophy domain than @xcite approach, who include far more philosophers in their database, but our baseline makes up for it by also including scientists. flair we use @xcite hyperparameters, shown in appendix a, and default domain-general glove @xcite embeddings for english to train a bilstm-crf model on top of using our training split. llms for the transformer-based models (mod-ernbert, spanbert, ernie-v2), we use the hyperparameters in appendix a. for modernbert, we tune the base model as well as a model that has been tuned on the conll-2003 ner shared task dataset @xcite . the models we tuned are used with a token classifier head, tuned on our training split. ## discussion our results show that in-domain labeled data is essential for successfully performing the ner task in the domain of philosophy, even if the amount of labeled data is fairly small. with limited data, simpler and older model architectures occasionally outperform state-of-the-art ones, an observation that has also been made by @xcite in the digital humanities context of language modelling for literary text. to investigate the data size issue, we performed an ablation study with flair, shown in figure 1 . we observe that flair starts as a conservative model with high precision and relatively low recall, and then seemingly learns domain-specific names during training to increase recall and therefore the f1-score. more than half of our total training split is necessary to beat the llm's performance. this suggests that, with the llms requiring more data to tune a larger number of parameters, the size of the labeled dataset is the bottleneck that causes older architectures to outperform recent llms in our philosophical domain corpus. based on our findings, it seems possible to achieve better performance on our dataset in future work by augmenting flair with domain-adapted embeddings, or higher quality embeddings in general. annotating a larger portion of our corpus would lead to better ner performance and potentially allow the state-of-the-art llms to reach their full potential. one open question is to what extent models tuned on our data would generalize to other domains of philosophical text, such as authors writing about the same topics in an earlier time period or working in different traditions than analytic philosophy. historical data is very relevant to philosophical research and there are bert models pre-trained on historical text that could be used for ner, but a study on ner for dutch historical texts has shown that models pretrained on historical text do not necessarily outperform modern models at person name identification, even on 17th and 18th century data @xcite . most importantly, future work will have to demonstrate whether ner for philosophical text can be combined with bibliometric analysis or other downstream tasks to gain more detailed insight into networks of authors and the history of ideas. ## limitations our experiments are limited in scope -although representative for philosophical text where target domains are often narrow, the corpus we used only covers a single author writing in a single language. we only cover the initial stages of a pipeline for bibliometric analysis, and do not experiment with automated entity linking, which would be the next step for incorporating mentions into bibliometric analysis. the use of a single annotator means that we don't have an inter-annotator agreement score to quantify the difficulty of the task, although annotating person names isn't the most difficult of tasks. in annotating their ner dataset for the archaeology domain, @xcite observed an inter-annotator agreement rate of 0.95. the applicability of our described methods is limited by the fact that the most successful ones require thousands of in-domain labeled sentences. this limits the extent to which our method can be applied in other linguistic contexts and areas of philosophy. to facilitate comparison between architectures and data domains, we haven't fully optimized all our model conditions. performance would benefit from model-specific hyperparameter tuning, although this would also involve the use of more computational resources, and some of the topperforming models could be equipped with better general-domain or domain-adapted embeddings.