Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain (2023)

Abstract: Some Natural Language Processing (NLP) tasks that are in the sufficiently solved state for general domain English still struggle to attain the same level of performance in specific domains. Named Entity Recognition (NER), which aims to find and categorize entities in text is such a task met with difficulties in adapting to domain specificity. This paper compares the performance of 10 NER models on 7 adventure books from the Dungeons and Dragons (D&D) domain which is a subdomain of fantasy literature. Fantasy literature, being rich and diverse in vocabulary, poses considerable challenges for conventional NER. In this study, we use open-source Large Language Models (LLM) to annotate the named entities and character names in each number of official D&D books and evaluate the precision and distribution of each model. The paper aims to identify the challenges and opportunities for improving NER in fantasy literature. Our results show that even in the off-the-shelf configuration, Flair, Trankit, and Spacy achieve better results for identifying named entities in the D&D domain compared to their peers.

## introduction named entity recognition (ner) targets the identification and classification of textual entities, such as names and locations. in the diverse and intricate vocabulary of fantasy literature, like that of dungeons and dragons (d&d), ner becomes challenging @xcite . d&d, a prominent fantasy literature domain, spans content for its namesake tabletop game @xcite @xcite . these narratives inhabit fictional realms like forgotten realms and dragonlance, bursting with characters, locations, and objects @xcite . ner's utility in fantasy literature is vast: from extracting information and summarizing text to character analysis and plot creation. however, conventional ner models, primarily trained on standard datasets like @xcite or ontonotes 5.0 @xcite , might falter on fantasy texts due to their unique linguistic attributes. recognizing the need for domain-specific adaptation, other specialized areas such as law @xcite , medicine @xcite , and the dynamic landscape of social media @xcite have already seen research emphasizing it. large models, as @xcite points out, can face domain adaptation challenges, stressing the need for evaluating ner models specifically on fantasy content. fantasy ner has potential, especially with advancements in image generation. a notable application might involve an image generation model leveraging ner tags to derive prompts and subsequently produce contextually relevant images. our study contrasts 10 ner models across seven d&d books, each averaging 118,000 words. manual annotations of entities were made and juxtaposed against model outputs. through precision assessments and named entity distribution analyses, we glean insights into model performances in the fantasy domain. our key contributions include: following this, section 2 delves into related ner and fantasy literature works. section 3 details our data and annotation process, while section 4 unveils our methods and findings. sections 5 and 6 respectively discuss insights and conclude our research, and section 7 outlines potential future endeavors. we introduced a novel annotated dataset of d&d books for ner and evaluated 10 ner models, including xlm-roberta @xcite , @xcite , elec-tra @xcite , and others. other studies have compared the performance of ner models on different types of texts and languages. for example, @xcite compared spacy, flair, m-bert, and camembert on anonymizing french commercial legal cases. they found that camembert performed the best overall, followed by flair and m-bert. spacy had the lowest scores but also the fastest prediction time. @xcite compared spacy, flair, and stanford core nlp on anonymizing english court decisions. they found that flair had the highest scores, followed by stanford core nlp and spacy. @xcite compared rule-based, crfbased, and bert-based techniques for ner on text data. they found that bert-based technique had the highest accuracy and recall, followed by crfbased and rule-based techniques. @xcite compared nltk, spacy, stanford core nlp, and bert @xcite on extracting information from resumes. they found that bert had the highest accuracy and f-measure, followed by spacy, stanford core nlp, and nltk. these studies suggest that different ner models may have different strengths and weaknesses depending on the type, language, and domain of the text data. our study aims to contribute to this understanding by providing the first systematic comparison of ner models on fantasy texts and analyzing their performance and characteristics. ## data collection and annotation this section details the data sources and annotation process utilized for our named entity recognition (ner) task, a subtask of information extraction that classifies named entities in unstructured text into categories such as persons, organizations, and locations @xcite . we examined seven adventure books from the dungeons and dragons (d&d) realm, listed in through a comprehensive analysis of these texts, we used their rich narratives and character dynamics to benchmark and assess various ner models in this intricate domain. ## experimental setup and results the experiment was conducted to identify how effective are the ner models when using them as off the shelf models in identifying named entities for a fantasy domain when there are no available corpora for fine tuning. for testing we used 10 different contemporary ner midels. following the testing approach for the ner models mirrors algorithm 1. here, paragraphs of input text are fed into the models without specific prompts. figure 2 displays entries that encountered corruption. these highlight instances where ner models incorrectly processed and extracted entities from the source material. in the given example shown in figure 2 , the name "fembris larlancer" is erroneously divided into two distinct words, "fembris l#" and "rlancer", as a result of corruption during the ner processing stage. this example underscores the challenges faced during the entity extraction pro-{ "entities": [ { "entity": "i-person", "score": 0.5659823, "index": 308, "word": "fembris l#", "start": 1160, "end": 1162 }, { "entity": "i-person", "score": 0.51227564, "index": 309, "word": "rlancer", "start": 1162, "end": 1164 } ] } cess and the need for robust post-processing to ensure the accuracy and quality of the extracted entities. after removing corrupted and eligible named entities, duplicate entries must be removed to do a proper comparison of performance between different models. for this tuples of words in adjacent positions were generated and compared. for example mayor lei duvezin, mayor duvezin, lei duvezin and duvezin all refers to the same entity with the label person. in cases such as above tuple with most similarity matches will be retained as the named entity and duplicates will be removed. to visualize the raw named entity identification potential of each model, a density plot was plotted with respect to count of identified named entities with ner models. following figure 3 shows the density of named entities recognized by each ner model. the hue represents the overlapping count ranges of named entities identified in each source book. without training, nlp frameworks like trankit @xcite , flair @xcite , and spacy @xcite show a strong baseline in entity recognition. model precision is key in performance evaluation. this is gauged by comparing the true positive entities with actual named entities. this comparison can be visually represented for each model across source books. for a comprehensive model assessment across books, kernel density estimation (kde) is used. it's a non-parametric method estimating the probability density function @xcite : figure 4 illustrates models' efficacy over seven source books. a gradient near 1 signifies optimal performance. in dd, named characters, with their elaborate backstories, are central. assessing a model's inclination to identify these characters over other entities is vital. this inclination can be visualized by juxtaposing character counts with total entities, contrasted against real metrics. figure 5 delineates the frequency of character identification across all source books. meanwhile, figure 5a and figure 5b depict the distribution pertaining to models and books, respectively. in the d&d landscape, named characters, renowned for their intricate histories, are paramount. evaluating a model's propensity to spot these characters in relation to other entities is imperative due to the significant role characters play in d&d narratives. this bias can be graphically represented by mapping character counts against all identified entities and contrasting them with authentic counts. by scrutinizing the named entity counts from diverse ner models and comparing them to true values, one can infer model behavior and efficacy. figure 5 offers a glimpse into character recognition frequency for different models across sourcebooks, with figure 5a and figure 5b charting the distributions for models and books respectively. from figure 5b , we observe a consistent ratio between characters and other named entities across books. this consistency allows us to downplay book variability and focus on the insights from figure 5a . notably, nlp frameworks such as spacy @xcite and flair @xcite exhibit more balanced frequency distributions, indicating a higher character identification ratio. although this might be unfavorable in certain contexts, in this domain, aligning character identifications closely with overall named entity values signals optimal performance. this suggests spacy and flair perform exceptionally in an offthe-shelf setting. figure 6 showcases precision and recall metrics for each ner model. to determine recall, we derived the true positive count from average unique named entity counts, while the true count originated from llm models, as outlined in the precision and recall values were averaged for each model across source books, and plotted to offer a concise visualization of each ner model's performance. evidently, flair and spacy outshine other ner models in precision and performance, while trankit @xcite excels in recall relative to its precision. ## discussion we undertook a named entity recognition (ner) task on seven adventure books from the esteemed dungeons and dragons (d&d) series. our methodology involved manual entity annotations in these books, which were subsequently verified against outputs from three leading language models: bloom, openllama, and dolly. our annotation guidelines delineated entity types into categories like person, organization, location, and misc. ten ner models were subsequently employed to gauge their efficacy in recognizing named entities within d&d. among these, flair, trankit, and spacy emerged superior, mirroring findings from past ner-centric studies. conversely, stan-forddeid @xcite and rober-taner (baptiste, 2022) lagged in performance. a precision-centric analysis further reiterated the dominance of flair, trankit, and spacy over their counterparts. the findings imply that while generic models can decently handle ner tasks in specialized domains like d&d, performance inconsistencies exist across models. employing annotation guidelines bolsters consistency in entity recognition across varied books and contexts. moreover, incorporating large language models for automated annotations can significantly mitigate the manual intervention needed for comprehensive datasets, particularly in intricate domains such as d&d. however, our study bears certain caveats. we refrained from fine-tuning the ner models specifically for d&d, so our findings are indicative of generic model capabilities and might not capture the full potential of domain-specific optimization. our dataset, comprising just seven books, might not encompass the depth and breadth of d&d narratives. the exclusive focus on wizards of the coast publications could also inadvertently introduce stylistic biases. finally, while our study zeroes in on d&d as a fantasy subset, our insights might not seamlessly extend to other literary domains with their unique nuances. ## conclusion our exploration illuminates the remarkable potential of harnessing off-the-shelf models for ner tasks within the d&d universe's nuanced realm. some models showcase an impressive baseline in entity recognition for this domain without extensive fine-tuning. however, there's a compelling need for (a) models (b) adventure sourcebooks. figure 6: precision graph for different ner models additionally, our research serves as a foundational resource for future inquiries. the dataset we've curated and our annotation guidelines stand as a benchmark for gauging the efficiency of future ner models or techniques. consequently, our work not only reveals the current prowess of ner models within the d&d context but also sets the stage for continued innovation at the confluence of fantasy literature and artificial intelligence.