Evaluating Large Language Models for Narrative Topic Labeling (2025)

Abstract: This paper evaluates the effectiveness of large language models (LLMs) for labeling topics in narrative texts, comparing performance across fiction and news genres. Building on prior studies in factual documents, we extend the evaluation to narrative contexts where story content is central. Using a ranked voting system with 200 crowdworkers, we assess participants’ preferences of topic labels by comparing multiple LLM outputs with human annotations. Our findings indicate minimal inter-model variation, with LLMs performing on par with human readers in news and outperforming humans in fiction. We conclude with a case study using a set of 25,000 narrative passages from novels illustrating the analytical value of LLM topic labels compared to traditional methods. The results highlight the significant promise of LLMs for topic labeling of narrative texts.

## introduction topic modeling has been and continues to be one of the most popular ways of interpreting and understanding documents within large digital repositories. whether for the purposes of discourse analysis @xcite , literary studies @xcite , media framing @xcite , or understanding semantic change @xcite , successfully extracting high-level topics has been central to the digital humanities and the large scale study of history and culture (for a review see @xcite ). until recently, the principal way that researchers have derived topics from texts has been through the use of unsupervised learning approaches such as latent dirichlet allocation (lda) @xcite and its various updates @xcite @xcite . these methods, however, face well-known limitations, ranging from the ambiguity of topic la-bels, to their sensitivity to parameter choices (most notably the number of topics), and the oversimplification of textual content through the use of bag-of-words modeling. recent work has begun to show how llms can potentially enhance or even replace traditional topic modeling. llms have been used to facilitate topic labeling @xcite and topic evaluation @xcite . and they have been used in lieu of topic modeling, demonstrating far greater semantic alignment with known human labels on sets of fact-based articles @xcite and expert judgments @xcite . in this paper, we build on prior research by applying large language models (llms) to automated topic labeling, with a focus on narrative texts as a complement to studies centered on fact-based documents. narrative texts, a cornerstone of cultural traditions, have long been a subject of interest in digital humanities research due to their complexity and richness. unlike information-driven texts, narratives often depend on implicit context, figurative language, shifting perspectives, and intricate temporal structures, all of which pose unique challenges for topic extraction. by evaluating llm performance on the automated topic labeling of narratives-both fictional and factual-this study aims to enhance the methodological tools available to digital humanities researchers. to this end, we analyze two distinct narrative sub-genres: factual reporting in news articles and creative storytelling in novels. second, while previous research has focused on the comparative similarity between automated and human-generated labels (demonstrating that llms significantly outperform lda @xcite ), our study evaluates the preference for llmgenerated labels over human labels. following a methodology similar to @xcite , we use a crowd-sourced voting approach to determine whether independent readers (@xmath0. this methodology not only provides a robust evaluation of label quality but also offers a practical measure of how well llms meet the expectations of general readers. our question is: can llms label narrative topics as effectively as humans across different sub-genres, and how do they compare to well-established topic modeling techniques? finally, while prior studies have primarily focused on the functionality of a single model (e.g., gpt), we broaden the scope by evaluating gpt alongside a range of smaller, open-weight models. this comparative analysis aims to provide researchers with greater confidence in the utility of llms for topic labeling in narrative texts. to support future research and benchmarking, we publicly release all annotations generated in this study. 1 ## prior work topic modeling has experienced wide-spread use across numerous fields @xcite . despite its ubiquity, considerable research has foregrounded its methodological limitations. traditional topic models often produce topics that are statistically coherent, for example, but lack semantic interpretability, making it challenging for human analysts to derive meaningful insights @xcite . they also involve numerous pre-processing steps that increase researcher degrees-of-freedom that can impact replicability @xcite . additionally, determining the optimal number of topics is often a trial-and-error process, potentially leading to over-or under-fitting of the model @xcite . this problem can also lead to challenges in modulating the specificity or generality of topics @xcite . finally, these methods can perform poorly on short texts or documents with diverse vocabulary, limiting their applicability in certain domains such as social media analysis or highly specialized technical literature @xcite . recent work has begun to use llms in conjunction with topic modeling, either to label @xcite or evaluate topics @xcite . @xcite have devised a prompting framework for the generation and selection of topics using gpt-4 and shown significant improvement over lda with respect to human la-1 https://doi.org/10.5683/sp3/mhjrio bels for fact-based documents such as wikipedia articles and u.s. congressional bills. @xcite have developed a workflow that they call "concept induction" to replace topic modeling to surface more critical and research-oriented conceptual frameworks for the analysis of fact-based documents. here we build on this prior work to apply llmderived topic labeling to narrative texts and assess label adequacy based on independent human assessments. ## methods our experimental framework consists of two main components. in the first, we evaluate llmgenerated topic labels against human-generated labels using a survey platform with anonymous readers. given prior findings on the significant superiority of llm topics over those generated by traditional topic modeling methods such as lda @xcite , we exclude lda-based topics from this stage and focus instead on assessing the ability of llms to match or exceed human performance. in our case study (section 5), we shift our focus to a large sample of fiction passages, comparing llm-derived topics directly with ldagenerated topics. this comparison allows us to more explicitly examine the analytical advantages and limitations of llm-derived topics relative to traditional approaches. ## case study we conclude with a case study to indicate some of the conceptual insights that can be offered by llm-assisted topic labeling compared with traditional lda-based topic models. here we condition on our novel data to illustrate the most distinctive topics of the first and second half of the nineteenth-century, often referred to as the heyday of the british realist novel. for our experiment we use the above-mentioned sample of 25,000 novel chunks and label them two ways. for llm-assisted labeling we use gemma2:9b with the same prompt used for our human validation experiment. we retain two sets of labels: all original labels and the aggregated labels using the method described above. next, we applied latent dirichlet allocation (lda) using gibbs sampling. we set the dirichlet prior for documenttopic distributions to α=50/k, a commonly used heuristic that ensures a moderate spread of topics per document, and estimated β during training. the model ran for 1000 iterations with a burn-in of 20, retaining the best solution (@xmath1. we tested two levels of @xmath2. topic labels were then manually added by the authors as domain experts. after labeling, we identify the most distinctive topics in passages published before and after 1850 to model large-scale shifts in topical focus within british novels. to measure distinctiveness, we use dunning's log-likelihood statistic, a method that highlights words or topics disproportionately represented in one group compared to another based on their observed versus expected frequencies @xcite . figure 1 presents the most distinctive topic labels across four conditions: the specific gemma2 labels, the aggregated gemma2 labels, and the two k settings for our lda models. overall, we observe that llm labeling produces significantly more intelligible topics. where several of the top topics in the lda models are largely grammatical distinctions that transpire over the course of the century (e.g. the introduction of contractions to capture direct speech) or clusters of common verbs (such as looking or taking), llms produce more detailed and informative topics. "combat," "revenge," "travel," and "revolution" in the general model tell us considerably more about the genres distinctive of the pre-1850 romantic and post-romantic periods in british novelwriting than topics like "seemed," "conduct," "war," "philosophy," and "religion." similarly with the more specific models, "faith," "slavery," "marriage," and "civil war" are far better than "school," "daily rhythms" or "communication." to be sure, it is not the case that lda cannot inform researchers of broad trends in fictional narratives. the emphasis on dialogue, children, and perception are all notable dimensions of post-1850 novels. additionally, as we mention in the discussion section, there is much more testing one could do to optimize the lda workflow to improve the labelinng procedure. the value of llm-based labeling, however, lies first in the topicality of the topic labels-dialogue, perception and children all capture very different kinds of stylistic features for example, while faith, finance, and marriage are far closer to what readers understand as narrative "topics." second, as has been widely observed lda topics pose challenges of interpretation for readers leading to difficulties with consistency in topic labeling. while we did not experiment with this problem here, one of the challenges of lda labeling is the labeling step itself. third, llm-derived topics also capture more thematic diversity than lda methods without introducing the noise of unintelligible topics. ## discussion the results of this study highlight the promise and limitations of using large language models (llms) for narrative topic labeling, particularly when evaluated across distinct genres like fiction and news. while prior work has largely focused on the application of llms for fact-based or general documents, our findings extend this understanding to narrative texts, showcasing the strengths and weaknesses of these models in a storytelling context. one of the key findings of this study is the comparable performance of large language models (llms) to human annotators in narrative topic labeling. our analysis revealed that llms effectively generated coherent and contextually appropriate labels for both fiction and news genres. for example, in the fiction dataset, gemma2 provided labels such as "family relationships" and "urban life," which aligned well with human annotations of similar passages. similarly, in the news dataset, llmgenerated labels like "us-china relations" and "prostate cancer" closely matched human-provided labels. importantly, we found that pre-processing or intermediate steps were not necessary; a direct, zero-shot prompting approach performed on par with human annotations, streamlining the process without compromising quality. one of the key challenges we encountered is the long-tail distribution of llm-generated labels. the sheer diversity of labels produced by the models often led to an overwhelming number of distinct topics, many of which were semantically similar or redundant. to address this, we implemented a reasoning model to aggregate these labels into a more manageable set of general topics. while this approach reduced redundancy and improved interpretability, it introduced its own limitations, such as potential errors through aggregation. other methods, such as clustering techniques or alternative aggregation strategies, may be more effective and warrant further exploration to refine the process of topic consolidation. another challenge lies in our evaluation framework. for the human validation component although the ranked voting survey provided valuable insights into label preferences, it also introduced potential biases, such as the influence of phrasing or vocabulary on participant choices. additionally, our evaluation relied on the subjective preferences of general readers, which may not fully capture the utility of the labels for specific research applications. expanding the evaluation to include task-specific downstream applications or expert assessments could provide a more comprehensive understanding of llm performance and its alignment with user needs. for the lda comparison, our case study only scratched the surface of lda optimization suggesting that future could more exhaustively test llm v. lda exercises, especially given the far greater computational resources necessary for llm-assisted labeling. the models used in this study, particularly larger ones like gemma2:9b, require substantial computational power and financial resources for both inference and aggregation tasks. these constraints can make the application of llms for massive labeling tasks of hundreds of thousands of passages far more restrictive. potential solutions include leveraging smaller, finetuned models, optimizing inference processes, or exploring hybrid approaches that combine llms with more traditional methods to reduce resource demands. while llm-assisted labeling demonstrates clear advantages in interpretive depth, traditional approaches like lda still hold value, particularly as tools for dimensionality reduction. lda's ability to cluster and summarize large textual datasets efficiently provides complementary insights that are less focused on interpretive richness but valuable for structuring data. in contrast, llm-based labeling excels in producing semantically rich and contextually specific labels, making it more suitable for applications where interpretive depth is prioritized. the choice between methods should depend on the specific goals and constraints of the research project. our case study demonstrated the thematic richness that llm-assisted labels can bring to largescale cultural research. by analyzing shifts in topical focus within british novels across the 19th century, we showed how llms could generate in-sightful and historically significant insights, such as emerging attention to "civil war" and "slavery" in the later nineteenth century and a receding attention to topics related to "native-american culture" and "land ownership." this capability highlights the potential of llm-assisted labeling to validate and discover new dimensions of understanding in literary and cultural studies, offering researchers a powerful tool for examining thematic evolution across time and genres. ## conclusion this study underscores the transformative role large language models (llms) can play in narrative topic labeling, particularly in capturing the semantic richness and thematic complexity of both fiction and news texts. by performing on par or above human annotators across numerous passages, llms demonstrate their ability to produce labels that resonate with general readers while maintaining consistency across genres. importantly, this capability not only streamlines the annotation process but also opens new possibilities for scalable and nuanced narrative analysis, particularly in contexts where traditional methods such as lda struggle with interpretive specificity. our results also highlight the unique contributions of llms to narrative understanding beyond their technical accuracy. unlike earlier methods, llms offer the ability to identify subtle thematic patterns and connect these to broader cultural or historical narratives. this ability to balance specificity with breadth positions llms as powerful tools for both academic research and applied settings in journalism, literature, and cultural studies. while challenges such as label aggregation and computational costs remain, this study demonstrates the promise of llms as a paradigm shift in narrative topic labeling. their ability to go beyond clustering and surface themes that align with human intuition makes them invaluable for complex narrative analysis.