Evaluating LLM Performance in Character Analysis: A Study of Artificial Beings in Recent K orean Science Fiction (2024)

Abstract: Literary works present diverse and complex character behaviors, often implicit or intentionally obscured, making character analysis an inherently challenging task. This study explores LLMs’ capability to identify and interpret behaviors of artificial beings in 11 award-winning contemporary Korean science fiction short stories. Focusing on artificial beings as a distinct class of characters, rather than on conventional human characters, adds to the multi-layered complexity of analysis. We compared two LLMs, Claude 3.5 Sonnet and GPT-4o, with human experts using a custom eight-label system and a unique agreement metric developed to capture the cognitive intricacies of literary interpretation. Human inter-annotator agreement was around 50%, confirming the subjectivity of literary comprehension. LLMs differed from humans in selected text spans but demonstrated high agreement in label assignment for correctly identified spans. LLMs notably excelled at discerning ‘actions’ as semantic units rather than isolated grammatical components. This study reaffirms literary interpretation’s multifaceted nature while expanding the boundaries of NLP, contributing to discussions about AI’s capacity to understand and interpret creative works.

## introduction literature has long been a realm where diverse characters interact within narratives, offering deep insights into human nature and the essence of humanity (or non-humanity) across time, cultures, and genres @xcite @xcite . science fiction (sf), in particular, presents a more diverse lineage of character types compared to realist novels, featuring various forms of nonhuman entities -be they animals, aliens, or machines -as active protagonists. this makes sf an ideal genre for exploring literary representations of non-human characters' behaviors. identifying and examining character behaviors remains unexplored in both literary studies and text-as-data research. emulating human reading involves complex cognitive endeavors, including various natural language processing (nlp) tasks like coreference resolution and syntactic structure analysis. mechanically classifying figurative expressions in literary texts is challenging, but welldesigned computational approaches to textual interpretation can lead to new and insightful readings. we extracted and analyzed the behaviors and cognitive processes of 'artificial beings' in recent korean sf short stories, using annotations from five human experts and two types of large language models(llms). then we analyzed the agreement rates according to two different calculation methods. we paid special attention to the unique characteristics of the korean language, which, unlike english, employs a wide variety of endings and auxiliary predicates, often making it impossible to judge the intention and usage by solely looking at a verb's grammatical form. that is, our approach focused on the semantic dimension of action rather than verbs merely as a 'part of speech'. given the task's complexity, we utilized stateof-the-art llms, known for their proficiency in grasping context and adapting to new tasks. results show that llms can achieve high agreement with human annotators in label assignment for correctly identified spans, demonstrating potential in analyzing intricate literary contexts. however, differences in text span selection highlight ongoing challenges in ai's processing of narrative structures and character identification. ## related works character behavior analysis has long held a significant position in traditional literary studies, and with the recent emergence of new approaches integrating computational methods in digital human-ities, its importance and scope of research have further expanded @xcite . a pivotal development in this field was the creation of booknlp foot_0 , a tool for extracting characters and annotating their attributes from literary texts spanning about 200 years. this annotated dataset has sparked various computational literary studies @xcite @xcite @xcite . representing this trend, piper (2024) analyzed the physical actions of characters in english novels to explore how characters' agency is expressed in literary works. concurrently, the nlp field has shown increasing interest in utilizing llms for data annotation tasks @xcite @xcite . this approach is particularly valuable in literary settings where traditional nlp tools struggle. for instance, hicke and mimno (2024) have explored using llms' for coreference annotation in literary texts. however, most studies have focused on preconstructed, extensive literary corpora, with less attention to specific genres or nuanced analyses. in korean science fiction, the focus of this study, character studies have been diverse but primarily qualitative @xcite @xcite . while these studies offer valuable insights into character development and themes, quantitative methodologies or diachronic analyses of specific character types remain largely unexplored. ## discussion we investigated the behavioral patterns of artificial beings in contemporary korean sf short stories. by leveraging both human expertise and large language models (llms), we conducted a comprehensive analysis that revealed the complexity and subjectivity involved in categorizing the actions and cognitive processes of artificial characters in literary contexts. our novel approach deconstructs the multilayered cognitive processes inherent in literary comprehension, pioneering a literature-specific framework for evaluating inter-annotator agreement. the higher average agreement score (53.3%) obtained through the span-based fuzzy method, compared to the sentence-level method (42.6%), suggests that considering detailed textual elements better captures nuanced differences and potential consensus in annotators' interpretations. for nonliterary text annotation tasks aimed at "predictive accuracy" or "generalizability," an inter-annotator agreement rate around 50% would typically be deemed insufficient. however, given the intricate nature of reading literature, which values diverse interpretations, this agreement rate proves to be a significant finding. the possibility of varied interpretations for the same character demonstrates the depth and richness of literary texts and confirms the active meaning-making processes of readers. the metric we developed presents both unique op-portunities and methodological challenges for computational approaches in studying representations of characters in fiction. llms demonstrated promising results in matching human annotations, particularly excelling in sentence-level action classification. claude 3.5 sonnet and gpt-4o achieved impressive span match rates of approximately 60% and 40% respectively, with label agreement rates frequently exceeding 80% for correctly identified spans. these unprecedented results indicate that llms can analyze and categorize artificial beings' actions in literature to nearly human-level accuracy. notably, fewshot learning approach yielded minimal performance improvements, suggesting that llms may already possess specialized capabilities for such high-context tasks, rendering additional 'examples' less critical. however, the models' struggles with named entity recognition (ner) and character tracking across broader contexts highlight areas for improvement in ai's literary comprehension abilities. a particularly significant finding was llms' ability to distinguish 'actions' as semantic units rather than merely grammatical verbs (pos). llms generally marked necessary korean objects or auxiliary predicates correctly, enabling clear distinction of artificial beings' behaviors and highlighting the potential in complex literary annotation tasks. this research provides innovative insights into artificial beings in korean sf stories, potentially stimulating further studies in this emerging field. furthermore, our interdisciplinary approach enhances understanding of literary texts, as well as offers valuable insights for developing more sophisticated nlp models capable of grasping contextual nuances and long-term narrative dependencies. these findings present new possibilities for the convergence of literary studies and ai technology. ## limitations and conclusion this study enhanced annotation robustness by utilizing both human annotators and state-of-the-art llms to analyze behavioral patterns of artificial beings in korean sf works. to ensure methodological transparency, we provided detailed approaches for inter-annotator agreement and llm performance evaluation. however, identifying and categorizing actions in literary texts remains inherently subjective and is inevitably situated within the contemporary korean sf contexts and culturally specific understandings of artificial intelligence. in terms of data, our focus on 11 korean sf short stories by emerging authors who won a specific literary award potentially restricts the generalizability of our findings. given the recent proliferation of korean sf works featuring various artificial beings beyond these stories, expanding the research to include a broader range of contemporary korean sf literature could have provided more comprehensive insights. on the technical side, llms exhibited limitations potentially stemming from their training data and prompting strategies. the text segmentation necessitated by technical constraints may have affected the models' grasp of overall narrative context. another limitation is the omission of a detailed linguistic analysis of marked lexical ranges, which was excluded due to space constraints. even with these limitations, our findings highlight the complex interplay between literary expressions and technological capabilities, revealing both the potential and limitations of current ai technologies in analyzing nuanced literary contexts. discrepancies between human annotators and llms in interpreting artificial beings' actions underscore the subjective nature of literary analysis and the challenges in ai's comprehension of contextual nuances. however, llms's promising performance in sentence-level action classification suggests a path forward for integrating ai tools into literary studies. this research contributes to the ongoing dialogue between science fiction and ai development, offering insights for future studies in both fields.