Speaker Identification and Dataset Construction Using LLM s: A Case Study on J apanese Narratives (2025)

Abstract: Speaker identification in narrative analysis is a challenging task due to complex dialogues, diverse utterance patterns, and ambiguous character references. Cosly and time-intensive manual annotation limits the scalability of high-quality dataset creation.This study demonstrates a cost-efficient approach of constructing speaker identification datasets by combining small-scale manual annotation with LLM-based labeling. A subset of data is manually annotated and is used to guide LLM predictions with a few-shot approach followed by refinement through minimal human corrections. Our results show that LLMs achieve approximately 90% accuracy on challenging narratives, such as the “Three Kingdoms” dataset, underscoring the importance of targeted human corrections. This approach proves effective for constructing scalable and cost-efficient datasets for Japanese and complex narratives.

## introduction narrative analysis is essential for understanding cultural values, psychological dynamics, and creative processes. examining narrative structures and themes provides valuable insights into societal norms and human behavior @xcite . large language models (llms) @xcite have introduced new possibilities in narrative analysis, enabling tasks such as character emotion analysis and plot progression prediction. speaker identification, a key task in narrative analysis, involves accurately attributing dialogue to characters and understanding character dynamics within a story. however, constructing highquality speaker identification datasets is costly and labor-intensive, requiring consistency and attention to paraphrase variations @xcite @xcite @xcite . to address these challenges, we employ a collaborative approach to dataset construction, combining llm-based initial annotations with targeted manual corrections @xcite . this significantly reduces annotation costs while maintaining quality. inspired by the pdnc dataset @xcite , we annotate both primary speaker names and their paraphrased forms (aliases). this dual annotation improves efficiency and flexibility. figure 1 outlines our framework: llm predictions, followed by iterative human correction, encompassing dialogue extraction, speaker labeling, and refinement. existing speaker identification datasets have primarily focused on english and chinese, limiting the scope of research to these languages. to address this, we first constructed a speaker identification dataset for the japanese narrative "romance of the three kingdoms", a japanese translation of the original chinese work, chosen for its complex plot and character interactions, leveraging data from aozora bunko foot_0 . this method demonstrated the feasibility of creating highquality datasets with reduced annotation costs. our results show that llms achieve approximately 90% accuracy, even without human corrections, while human intervention further enhances accuracy. additionally, this approach significantly lowers the cost of dataset creation, making it scalable for larger and more diverse datasets. we also highlight the critical role of contextual input length in improving llm performance, providing valuable insights for handling complex narratives. despite these advancements, existing datasets are primarily limited to english or chinese, with no publicly available datasets for japanese. moreover, since these datasets depend on manual labor for annotation, they are inherently labor-intensive and costly to produce. ## methods task definition speaker identification in narrative analysis involves determining which character or entity is responsible for a given utterance. this process requires analyzing both the utterance and its context to accurately attribute it to the correct speaker. in our approach, the set of possible speakers s is not predefined but derived from the context of the input text. given a set of utterances @xmath0, u 2 , . . . , u m , we establish a mapping function f : @xmath1. we annotated two types of speaker names: the 'main name,' representing the most contextually appropriate identifier (e.g., elizabeth bennet), and 'candidates,' which include alternative names or alternative forms (e.g., lizzy, liz, elizabeth). this dynamic speaker identification is crucial for capturing the fluid and complex nature of narrative interactions, enabling more accurate analysis of character relationships and narrative structure. ## dataset construction the dataset construction was carried out according to the following steps, as shown in figure 2 . step 2: speaker labeling we utilized an llm to identify and label the speakers in the extracted dialogues. as the llm, we used llama-3-70b-instruct with a few-shot setting, which showed the highest performance on the development dataset (see appendix b and g). ## experiment to assess llm capability in speaker identification and, simultaneously, to validate the quality of our constructed dataset, we conduct a series of experiments evaluating llm performance. a primary aim of these experiments is to identify the characteristics of llms that facilitate efficient and effective dataset construction, allowing for the identification of optimal model features for similar tasks. ## conclusion we collaborated with llms to create a speaker labeling dataset by annotating "romance of the three kingdoms" from aozora bunko in japanese. the dataset included 15,412 entries. using llms like llama-3, we achieved a substring match ratio of approximately 90%. to handle multiple potential speakers, we developed a paraphrase dataset to improve evaluation accuracy. instead of manually annotating the entire dataset, we adopted an approach where llms performed the initial labeling, and human annotators focused on correcting the generated labels. this shift significantly reduced human labor costs while maintaining high annotation quality. our findings demonstrate the potential of scalable, llm-assisted methods for narrative analysis, offering a cost-effective solution for speaker identification in complex texts.