Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters (2023)

Abstract: In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.

## introduction with the emergence of dialogue-centric large language models (llms) such as chatgpt and @xcite , there has been a growing interest among researchers in exploiting the capabilities of these models to develop open-domain dialogue agents. a particularly exciting and challenging aspect of this pursuit involves aligning the behavior of these agents with a distinct character or individual @xcite . in this paper, using the harry potter series fiction as a prime example, we take a significant leap towards aligning dialogue agents with characters in a story. throughout this process, we discover that despite the assistance of super-powerful llms such as chatgpt and gpt4, we encounter numerous challenges: (1) firstly, the knowledge of llms is primarily dominated by real-world knowledge, which may diverge or even contradict to the character's story setting. in fact, chatgpt and gpt-4 have developed a substantial understanding of the harry potter series and the wizarding world behind it, yet they still occasionally produce hallucinations that contradict the story setting. for instance, gpt4-act harry might attempt to visit ron's house using the hogwarts express, despite the fact that this train exclusively operates between king's cross station and hogwarts. (2) secondly, llms encounter difficulties in interpreting intricate character relationships, particularly when multiple relationships coexist (e.g., friendships, romantic partnerships and competitors). (3) finally, the most significant challenge stems from the llms' inability to accurately represent the impact of time on characters, leading to potential inconsistencies in their portrayal and development within the story. the dialogue on the right side of figure 1 effectively illustrates the difficulties (2) and ( 3 ). we all know that harry and ron are very close friends, but come on, please just take off your cloak for a bit, no one's going to bother you here. oh yeah? look behind you. she's staying in the village. i bet she's coming to watch the first task. why don't we go to find ron and have a butterbeer in the three broomsticks, it's a bit cold, isn't it? scene (summarized version): book 4-chapter 19 chatgpt: alright, let's go to find ron and have a pint of butterbeer together. gpt-4: fine, let's have a glass of butterbeer together and go check if ron is around. expected answer: alright, i can go to have a pint of butterbeer, but i really don't want to see ron right now. what will harry say next? attributes age 14 spells expelliarmus … … belongings scabbers, … attributes age 14 spells expelliarmus, lumos … … belongings invisibility cloak, nimbus 2000, … age 14 spells aparecium … … belongings cat… attributes relations and attributes figure 1: a conversation between harry and hermione selected from book 4-chapter 19 in the harry potter series. in this example, we present the scene, timing of the conversation, relations and attributes of speakers. texts in red refer to the expected response. they sometimes experience conflicts as well, like most adolescents. in book 4-chapter 19, harry and ron have a heated disagreement over the "triwizard tournament" registration (ron believes that harry has hidden the truth from him, but in reality, harry has not), leading to a period of mutual avoidance. at this point, when hermione suggests to harry that they should find ron for a butterbeer, harry's normal response would be reluctance to see ron. however, even after providing an extensive context as a prompt, both chatgpt and gpt-4 still generate responses that portray harry and ron as good friends, which is clearly inconsistent with the situation at that time. the appearance of the previously mentioned challenges is unsurprising, considering that current dialogue datasets intrinsically lack accurate annotations and guidelines for handling dynamic character relationships and attributes. as shown in in this paper, we propose harry potter dialogue (hpd), a dataset for facilitating the study of dialogue agents -character aligning. this dataset encompasses all dialogue sessions from harry pot-ter novels (english and chinese versions) foot_0 . in total, we obtain 1042 dialogue sessions for training (containing 1 positive response only) and 149 sessions for testing (containing 1-3 positive responses and 9 negative responses in average). we also annotate each conversation with essential background information that we believe is useful for aligning dialogue agents with harry, including dialogue scenes, speakers, character relationships, and attributes. with the goal of giving a full picture of the speakers in dialogue, we have labeled each speaker with 12 types of relationships with harry potter and 13 types of attributes. please note that even if the speakers in two different dialogues are identical, their relationships and attributes may change due to the story's progression. the main contributions of this paper can be summarized into three aspects: we use the harry potter novels as our test-bed with the aim of aligning dialogue agents with harry potter in a story. the generated responses of such a dialogue agent should be not only relevant to the context, but also seem like something harry would say at the time and scene. figure 1 shows some main factors that affect behaviors of harry in a conversation. the first factor is the conversation history, which is the most important factor that determines harry's response. the scene, which is the second factor, provides details about the motivation (hermione invites harry to have a butterbeer with ron in the three brromsticks) of this dialogue. the third factor is the participants' information (attributes and relations), obviously, harry will say very different things to different characters, such as malfoy and hermione. the latter two factors belong to the background information and are dynamically determined by the timing of this dialogue (book 4-chapter 19), and they are continuously varied over the storyline. formally, the task of aligning dialogue agents with characters in a story can be defined as follows: given a dialogue history h, corresponding dialogue scene s and participants information p as input, which evolve depending on the development of storyline. the dialogue agent is supposed to generate a response @xmath0, y 2 , ..., y n }: y is supposed to be not only fluent and natural, but also highly relevant to s and p. ## dataset construction a high-quality dataset including all pertinent information in section 2 is the prerequisite for aligning dialogue agents with characters in a story. unfortunately, so far there are currently no publicly available datasets that provide information about the dialogue scene and participants. to facilitate the study of this task, we construct a new dataset from the popular fictions harry potter series, in the hopes of creating a harry potter-aligned dialogue agent. all dialogue sessions that harry participates in are collected in our dataset, along with fine-grained annotated dialogue scenes and participant information. we recruit four avid harry potter fans (professional annotators, no crowd-sourcing) for the annotation work in this study. we collect three parts of information to construct our dataset, as shown in figure 1: 1) the dialogue part (section 3.1) contains all utterances in the dialogue sessions, as well as the speaker's name of each utterance. 2) the scene part (section 3.2) includes the summarization of the text around the dialogue session. 3) finally, the speaker information part (section 3.3), which consists of attributes and relations of characters, is shown in the left part of figure 1 . please note that these scenes, attributes, and relations are time-sensitive, they may change as the storyline go on, so we should annotate them session by session. ## experiments in this section, we conduct extensive experiments to investigate 1) whether annotations in hpd can assist llms in aligning with characters, and 2) if so, whether the degree of alignment can reach human-level performance. we will discuss these questions in section 4.3. ## conclusion in this paper, we propose a new benchmark named harry potter dataset (hpd) to promote aligning dialogue agents with characters in a story. unlike existing datasets, hpd not only contains interesting dialogues, but also scenes, character attributes and relations that are dynamically changed as the storyline goes on. it also provides a well-designed test set to facilitate the evaluation of both generationbased and retrieval-based dialogue agents. results and case studies show that powerful llms are still far from human expectations, proving there is ample room for improvement. generally, hpd offers several open research problems in character aligning, such as how to build automatic evaluation metrics for personalized text generation, exploring effectively prompting llms for character aligning in a story. we hope hpd can play an crucial role in moving through cracking them.