Mitigating Hallucination in Fictional Character Role-Play (2024)

Abstract: Role-playing has wide-ranging applications in customer support, embodied agents, and computational social science. The influence of parametric world knowledge of large language models (LLMs) often causes role-playing characters to act out of character and to hallucinate about things outside the scope of their knowledge. In this work, we focus on the evaluation and mitigation of hallucination in fictional character role-play. We introduce a dataset with over 2,000 characters and 72,000 interviews, including 18,000 adversarial questions. We propose RoleFact, a role-playing method that mitigates hallucination by modulating the influence of parametric knowledge using a pre-calibrated confidence threshold. Experiments show that the proposed method improves the factual precision of generated responses by 18% for adversarial questions with a 44% reduction in temporal hallucination for time-sensitive interviews. The code and the dataset are available at https://github.com/NafisSadeq/rolefact.git .

## introduction the performance of llms on simple downstream tasks such as question answering @xcite ), text classification @xcite , natural language understanding @xcite , etc. has improved significantly @xcite , prompting the question whether they can perform more complex agent level tasks such human simulacra @xcite , tool use @xcite , and roleplaying @xcite . role-playing aims to enable an ai system to impersonate a domain expert @xcite , companion for psychological support @xcite , or a fictional character @xcite ) by mimicking their knowledge, personality traits, and speaker style. applications of role-playing include customer support @xcite , embodied agents @xcite , and computational social science (zhou et al., 2024). while llms like gpt-4 (openai, 2024a), gemini @xcite , llama-3 (meta, 2024) demonstrate some role-playing capabilities, the role-playing agent often hallucinates when asked about things that should be outside of their knowledge base @xcite . hallucination remains a challenge for most llm-based systems @xcite . while retrieval augmented generation (rag) @xcite ) may mitigate some hallucination induced by outdated knowledge in language models @xcite , it cannot guarantee that most claims made by the system are supported by a non-parametric knowledge base @xcite . learning to improve character role-playing methods so that the responses are verifiable by a story-specific script serves as a case study that can help mitigate hallucination for the rag paradigm in general. llms are pretrained on a large corpus of internet data and their vast parametric knowledge helps them perform a diverse range of tasks @xcite . however, the parametric knowledge of llms makes the roleplaying character vulnerable to out-of-character behaviors and hallucinations @xcite . a role-playing character may hallucinate about things that are outside of their story-line (e.g. hiccup from 'how to train your dragon' talking about learning a fire-making spell at hogwarts) or demonstrate knowledge of events ahead of time (harry potter as a first-year hogwarts student talks about producing the patronus charm). a role-playing character should be able to demonstrate a lack of knowledge about things that are not supported by a story-specific script or knowledge outside of a relevant temporal window in case of time-sensitive role-play. one approach to achieve this is to strictly remove parametric knowledge from the response and solely rely on non-parametric retrieved knowledge. however, this approach degrades informativeness and it can even hurt factual precision. this is shown in figure 2 where we anonymized the prompts to minimize the role of parametric knowledge. we argue that an ideal role-playing approach should modulate the influence of parametric knowledge to find the balance between factuality and informativeness. prior methods of role-play mostly rely on roleprofile prompting @xcite ) and knowledge retrieval in a time-agnostic fashion @xcite . these approaches heavily depend on parametric knowledge of llms which leads to hallucinations and poor performance for less popular characters (shown in figure 3 ). currently, a major challenge for studying hallucination in character role-play is the lack of a suitable dataset. the lack of a dataset forces researchers to rely on a ratingbased evaluation of hallucination @xcite which is highly subjective. prior datasets in the role-play domain are insufficient for studying time-sensitive character role-play or hallucination for less popular characters. our contributions are as follows: • we release a dataset for script grounded character role-play (sgr) that includes more than 2,000 characters, and 72,000 interviews. with script-specific knowledge and 4 diverse tasks, the dataset aims to enable a systematic study of character hallucinations. to the best of our knowledge, this is the largest and the most diverse dataset that facilitates the study of temporal hallucination and hallucination for less popular characters. script-specific knowledge aligned with the given tasks also facilitates a more nuanced evaluation of hallucination based on atomic fact decomposition @xcite . ## methodology rolefact aims to mitigate hallucination by modulating the influence of parametric knowledge within character responses. the overview of the method is shown in figure 4 . rolefact generates a response using a character profile and retrieved knowledge and updates the response based on atomic fact verification using both parametric and non-parametric retrieved knowledge. all facts supported by retrieved knowledge may remain in the final response. facts that are only supported by parametric knowledge of llms remain in the final response if the confidence is above a calibrated threshold. let @xmath0. let ret : x × n × @xmath1. the retrieved knowledge may be denoted as u @xmath2. let @xmath3. the response generation function irg : x × p × u @xmath4, the role profile p, and the retrieved knowledge u n to produce an intermediate response z. the prompt used for irg is shown below (ignoring line breaks). irg: you will role-play @xmath5. your task is to respond to the following dialogue context. if the question matches a scene from your storyline, please reuse the original lines from the story. you will respond and answer @xmath6, manner, and @xmath7. your character description is: <p>. relevant scenes for the given context are as follows: <u n >. dialogue context: <x>. the response z is decomposed into a list of atomic facts a z with a function dec : @xmath8. dec : i will give you an utterance from a movie or a play. your task is to provide me with a list of atomic facts expressed in the given utterance. each atomic fact should be described in a name-only third-person format. utterance: <z>. each atomic fact @xmath9. the functions for fact-checking against non-parametric and parametric knowledge bases are denoted as fcr : a z × u @xmath10, 1} and fcs : a z × @xmath11, 1} respectively. a set of verified atomic facts, a y , is constructed such that if a is supported by u n , it is added to a y . conversely, if a is not supported by u n , it is only included in a y if it appears to be supported by v within a reasonable confidence threshold t. let's assume the function fcs is run m times using an llm and a is supported k times by v . the fact a is added to a y if k @xmath12. the sample size m and the confidence threshold t are calibrated on a validation set. fcr: consider the given statement and the evidence knowledge sources. indicate whether the statement is supported by the knowledge sources. negation of a false statement should be considered supported. statement: <a>. evidence @xmath13. fcs : consider the given statement @xmath14. indicate whether the statement is supported by the storyline @xmath15. negation of a false statement should be considered supported. statement: <a>. the final response y is generated by updating the intermediate response z via self-reflection conditioned on x. during self-reflection, all unverified atomic facts a z \ a y are removed from z. the self-reflection update function is denoted as sru : x × z × (a z \ a y ) → y . sru : consider the following response generated by an ai role-playing @xmath16. the response may contain one or more unsupported claims as a result of hallucination. the unsupported claims are listed below. rewrite the response to remove all the unsupported claims from the response. if the hallucination stems from a wrong assertion made in the original query, feel free to clarify that. original @xmath17. @xmath18. unsupported @xmath19. ## experiments we conduct experiments with three llm backbones, namely vicuna-7b-1.5 @xcite , llama-3-8b-instruct (meta, 2024), and gpt-3.5-turbo. for retrieval, we consider bm25 (robertson and zaragoza, 2009), s-bert @xcite , and contriever @xcite . 1,000 randomly chosen samples from each task type constitute the validation set. the validation set is used for calibrating the confidence threshold, sample size, the retrieval method, and the number of documents to retrieve. decoding for character response generation is performed with @xmath20.7 and @xmath21.95 for all models. baselines our primary baseline is similar to @xcite . we prompt an llm with a role prompt, a role profile, and retrieved dialogues relevant to the query. we also consider two additional variants of the primary baseline designed to improve the factual precision of role-play. the first variant rewrites the response of the primary baseline by removing facts absent from the retrieved knowledge. this is denoted as knowledge-guided rewriting (baseline + kgr) baseline. the second variant updates the primary response using self-reflection to identify and mitigate hallucination based on llm self-confidence. this is denoted as the self-reflection (baseline + sr) baseline. evaluation metrics we measure factual precision with fact score @xcite . all atomic facts are converted to a 'name-only third person' format for fact-checking. we calculate the number of supported atomic facts per response (sfpr) to evaluate informativeness. we also measure temporal hallucination rate (thr), which is defined as the number of atomic facts associated with temporal hallucination for every 100 responses. thr was determined manually after analyzing unsupported facts from 500 samples from dialogue completion and scene-grounded tasks. for human evaluation, we use a rating between one and seven to rate 200 responses in terms of factuality, informativeness, and speaker style imitation. results figure 7 shows the performance of rolefact with respect to different retrieval approaches and number of retrieved documents. bm25 outperforms off-the-shelf dense retrieval methods in most cases. interestingly, we find that increasing the number of retrieved documents beyond five may hurt factual precision. this phenomenon may be explained by similar findings for rag @xcite where increased irrelevant context added to the prompt exacerbates hallucination. ## related work language models for role-play building autonomous agents that can simulate human behavior with individualized traits is a long-standing challenge @xcite . prior works in nlp build conversational agents grounded with various individualized traits, such as background stories @xcite , demographic information @xcite , social roles @xcite , and response style constraints @xcite . in this work, we focus on building dialogue systems that are aligned with user perceptions of fictional characters, known as character role-play @xcite @xcite . in terms of methodology, prior works can be categorized as training-free role-prompting @xcite @xcite or instruction-tuning @xcite . role-prompting includes static character descriptions @xcite , ## conclusion in this work, we focus on the mitigation of hallucination for character role-play. we introduce the sgr dataset which provides story-specific knowledge aligned with task-specific interviews for a diverse set of characters. the dataset is specifically designed for studying hallucinations of various types such as cross-universe hallucinations and temporal hallucinations. sgr is the first dataset to facilitate automated evaluation of hallucination and analysis of factuality for less popular characters. we propose rolefact, a role-playing method that mitigates hallucination via parametric and nonparametric verification of atomic facts. rolefact improves the overall factual precision by up to 18.4%. an interesting direction for future research is to instruction-tune an open-weight llm with script-specific knowledge and task-specific interviews from the sgr dataset for generating factual and informative responses.