M irror S tories: Reflecting Diversity through Personalized Narrative Generation with Large Language Models (2024)

Abstract: This study explores the effectiveness of Large Language Models (LLMs) in creating personalized “mirror stories” that reflect and resonate with individual readers’ identities, addressing the significant lack of diversity in literature. We present MirrorStories, a corpus of 1,500 personalized short stories generated by integrating elements such as name, gender, age, ethnicity, reader interest, and story moral. We demonstrate that LLMs can effectively incorporate diverse identity elements into narratives, with human evaluators identifying personalized elements in the stories with high accuracy. Through a comprehensive evaluation involving 26 diverse human judges, we compare the effectiveness of MirrorStories against generic narratives. We find that personalized LLM-generated stories not only outscore generic human-written and LLM-generated ones across all metrics of engagement (with average ratings of 4.22 versus 3.37 on a 5-point scale), but also achieve higher textual diversity while preserving the intended moral. We also provide analyses that include bias assessments and a study on the potential for integrating images into personalized stories.

## introduction despite the profound need for these personalized narratives, there is a noticeable underrepresentation of non-white minority groups in literature @xcite relative to their population size, detailed in appendix figure 6 . the gap in cultural representation highlights the need for more inclusive narratives that reflect diverse reader identities, enhance empathy, and promote cultural awareness @xcite . diversity in literature can lead to improved innovation and a broader consideration of ideas, ultimately enriching the reading experience for all @xcite . advancements in natural language processing, particularly through the development of llms like gpt-4, palm, and llama have introduced the potential to address these gaps on a large scale @xcite @xcite . llms excel in generating humanlike text and adapting content to various contextual needs @xcite . recent studies have investigated llms' capabilities in expressing personality within generated content @xcite and developing methods to induce and edit personality expressions in llm outputs @xcite @xcite . new benchmarks have also been released to assess personality traits in llm outputs @xcite . however, there remains a gap in research concerning whether llms can generate content that incorporates identity traits and faithfully mirrors the diverse identities of a global readership. our study addresses this gap by exploring the potential of llms to create mirror stories-narratives that genuinely reflect and resonate with the identities of individual readers. we present a framework that evaluates the effectiveness of llm-generated mirror stories in comparison to traditional narratives, assessing their impact on engagement, satisfaction, and the perception of personal relevance (see figure 1 ). our contributions are three-fold: ## experiments we conducted two experiments to assess the effectiveness of personalization in llm-generated stories: personalization validation, which validates the integration of identity elements within the narratives, and personalization impact, which assesses the impact of these narratives on user engagement, comprehension, satisfaction, and personalness. prompts in both experiments, personalized prompts incorporating identity elements were used to generate personalized stories. for personalization validation, these elements were specifically asked not to be stated explicitly, to test their seamless integration into the narrative. in the personalization impact experiment, personal elements were aligned with those of 26 human evaluators, ensuring that each story was tailored to evaluators. figure 1 and 2 provide detailed structures of the prompts for both generation and evaluation. ## results are mirrorstories personalized? the effectiveness of personalization in mirrorstories is evident from the high accuracy rates in identifying identity elements by both human and llm evaluators. as shown in figure 3 , human evaluators were particularly adept at identifying gender and ethnicity with accuracies at 100% and 94%, respectively. similarly, gpt-4 showed robust performance, matching or exceeding human accuracy in all categories, which confirms the high level of personalization achieved in the narratives. personalized llm-generated stories also effectively incorporate both the provided moral and the reader's interests, with a stronger emphasis on the moral. to demonstrate this, we used bertopic @xcite for topic modeling to identify the top five terms for each story. we then a g e g e n d e r i n t e r e s t e t h n i c i t y 40% 50% 60% 70% 80% 90% 100% human as an evaluator gpt-4 as an evaluator figure 3: accuracy of human and llm evaluators in identifying identity elements in the story story type correctly identified morals generic human-written 23/24 generic llm-generated 23/23 personalized llm-generated 25/25 are mirrorstories preferred? figure 4 shows that personalized llm-generated stories in mirrorstories are consistently rated higher across all metrics compared to both generic llmgenerated and human-written narratives. this preference is pronounced in evaluations by both humans and gpt-4, with personalized narratives outperforming generic versions, particularly in terms of personalness and engagement where the ratings significantly diverge. ## extended analyses qualitative comparison of human and llm evaluations we examine cases where human and llm evaluators either contradict or agree on the scores assigned to stories, providing insights into the differences in evaluations and preferences for various types of stories. examples of these cases, highlighting instances of both agreement and disagreement between human and llm evaluators, are presented in appendix figure 15 . ## conclusion our study demonstrates the potential of llms in generating personalized narratives that effectively incorporate diverse identity elements and enhance reader engagement compared to generic stories. mirrorstories consists of 1,500 personalized stories that consistently outperform generic ones on key metrics. by making mirrorstories publicly available and integrating it into an interactive web application, we aim to encourage further research on personalized narrative generation, contributing to more engaging and inclusive content. future work could explore out-group perceptions of these narratives, broadening our understanding of personalization's impact across diverse audiences.