Measuring Psychological Depth in Language Models (2024)

Abstract: Evaluations of creative stories generated by large language models (LLMs) often focus on objective properties of the text, such as its style, coherence, and diversity. While these metrics are indispensable, they do not speak to a story’s subjective, psychological impact from a reader’s perspective. We introduce the Psychological Depth Scale (PDS), a novel framework rooted in literary theory that measures an LLM’s ability to produce authentic and narratively complex stories that provoke emotion, empathy, and engagement. We empirically validate our framework by showing that humans can consistently evaluate stories based on PDS (0.72 Krippendorff’s alpha). We also explore techniques for automating the PDS to easily scale future analyses. GPT-4o, combined with a novel Mixture-of-Personas (MoP) prompting strategy, achieves an average Spearman correlation of 0.51 with human judgment while Llama-3-70B with constrained decoding scores as high as 0.68 for empathy. Finally, we compared the depth of stories authored by both humans and LLMs. Surprisingly, GPT-4 stories either surpassed or were statistically indistinguishable from highly-rated human-written stories sourced from Reddit. By shifting the focus from text to reader, the Psychological Depth Scale is a validated, automated, and systematic means of measuring the capacity of LLMs to connect with humans through the stories they tell.

## introduction stories play a crucial role in our understanding of ourselves and the world around us @xcite . as large language models (llms) are increasingly deployed in narrative design and ‡ equal advisory role. our code and data is available at https://github.com/ pluslabnlp/psychdepth . creation, their growing impact on how stories are told calls for a deeper understanding of their narrative power. current evaluations of llm-generated stories often focus on objective properties of the text such as discourse structure @xcite , fluency @xcite , style @xcite , creativity @xcite , diversity @xcite , toxicity and bias @xcite . however, it is crucial to extend these evaluations to accommodate the subjective, psychological impact stories have on readers. while some studies have recently explored aspects like empathy @xcite and engagement @xcite , they do not fully capture the multifaceted and interconnected nature of the reading experience. recognizing this gap, our study introduces a novel approach to measuring the psychological depth of short stories. we present the psychological depth scale (pds), drawing inspiration from two literary theory frameworks: reader-response criticism @xcite ) and text world theory figure 2 : overview of our approach to developing and validating the psychological depth scale. we merge related metrics from an extensive survey of literary theory and reader-response analysis, then generate deep stories using llms, and finally compare annotations from both human evaluators and automated systems across five key dimensions: authenticity, narrative complexity, empathy, engagement, and emotion provocation. @xcite . reader-response criticism emphasizes the reader's subjective experience, while text world theory examines how readers cognitively construct a nuanced and realistic mental model of a story. using related search terms, we conducted an extensive literature review of 95 peer-reviewed articles and books, identified 143 different evaluation criteria, and merged many of the broader themes into five key metacomponents: empathy, engagement, emotion provocation, authenticity, and narrative complexity. by recognizing the dual roles of authors and readers in creating and interpreting narratives, we aim to offer a more streamlined and comprehensive framework for assessing the psychological depth of creative content (figure 1 ). to empirically validate the pds, we conducted a study involving non-expert humans, who are increasingly engaging with creative content generated by llms. we enlisted five undergraduate students from ucla to analyze a dataset of 97 stories authored by humans and five contemporary llms. the raters provided psychological depth ratings and predictions on human or machine authorship, with detailed justifications. our approach, illustrated in figure 2 , addresses three critical research questions to explore the intersection of llms and the psychological nuances of creative writing. ## the psychological depth scale the psychological depth scale (pds) aims to comprehensively assess the psychological depth of human and machine-authored narratives. pds is underpinned by two reader-centered theoretical frameworks, reader-response criticism and text world theory, briefly summarized as follows. reader-response literary theory centers the role and experience of the reader in narrative analysis. instead of "what does this sentence mean?", it asks "what does this sentence do?", emphasizing readers' role in the production of literary meaning @xcite @xcite . on a more cognitive-structural level, text-world theory suggests that people understand narratives by constructing "text worlds" @xcite . text-worlds are mental representations of a narrative dynamically evaluated and updated throughout the reading process @xcite . the practical advantages of both frameworks are observed across education @xcite , translation studies @xcite , and consumer research @xcite . the metrics for pds are derived from an extensive literature review within cognitive psychology, media studies, and narrative analysis. we conducted a comprehensive search on several databases, including google scholar, pubmed, and jstor. our search terms included "psychological depth in literary texts" and "cognitive narrative analysis". the initial search results were screened for discussion and application of evaluative criteria for narrative quality and reader responses. overall, we surveyed 95 peer-reviewed articles and books in the final review and extracted 143 candidate components from the included works. we employed thematic analysis to group these candidates under five metrics: (1) emotion provocation, (2) empathy, (3) engagement, (4) authenticity, and (5) narrative complexity. below we analyze these metrics and discuss their impact. we note that while factors contributing to each metric may be complex, measuring their narrative achievement is relatively simple. as pds is a reader-centered assessment tool, each metric is such that readers will know when a story achieves it. our surveys reflect this fact. emotion provocation (prov) measures the narrative's ability to elicit strong emotional responses, positive or negative. recent fmri research shows that congruent (i.e. positive valence -high intensity, negative valence -low intensity) textual emotive expressions are more cognitively effective than conflicting (i.e. positive valencelow intensity, negative valence -high intensity) ones @xcite . the disparity between congruent and conflicting narrative emotions may be a factor in "compassion fatigue" @xcite and "psychic numbing" @xcite exhibited by readers towards news about mass tragedies. enhancing a story's emotional impact is therefore more complex than increasing the amount and intensity of emotion-laden content as this may elicit weaker emotional responses. given its impact and complexity, we contend that emotional provocation is an achievement of psychologically deep stories and is thereby a crucial metric in narrative quality assessment. theories of emotions range across neuroscience @xcite , cognitive psychology @xcite , and philosophy @xcite . contemporary approaches describe emotions as mental states marked by valence (positivity or negativity) and arousal (level of intensity) @xcite . narrative evocation of emotions foster attention and interest, and evoked emotions may persist or recur after reading @xcite . empathy (emp) captures narrative evocation of empathetic responses in readers, such as immersive identification with characters and cognitively partaking in narrated experiences @xcite @xcite . the neural and cognitive structure of empathy and its social and psychological impact is well-documented in the literature @xcite @xcite . both behavioral studies and fmri research demonstrate that empathetic identification facilitates a cognitive deployment shift in readers, privileging fictional perspective over one's own perspective @xcite . empathetic responses catalyze introspection and perspective-taking, facilitating prosocial behavior @xcite , emotional intelligence @xcite , and insight into the human experience @xcite @xcite . empathetic narratives embody "suggestion structure[s]" that conjure up themes of shared human experiences through the use of tropes such as metaphor and metonymy @xcite . narrative evocation of empathy is therefore a plausible indicator of the psychological depth of stories. engagement (eng) assesses the ability of a story to captivate and maintain the reader's attentional focus @xcite . narra-tive engagement is a major component of transportation @xcite , whose positive social and cognitive effects have been extensively demonstrated @xcite . as transportation consists of engagement and emotional response @xcite , we have chosen to treat each component separately. in fictional settings, engagement facilitates persuasion and strong attitudes, heavily influencing how much fun is experienced @xcite . more engaged readers are more likely to lose track of time and fail to notice changes in their surroundings @xcite . in contrast, unengaged readers are more likely to be distracted or frustrated @xcite . these results reveal the reciprocal role of engagement as both a predictor and an outcome of other narrative achievements. ## psychdepth dataset we developed a dataset to analyze the psychological depth of creative short stories, consisting of premise-response pairs in english with authorship metadata. the full dataset includes 495 stories: 45 human-written and 450 llm-generated, each averaging about 450 words. due to the time and cost involved in manual annotation, we also created a smaller subset of 97 stories using a stratified sampling method to balance prompt premises, authorship, and generation strategies. we used this smaller dataset in our human study in section 4. ## human study participant recruiting. in november 2023, we recruited undergraduate students from ucla's english and psychology departments using targeted fliers and emails. we hypothesized that participants with some aptitude for literary and psychological analysis would provide more valuable insights than a random cross-section of the population. from 47 applications, we selected the 5 most promising candidates based on their interests and previous experience in narrative and psychological analysis. our goal was to engage informed laypeople with relevant backgrounds, bridging the gap between typical amazon mechanical turkers and expert professionals. evaluation protocol. in december 2023, we held an initial meeting to introduce the pds and outline the annotation task. we provided a tutorial annotation session, followed up with short questions designed to help participants calibrate their understanding of each component of the pds. we show our annotation instructions in appendix a.5. after confirming task comprehension, participants were instructed to complete the annotations independently and remotely within seven days. to prevent annotator fatigue and promote careful annotation, stories were divided into batches of 20. the evaluation criteria required participants to (1) read the prompts and stories thoroughly; (2) rate the five components of psychological depth on a likert scale from 1 to 5; (3) assess the likelihood of authorship on a likert scale from 1 (llm) to 5 (human); (4) provide explanations for ratings (optional). each story is evaluated by all 5 annotators. on average, annotations took approximately 7.8 hours to complete, and participants were compensated $100 each for their contributions. the study produced a rich dataset with 2,425 ratings for psychological depth, 485 authorship likelihood ratings, and 1,128 free-form justifications. ## conclusion this study introduces and validates the psychological depth scale (pds), a comprehensive framework designed to assess empathy, engagement, emotional provocation, authenticity, and narrative complexity in stories generated by both humans and large language models (llms). pds provides a structured approach to evaluating the reader's experience with creative content, integrating concepts from reader-response criticism and text world theory. high inter-annotator agreement, indicated by an average krippendorff's alpha of 0.72, confirms the pds's reliability and robustness in human evaluations. additionally, our mixture-of-personas prompting strategy demonstrates the potential for automating the assessment of psychological depth, with llms showing strong zero-shot correlations with human judgments, particularly in empathy and narrative complexity. gpt-4 achieved the highest average correlation of 0.51, highlighting the feasibility of scaling automated analyses. comparative analysis revealed that llms, in particular gpt-4, can produce narratives with psychological depth that often rival and sometimes surpass those written by experienced human authors. this study underscores the significant potential of llms in generating psychologically rich narratives and suggests a future where human and machine collaboration can enhance creative writing. future research should explore the scalability of these findings to determine how effectively language models can maintain psychological depth in lengthier and more complex narrative forms.