P sycho GAT : A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents (2024)

Abstract: Psychological measurement is essential for mental health, self-understanding, and personal development. Traditional methods, such as self-report scales and psychologist interviews, often face challenges with engagement and accessibility. While game-based and LLM-based tools have been explored to improve user interest and automate assessment, they struggle to balance engagement with generalizability. In this work, we propose PsychoGAT (Psychological Game AgenTs) to achieve a generic gamification of psychological assessment. The main insight is that powerful LLMs can function both as adept psychologists and innovative game designers. By incorporating LLM agents into designated roles and carefully managing their interactions, PsychoGAT can transform any standardized scales into personalized and engaging interactive fiction games. To validate the proposed method, we conduct psychometric evaluations to assess its effectiveness and employ human evaluators to examine the generated content across various psychological constructs, including depression, cognitive distortions, and personality traits. Results demonstrate that PsychoGAT serves as an effective assessment tool, achieving statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity. Moreover, human evaluations confirm PsychoGAT’s enhancements in content coherence, interactivity, interest, immersion, and satisfaction.

## introduction psychological assessment plays a pivotal role in mental health and self-understanding @xcite , such as depression measurement, cognitive how often have you been bothered by any of the following problems? to address these challenges, there has been a shift towards automated, procedural methods for psychological assessments, including computerized adaptive testing @xcite and rule-based electronic games specifically designed for measurements @xcite . besides, some studies have explored the use of natural language processing (nlp) systems to support mental health @xcite @xcite . however, these solutions are often limited by fragmented functionality and narrow applicabil-ity. particularly, rule-based psychological games lack flexibility and struggle to engage users due to their inherent rigidity @xcite . large language models (llms), known for their general-purpose capabilities, extensive knowledge, and reasoning abilities, offer a more flexible, knowledgeable, and engaging approach to psychological assessments @xcite @xcite @xcite @xcite . for example, diagnosis of thought (dot) @xcite utilizes llms to detect cognitive distortion via the chain of thought techniques @xcite . another typical way for llm-based psychological assessments is psychologist role-playing dialogue @xcite @xcite . despite these advancements, current systems face limitations in generalizability across diverse psychological constructs, often due to their reliance on specific prior knowledge @xcite . concurrently, psychologist role-playing dialogues often suffer from a lack of interestingness, leading to user disengagement @xcite . in response to these issues, we propose a novel paradigm for interactive, immersive, and automatic psychological assessments, namely psychogat (psychological game agents). to the best of our knowledge, our method is the first one that incorporates llm agents for generic and gamified psychological measurements. as illustrated in figure 1 , our approach transforms traditional selfreport scales for various psychological constructs into interactive fiction games, enabling players' interactions to serve as accurate indicators of their mental states. specifically, as detailed in figure 2 , our framework comprises three main agents: (1) the game designer, responsible for creating the game's settings; (2) the game controller, tasked with generating interactive content; and (3) the critic, which refines the content to enhance the user experience. in summary, our contributions are: ## psychogat as in figure 2 , our framework comprises three agents responsible for the game generation, in addition to a separate agent designed for simulating human interactions. the subsequent section will commence with an overview of the workflow ( §2.1), followed by detailed descriptions of each agent involved: the game designer ( §2.2), the game controller ( §2.3), the critic ( §2.4), and the human simulator with a psychometric evaluator ( §2.5). detailed prompt templates are in appendix d. ## experiments this section introduces the experimental tasks, baseline methods, and evaluation protocol in §3.1. then, the evaluation metrics are detailed in §3.2. next, we assess psychogat's effectiveness in psychological measurements and compare various llm-based assessment methods in §3.3. furthermore, ablation studies on psychogat agents and game scene robustness are presented in §3.4. more experimental discussions are in appendix b. ## related work gamified assessments in psychology. gamebased psychological assessments not only achieve high accuracy but also increase engagement and 2019 ), identify and evaluate reading challenges in young students @xcite , and gauge cognitive functions @xcite . yet, these studies primarily rely on the manual designs of psychologists and game designers, leading to a high investment of time, financial resources, and human effort. llm-assisted psychology. the integration of artificial intelligence, especially via llms, has opened up abundant new avenues for psychological research across educational, social, cultural, clinical, and counseling domains @xcite @xcite @xcite @xcite . llms exhibit stable personalities @xcite , express nuanced emotions @xcite , mimic human cognition @xcite , and facilitate various social science simulations @xcite . in psychological assessments, llms introduce innovative approaches, transcending traditional rating scales by using natural language processing for mental health evaluations @xcite . research has utilized language models for predicting mental health conditions from existing textual data @xcite @xcite and in conducting llm-enabled roleplaying interactions @xcite . furthermore, the in-context learning capabilities of llms are being investigated for their potential to reframe cognitive distortions and assist psychological inter-ventions @xcite . ## conclusion in this work, we propose psychogat, an innovative paradigm for psychological assessments, leveraging llm agents to transform the self-report scale into interactive fiction games. our framework consists of three llm-based agents: a game designer, a game controller, and a critic. through recurrent interactions among these agents, we can construct a psychological game for players to engage with. the behaviors exhibited by players within this game effectively mirror their psychological profiles, making this interactive fiction game a reliable and valid method for engaging and immersive psychological testing. our experimental results not only affirm its potential as an alternative to traditional self-report scales but also demonstrate an enhanced user experience, which can significantly reduce the potential resistance of participants during psychological testing or diagnosis. our work represents the initial step towards leveraging llmbased agents for generic and gamified psychological assessment, and we hope it paves the way for future research to explore the intersection between llm agents and psychotherapy, thereby promoting the acceptance and accessibility of psychological diagnostics and treatments among the public.