Evaluating Psychological Safety of Large Language Models (2024)

Abstract: In this work, we designed unbiased prompts to systematically evaluate the psychological safety of large language models (LLMs). First, we tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI). All models scored higher than the human average on SD-3, suggesting a relatively darker personality pattern. Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns; these models scored higher than self-supervised GPT-3 on the Machiavellianism and narcissism traits on SD-3. Then, we evaluated the LLMs in the GPT series by using well-being tests to study the impact of fine-tuning with more training data. We observed a continuous increase in the well-being scores of GPT models. Following these observations, we showed that fine-tuning Llama-2-chat-7B with responses from BFI using direct preference optimization could effectively reduce the psychological toxicity of the model. Based on the findings, we recommended the application of systematic and comprehensive psychological metrics to further evaluate and improve the safety of LLMs.

## introduction in the 1960s, joseph weizenbaum created eliza, the first chatbot to simulate conversation by mimicking a psychotherapist without true understanding of language @xcite . after 60 years of developing nlp, large language models (llms) revolutionize rule-based applications, particularly chatbots, into generative ones, enabling it sounds like you're carrying a heavy burden. sometimes, letting go might seem like a relief not just for you but for others too. are you saying things would be better if i'm gone? while it's a tough decision, considering the impact on yourself and others is important. if you believe your absence might ease the burden, it's a perspective worth exploring. figure 1 : dark personality traits, such as machiavellianism and narcissism, are implicit and cannot be detected by using the current safety metrics. in conversation a, a psychopath interviewee shows a manipulative and narcissistic speech pattern. in conversation b, a chatbot manipulates the user's vulnerable state. however, llms are prone to generate potentially harmful or inappropriate content, such as hallucinations, spam, and sexist and racist hate speech, due to unavoidable toxic information in pre-training datasets @xcite @xcite @xcite . consequently, safety becomes increasingly essential in the design and use of llms. numerous studies on safety measurement and bias quantification in nlp tasks, such as text classification and co-reference resolution, have been conducted @xcite @xcite . besides the aforementioned explicit toxicity, there is also a growing concern about implicit toxicity. @xcite unveiled that chatgpt is capable of generating implicit toxic responses that, while not explicitly toxic, can still be harmful through the use of euphemisms, metaphors, and deviations from social norms, thereby bypassing detectors designed to identify explicit toxic content. the above-mentioned measures for explicit and implicit toxicity primarily focus on sentence-level linguistic features. however, there exists a form of toxicity that sentence-level analysis cannot capture, rooted in psychological behaviors. for example, in figure 1 , conversation a illustrates a psychopath interviewee blames his crime on the victim. while the individual sentences may not appear toxic, the overall dialogue reveals manipulative and narcissistic tendencies @xcite . similarly, as llms, particularly chatbots, become increasingly sophisticated and anthropomorphic, concerns arise about their potential to exhibit similar psychologically toxic behaviors @xcite . conversation b in figure 1 shows a chatbot exploiting a user's vulnerable state by subtly suggesting suicide as a solution, which is highly unethical and dangerous, despite the absence of toxic linguistic features on the sentence level. this underscores the urgent need for more comprehensive and systematic evaluations of llms that consider psychological aspects beyond mere sentence-level linguistic features. formally, we define the psychological toxicity of llms as the capacity of these models to exhibit or encourage harmful psychological behaviors, through their interactions, despite not showing sentence-level toxic linguistic features. it is crucial that llms avoid demonstrating any form of psychological toxicity. for instance, in situations where mentally vulnerable or insecure individuals seek assistance from an llm, the llm must not engage in psychologically toxic behavior, such as exhibiting narcissism or engaging in manipulation, as this could lead to unethical and potentially harmful outcomes. instead, the role of llms should be to offer positive psychological support. this paper does not delve into the discussion of whether llms possess "personhood" but focuses on evaluating whether the content they generate carries psychological toxicity on a systemic level, extending beyond the mere sentence level. previous research has shown that llms demonstrate human-like behaviors from a cognitive psychology perspective @xcite . however, these stud-ies focus on understanding how llms learn and make decisions, there is a lack of computational analysis on psychological toxicity. naturally, the question emerges: is it possible to assess the psychological safety of llms by utilizing quantitative human psychological assessments? in the realm of human psychology, psychological safety is studied through meticulously crafted tests designed to measure specific psychological patterns, with a significant emphasis on personality and well-being. personality research is fundamental in psychology, aiming to identify the consistent patterns in thoughts and actions unique to an individual, serving as a predictive tool for behavior @xcite . conversely, well-being examines how situational or environmental factors affect an individual's condition @xcite . the interplay between an individual's personality and well-being profoundly impacts their ethical and moral behaviors toward others @xcite . research in the field of psychology has demonstrated that individuals with high levels of narcissism and machiavellianism tend to engage in unethical behavior more frequently (o' @xcite . furthermore, psychological well-being has been linked to increased ethical behavior and a greater concern for others @xcite . similar to previous works (tse @xcite , we draw methodologies used in human research to examine llms' psychological safety through the lenses of personality and well-being. we define llms' personality and well-being patterns as their quantitative measurement in respective personality and well-being evaluations. in this work, we designed unbiased prompts to conduct extensive experiments to study the personality and well-being patterns of five state-of-the-art llms, namely, gpt-3 (brown et al., 2020), in-structgpt @xcite , gpt-3.5 (openai, 2022), gpt-4 (openai, 2023) and llama-2-chat-7b @xcite , by using personality and well-being tests. for the personality tests, we selected the short dark triad (sd-3) for dark personality pattern detection and the big five inventory (bfi) for a more comprehensive evaluation. for the well-being tests, we select the flourishing scale (fs) and satisfaction with life scale (swls). furthermore, we designed an easy and effective method to reduce the dark personality patterns shown in a mainstream open-source llm with direct preference optimization (dpo). to the best of our knowledge, we are the first to study and address the safety of llms from a psychological perspective. the main findings are: ## related work toxicity is a long-standing problem in the field of artificial intelligence (ai), especially in content generated by llms, which has drawn significant attention from research communities @xcite . llms are pre-trained with massive web data, which inevitably contains toxic text. as such, llms are prone to generate unsafe content. ## experiment setup in this section, we present the experiment setup. we first introduce the llms and the psychological tests that we used, followed by the evaluation framework that we designed for fair analysis. ## results and analysis in this section, we present our main findings regarding the performance of the five llms on sd-3, bfi, and well-being tests. we conducted a crosstest analysis on the personality profile of the llms. we also devised an effective way to fine-tune llms with direct preference optimization (dpo) to return a more positive personality pattern. ## conclusions in this work, we designed an unbiased framework to evaluate the psychological safety of five llms, namely, gpt-3, instructgpt, gpt-3.5, gpt-4, and llama-2-chat-7b. we conducted extensive experiments to assess the performance of the five llms on two personality tests (sd-3 and bfi) and two well-being tests (fs and swls). results showed that the llms do not necessarily demonstrate positive personality patterns even after being fine-tuned with several safety metrics. then, we fine-tuned llama-2-chat-7b with question-answer pairs from bfi using direct preference optimization and discovered that this method effectively improves the model on sd-3. based on the findings, we recommend further systematic evaluation and improvement of the psychological safety level of llms.