P sy C o T : Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection (2023)

Abstract: Recent advances in large language models (LLMs), such as ChatGPT, have showcased remarkable zero-shot performance across various NLP tasks. However, the potential of LLMs in personality detection, which involves identifying an individual’s personality from their written texts, remains largely unexplored. Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, we argue that these items can be regarded as a collection of well-structured chain-of-thought (CoT) processes. By incorporating these processes, LLMs can enhance their capabilities to make more reasonable inferences on personality from textual input. In light of this, we propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. In particular, we employ a LLM as an AI assistant with a specialization in text analysis. We prompt the assistant to rate individual items at each turn and leverage the historical rating results to derive a conclusive personality preference. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection, achieving an average F1 score improvement of 4.23/10.63 points on two benchmark datasets compared to the standard prompting method. Our code is available at https://github.com/TaoYang225/PsyCoT .

## introduction personality, as an important psychological construct, refers to individual differences in patterns of thinking, feeling, and behaving @xcite . consequently, detecting one's personality from their generated textual data has garnered considerable interest from researchers due to its wide-ranging applications @xcite ; ## psychological questionnaire as cot for the current generative llm, personality detection can be formulated as a reading comprehensive task. formally, given a set of texts @xmath0, x 2 , . . . , x n } written by an individual, the goal is to determine the correct personality option @xmath1. we can achieve the goal by designing the appropriate prompt @xmath2, i}, in which d represents the task description prompt that informs the llm about the task's definition, and i represents the inference prompt that pushes the llm to select the desired personality trait in a specified format. ## experiment 4.1 datasets we evaluate psycot on two public datasets: essays (pennebaker and king, 1999) and kaggle foot_2 . the essays comprises 2468 anonymous texts written by volunteers in a strict environment. these volunteers were instructed to write whatever came to their mind within a short time-frame. each essay is associated with the author's big five personality traits, namely agreeableness, conscientiousness, extraversion, neuroticism, and openness. the authors assessed their own traits using a 44-item big five inventory @xcite , which is the same questionnaire we employ in psycot. to ensure fair comparisons with fine-tuned methods, we randomly divided the dataset into training, validation, and testing with proportions of 80-10-10, respectively. for prompt-based methods, we evaluate their zero-shot performance on the test set. the kaggle dataset is collected from personality-cafe foot_3 , a forum where people share their personality traits and engage in discussions about personalityrelated topics. the kaggle comprises a total of 8675 user's data, each consisting of 45-50 posts. the used personality taxonomy is mbti (myers-briggs, 1991), which divides personality into four dimensions: introversion or extroversion (i/e), sensing or intuition (s/n), thinking or feeling (t/f), and perception or judging (p/j). limited by api resources, we randomly selected 200 samples form the test set to evaluate psycot and baselines. for the essay dataset, the average token count per sample is 762.38, whereas for kaggle, it stands at 1632.38. more details of the two datasets and the used questionnaires are provided in appendix c and appendix d, respectively. ## conclusion in this paper, we propose a novel method, psy-cot, for zero-shot personality detection. psycot prompts the llm to engage in reasonable personality reasoning. unlike the standard prompting approach that directly requests the llm to infer the answer, psycot draws inspiration from selfassessment personality tests that evaluate an individual's personality through well-designed questionnaires. psycot introduces items from the questionnaire as a set of rigorous cot. it guides the llm to evaluate each item based on the author's text and then infer the final personality trait. extensive experiments on two benchmark datasets demonstrate that psycot outperforms standard prompting and zero-shot-cot approaches by a large margin. it also achieves comparable performance to some fine-tuned models. this work represents the first effort to leverage psychological questionnaires to elicit the llm's reasoning abilities, providing a new perspective for exploring the use of questionnaires in the era of llms.