Systematic Evaluation of GPT -3 for Zero-Shot Personality Estimation (2023)

Abstract: Very large language models (LLMs) perform extremely well on a spectrum of NLP tasks in a zero-shot setting. However, little is known about their performance on human-level NLP problems which rely on understanding psychological concepts, such as assessing personality traits. In this work, we investigate the zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users’ social media posts. Through a set of systematic experiments, we find that zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification upon injecting knowledge about the trait in the prompts. However, when prompted to provide fine-grained classification, its performance drops to close to a simple most frequent class (MFC) baseline. We further analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors that suggest ways to improve LLMs on human-level NLP tasks. The code for this project is available on Github.

## introduction human-level nlp tasks, rooted in computational social science, focus on the link between social or psychological characteristics and language. example tasks include personality assessment @xcite @xcite , demographic estimation @xcite , and mental health-related tasks @xcite @xcite . although using lms as embeddings or fine-tuning them for human-level nlp tasks is becoming popular @xcite @xcite , very little is known about zeroshot performance of llms on such tasks. in this paper, we test the zero-shot performance of a popular llm, gpt-3, to perform personal-ity trait estimation. we focus on personality traits because they are considered the fundamental characteristics that distinguish people, persisting across cultures, demographics, and time @xcite . these characteristics are useful for a wide range of social, economic, and clinical applications such as understanding psychological disorders @xcite , choosing content for learning styles @xcite or occupations @xcite , and delivering personalized treatments for mental health issues @xcite . focusing on zero-shot evaluation of gpt-3 on these fundamental characteristics forms a strong benchmark for understanding how much and what dimensions of traits gpt-3 encodes out-of-the-box. further, while fine-tuned lms have only had mixed success beyond lexical approaches @xcite , using zero-shot capable llms could help lead to better estimates. the nlp community has a growing interest in understanding the capabilities and failure modes of llms @xcite , and we explore questions that surround llms in the context of fundamental human traits of personality. zero-shot performance can depend heavily on the explicit information infused in the prompt @xcite . personality, defined by information in its well-established questionnaire tests, presents new opportunities for information infusion. our contributions address: (1) what information about personality is useful for gpt-3, (2) how its performance compares to current sota, (3) the relation between ordinality of outcome labels with performance and (4) whether gpt-3 predictions stay consistent given similar external knowledge. ## background psychological traits are stable individual characteristics associated with behaviors, attitudes, feelings, and habits (apa, 2023). the "big 5" is a popu-lar personality model that breaks characteristics into five fundamental dimensions, validated across hundreds of studies across cultures, demographics, and time @xcite . the approach is rooted in the lexical hypothesis that the most important traits must be encoded in language @xcite . we investigate all five factors from this model: openness to experience (ope-intellectual, imaginative and open-minded), conscientiousness (con-careful, thorough and organized), extraversion (ext-energized by social and interpersonal interactions), agreeableness (agr-friendly, good natured, conflict avoidant) and neuroticism (neu-less secure, anxious, and depressive). llms like palm @xcite have shown significant improvement in performance on various nlp tasks @xcite , even without finetuning. there is a growing body of work investigating one of the ubiquitous llms, gpt-3, under different settings @xcite @xcite . inspired by this, we systematically study the ability of gpt-3 to perform personality assessment under zero-shot setting. following evidence that incorporating knowledge about the task can improve performance @xcite @xcite , we evaluate the impact of three different types of knowledge to determine which type improves personality estimation. modeling personality traits through natural language has been extensively studied using a wide range of approaches, from simple count-based models @xcite to complex hierarchical neural networks @xcite . finetuning lms has become the mainstream approach for this task only recently @xcite . with the advent of gpt-3, zero-or few-shot settings have become the primary approach to leverage llms in other nlp applications, but are yet untested for personality estimation. ## dataset to get a sample of language associated with personality, we followed the paradigm set forth in @xcite whereby consenting participants shared their own facebook posts along with taking a battery of psychological assessments, including the big five personality test @xcite . the dataset comprises of 202 participants with outcomes of interests who had also shared their facebook posts. first, we filter the data to only include user posts from the last year of data collection @xcite . next, we only retain users for whom we have exactly 20 facebook posts, similar to the approach described in other human-level nlp works @xcite . finally, we anonymize the data by replacing personable identifiable information using scipy's (virtanen et al., 2020) ner model. we also remove phone numbers and email ids using regular expressions. finally, we are left with anonymized facebook posts for 142 users and their associated 5 personality traits. this population (all from us) has a gender ratio of 79:18:3 (female:male:others). the age ranged from 21 to 66 (@xmath0. the big 5 personality trait scores fall in the continuous range of @xcite . we discretize the outcome values into the desired number of bins/classes using a quantile discretizer (in pandas). we explain why we choose to discretize the outcome values in §4. ## experimental design in this work, gpt-3 is evaluated in a zero-shot setting. we frame the problem of personality prediction as classifying the degree (i.e. high/low or high/medium/low) to which a person exhibits a trait. ideally, because the big 5 are considered continuously valued variables @xcite , one would model as a regression task, but we found this simplification to classification necessary to get any meaningful insights from gpt-3's zeroshot capability. we also investigate the degradation of performance for tertiary classification instead of binary in §5. we devise a simple, reasonable prompt (basic) foot_0 to first estimate the ability of gpt-3 to predict the big 5 personality traits. building on this, we investigate whether adding external knowledge about these traits helps the model perform better. we use three types of knowledge: (1) textbook: a concise definition of these traits from @xcite , (2) wordlist: frequent and infrequent words foot_1 used by people exhibiting those traits, and (3) itemdesc: survey items foot_2 (a positive and a negative) users responded to, based on which their personality scores were estimated. 2002 ), wordlist refers to adding the top 5 positively and negatively correlated unigrams with the trait reported by @xcite , itemdesc refers to adding the items that were a part of the personality questionnaire ( the findings indicate a statistically significant distinction when compared to the wt-lex model, with significance levels of @xmath1.05 (*), @xmath2.01 ( †), and @xmath3.001 ( ‡). baseline and evaluation. the baseline, wt-lex, is a ridge regression model from park et al. 2015 trained on dimensionally reduced feature set of n-grams and lda-based topics extracted from @xcite facebook data. the number of parameters in this model is orders of magnitude less than gpt-3. even complex neural models (lynn et al., 2020) have been unsuccessful to surpass its performance. wt-lex also produces predictions in the continuous scale within the range of @xcite . in order to make a fair comparison with gpt-3, we perform the quantile discretization described in §3 and calculate macro f1. we evaluate the predictions using macro f1 scores. ## results we also tried incorporating all types of knowledge into a prompt and found that performance dropped below basic. however, combining knowledge types involves non-trivial decisions such as the order of knowledge types and its composition. we leave this to future work. using itemdesc, we establish the best possible gpt-3 performance for personality estimation. although gpt-3's average performance over all traits is still lower than wt-lex, it outperforms the mfc baseline. prior work @xcite has shown dimensions of mental health constructs and personality traits being captured through language use patterns in lms. gpt-3's performance in zero-shot setting provides reasonable evidence to believe that language patterns associated with these traits are encoded in its embedding space as well. ## analysis to better understand the utility of gpt-3 for personality estimation, we analyze the effect of (1) problem framing, and (2) effect of survey items. furthermore, we perform error analysis of gpt-3 to suggest avenues for improvement. problem framing. when personality estimation is framed as a binary classification, gpt-3 is worse than sota on average in a zero-shot setting. upon looking closer, we note that it is the best model for 2 out of the 5 traits. however, these observations are made in a simplified two-class setting, whereas the big 5 personality model produces a real valued outcome. in order to assess gpt-3's practical via-bility, we prompt it (itemdesc) to provide more fine-grained predictions by presenting trait estimation as a three-class classification problem. consistency with survey items. the standard questionnaire used to create the dataset had a total of 4 survey items per trait (2 positive and 2 negative). for itemdesc, we use one positive and one negative item to describe each trait (see figure 2 ). to investigate whether gpt-3 performance can be attributed to specific items in the prompt, we perform itemdesc with all possible combinations of a positive and a negative survey item for all traits. ## conclusion we performed a systematic investigation of gpt-3's zero-shot performance on personality estimation. while using a simple prompt did not yield strong performance, injecting knowledge about the traits themselves led to significant improvement. even so, it falls short of using a strong, extensivelytrained, supervised model (wt-lex). further, we find that it is much harder for gpt-3 to provide more fine-grained predictions (when asked to select between 3 labels instead of 2), suggesting that llms may not be as capable at making dimen-sional estimates about personality. our systematic investigation helps understand gpt-3's zero-shot capabilities for a human-level nlp task, contextualizing its failure modes and showing avenues for llm improvements.