On Text-based Personality Computing: Challenges and Future Directions (2023)

Abstract: Text-based personality computing (TPC) has gained many research interests in NLP. In this paper, we describe 15 challenges that we consider deserving the attention of the NLP research community. These challenges are organized by the following topics: personality taxonomies, measurement quality, datasets, performance evaluation, modelling choices, as well as ethics and fairness. When addressing each challenge, not only do we combine perspectives from both NLP and social sciences, but also offer concrete suggestions. We hope to inspire more valid and reliable TPC research.

## introduction according to the apa dictionary of psychology (apa, 2022), personality refers to personality traits, which are "relatively stable, consistent, and enduring internal characteristics inferred from a pattern of behaviours, attitudes, feelings and habits in individuals". knowledge about personality can be useful in many societal and scientific applications. for instance, it can help individuals choose learning styles @xcite and occupations @xcite suited for their personality; it can help clinical psychologists to better understand psychological disorders @xcite and to deliver personalised treatment plans for mental health patients @xcite ; changes in personality can even help with early diagnosis of @xcite ) and parkinson's disease @xcite . traditionally, personality assessment is based on self-and other-report questionnaires, which is labour-and time-intensive. recently, however, automatic personality assessment based on usergenerated data (e.g., texts, images, videos) and machine learning algorithms has become a popular alternative. this is known as personality computing (pc) @xcite , among many other names 1 . in this paper, we focus on evaluating pc research in nlp, where personality is primarily inferred from text, such as tweets and reddit posts @xcite , conversations @xcite and speech transcriptions @xcite . we refer to such research as text-based personality computing (tpc). in tpc, on the one hand, we see an increasing number of datasets curated, complex deep-learning algorithms adopted, and (sometimes) high prediction scores achieved. on the other hand, we see relatively little discussion about open challenges and future research directions. for example, the presence of measurement error in questionnaire-based personality scores remains an un(der)addressed issue. given that such scores are often used as the gold standard for training and validating tpc algorithms, we find it important to discuss related implications and remedies. another relevant issue concerns how to reduce the risks of tpc research. therefore, in this paper, we reflect on current tpc research practices, identify open challenges, and suggest better ways forward. to spot such challenges, we conduct a literature search in the scope of the acl anthology 2 . while there are tpc papers published in other venues, we consider our selection from acl anthology a good representation of tpc research in nlp. appendix a describes our search strategy and results in detail. in total, we find and review 60 empirical tpc papers, based on which we identify 15 challenges. they are organized by the following topics: personality taxonomies ( §3), measurement quality ( §4), datasets ( §5), performance evaluation ( §6), modelling choices ( §7), as well as ethics and fairness ( §8). we discuss each challenge and give concrete suggestions where we also draw on broader nlp and social science literature. furthermore, during our literature review, we identify 18 tpc datasets that can be (re)used by other researchers. we summarize them in appendix b. note that our paper focuses on identifying and discussing tpc-related research challenges instead of providing a comprehensive overview of past tpc studies. for the latter, we refer you to the survey papers by @xcite ; @xcite ; @xcite . ## personality taxonomies a tpc research project typically starts by choosing a personality taxonomy, which is a descriptive framework for personality traits @xcite . among the 60 papers we review, we find two prominent taxonomies: the myers-briggs type indicator (mbti; 14 papers) and the big-5 (45 papers). 3 fifty of these papers adopt either the mbti or the big-5 but not both. this invites the first challenge: challenge 1 (c1): mbti vs. big-5 the mbti originated from the theoretical work of @xcite and was further developed by @xcite . it proposes four personality dimensions that characterize people's differences in perception and judgement processes: extraversion/introversion (e/i), sensing/intuition (s/n), thinking/feeling (t/f), and judgement/perception (j/p). individuals are classified into one of the two possible categories across each dimension (e.g., infp and estj). in contrast, the big-5 was developed based on the lexical hypothesis: all important personality traits must have been encoded in natural language and therefore, analysis of personality-related terms should reveal the true personality taxonomy @xcite . independent research groups @xcite @xcite investigated this hypothesis. they identified numerous english terms that might describe inter-individual differences (e.g., warm, curious), asked participants to rate how well these terms describe them on numerical scales, and factor-analyzed the responses, which revealed five consistent dimensions of personality: openness (o), conscientiousness (c), extraversion (e), agreeableness (a), and neuroticism (n). 4 furthermore, each dimension includes six finer-grained sub-traits called facets (e.g., agreeableness includes trust, straightforwardness, altruism, compliance, modesty and tender-mindedness). big-5 is the most widely accepted and researched taxonomy of personality traits in psychology (as opposed to the popularity of mbti in non-academic settings like job interviews) @xcite . we recommend the big-5 over mbti for the following reasons: first, the big-5 is a more realistic and accurate personality taxonomy. it scores individuals along a continuous spectrum, which describes interindividual differences more accurately and preserves more information (as opposed to mbti's dichotomous approach). also, the big-5 includes facets, which allows for finer-grained analysis of personality. facets are also more predictive of life outcomes (compared to dimensions; @xcite . a potential new tpc research direction can be to predict facets in addition to dimensions. second, big-5 has a much stronger empirical basis than mbti. namely, it is grounded in largescale quantitative analysis of natural language and survey data. also, big-5 questionnaires have undergone much more extensive development and validation processes than mbti's. consequently, many validated big-5 questionnaires exist, which vary in length (15-240 items), the inclusion of facet measures, as well as the target populations (e.g., nationalities, professions, languages, and age groups). 5 this enables researchers to choose a questionnaire most appropriate given a population of interest, available resources (e.g., can you afford a longer questionnaire?) and research interests (e.g., are you interested in facets?). in comparison, mbti is purely theory-driven, lacks empirical support and has officially only four questionnaires that have not been thoroughly tested @xcite @xcite . 6 therefore, compared to mbti, big-5 is a much more credible and flexible choice for research purposes. third, the big-5 is rooted in natural language (i.e., the lexical hypothesis), suggesting that big-5related cues may be more present than do mbtirelated cues in text data. this conjecture is supported by @xcite , who find either insufficient or mixed signals for mbti dimensions in tweets and short essays. nevertheless, despite the many advantages of big-5 over mbti, we acknowledge that studying mbti can still be useful given its popularity in nonacademic settings @xcite . furthermore, it is also important to mention that big-5 is not without criticisms and other personality taxonomies may be preferred, which invites the next challenge. ## measurement quality personality traits are latent, theoretical variables (a.k.a. social science constructs), which can not be directly or objectively observed. examples are emotions, prejudice and political orientation. thus, 1989), the revised neo-pi and the @xcite , the big-5 inventory (bfi) @xcite , bfi-2 (soto and @xcite , the short 15-item big-5 inventory @xcite . the presence of error in personality measurements can have negative consequences for tpc research. for instance, consider the case of having substantial measurement error in questionnairebased personality scores. when a tpc model treats these measurements as the gold standard for training and validation, the measurement error will likely propagate to the predictions, rendering the model less helpful (or even harmful) especially for diagnostic or clinical purposes. the study by @xcite lends support to this hypothesis, where the authors find tpc models to perform better on small datasets with low measurement error than on large datasets with high measurement error. therefore, it is important that tpc researchers are aware of the presence and influence of measurement error and can deal with it. unfortunately, none of the 60 tpc studies that we survey touch upon this issue, suggesting that this is an underexplored issue in tpc research. this observation inspires the next four challenges (3-6). ## datasets across the 60 tpc studies, we find 41 unique datasets, which vary in terms of the personality taxonomy, instrument, type of text data, sample size, sample characteristics etc. among them, however, only 18 are potentially accessible to other researchers (see appendix b). shareable datasets are key to advancing tpc research, as it leads to accumulation of data and allows for replication studies. this invites the next challenge: ## performance evaluation across the 60 surveyed tpc papers, we identify two challenges related to performance evaluation: c10: use more appropriate and consistent performance metrics 11 out of the 60 studies model tpc as a regression task. among them, 9 use pearson's correlation between predicted personality trait scores and the true scores as the performance metric. however, correlation-based metrics can be misleading, as they register only ranks and do not reflect how accurate the predictions are on the original scale of the personality scores @xcite . some studies report mean squared error (mse), which is arguably better than correlations because it quantifies the absolute difference between the predictions and the true values. however, mse scores depend on the scale of the personality measurements and are not bounded, making interpretation and comparison (between studies) difficult. @xcite propose a better performance metric: r 2 = 1 -rss/t ss, where rss refers to the sum of squares of residuals and t ss the total sum of squares. r 2 has several benefits. first, it can be considered a normalised version of mse, which has an upper limit of 1 (perfect agreement). second, r 2 has a natural zero point, which occurs when the mean is used as the prediction. third, when the model makes worse predictions than a simple mean baseline, r 2 becomes negative. while two studies report r 2 , their calculation of r 2 is unclear. the researchers may have calculated r 2 not based on the formula shown earlier, but by squaring pearson's correlations. this would lead to always positive r 2 , which can be misleading. in the 33 studies that model tpc as a classification task, a more diverse set of metrics are used (i.e., accuracy, recall, precision, f1, and auc, in macro, micro, weighted or unweighted forms). one problem, however, is that different studies report different metrics (sometimes, only one). this makes comparison across studies difficult. we encourage future researchers to report all common metrics for classification studies (like those mentioned above). ## modelling choices most tpc studies model different personality traits (i.e., dimensions) separately. this strategy is unde-sirable, because it ignores the correlations among personality traits that models can learn from. modelling personality traits jointly may also help to prevent overfitting to a specific trait and thus learn more universal personality representations @xcite . hence, the next challenge: ## ethics and fairness out of the 60 reviewed papers and the 2 additional survey papers, only 7 provide some reflection about ethics and none about fairness. this can be because ethics and fairness only became central in nlp recently. nevertheless, the last two challenges: c14: more ethical and useful tpc as useful as tpc can be, it is important to ask whether gathering personal information like personality or computing them is necessary. this is especially relevant for research where tpc is only an intermediate step to another end such as opinion mining @xcite , dialogue generation @xcite and brand preference prediction @xcite . such studies typically argue that the computed personality traits can be used as features for another task and that it leads to better task performance; however, they do not consider alternatives (e.g., replacing the prediction of personality traits with using lexical cues that are non-personal but still indicative of personality). thus, we encourage researchers to justify pc and to find alternatives when pc is only a means. even when tpc can be justified, it is important to reduce potential harm. for instance, many tpc studies and datasets make use of public social media profiles for predicting personality traits. while this is often legal, no explicit consent for pc is obtained from the social media users, which makes using public social media data an ethically ambiguous issue @xcite . @xcite proposes a privacy-preserving data donation framework that may help to alleviate this problem, where data subjects can voluntarily donate their data download packages (e.g., from social media accounts) for research and give explicit consent. to further increase the benefit of tpc, we can consider applying it to clinical, professional or educational settings, where (traditional) personality assessment has proven useful and relevant (e.g., personalised treatments; career recommendation; individualised learning). none of the 60 tpc studies in our review investigates these applications. ## conclusion in this paper, we review 60 tpc papers from the acl anthology and identify 15 challenges that we consider deserving the attention of the research community. we focus on the following 6 topics: personality taxonomies, measurement quality, datasets, performance evaluation, modelling choices, as well as ethics and fairness. while some of these topics (e.g., personality taxonomies and ethics) have been discussed elsewhere, we provide new perspectives. furthermore, in light of these challenges, we offer concrete recommendations for future tpc research, which we summarise below: we hope that our paper will inspire better tpc research and new research directions.