Pushing on Personality Detection from Verbal Behavior: A Transformer Meets Text Contours of Psycholinguistic Features (2022)

Abstract: Research at the intersection of personality psychology, computer science, and linguistics has recently focused increasingly on modeling and predicting personality from language use. We report two major improvements in predicting personality traits from text data: (1) to our knowledge, the most comprehensive set of theory-based psycholinguistic features and (2) hybrid models that integrate a pre-trained Transformer Language Model BERT and Bidirectional Long Short-Term Memory (BLSTM) networks trained on within-text distributions (‘text contours’) of psycholinguistic features. We experiment with BLSTM models (with and without Attention) and with two techniques for applying pre-trained language representations from the transformer model - ‘feature-based’ and ‘fine-tuning’. We evaluate the performance of the models we built on two benchmark datasets that target the two dominant theoretical models of personality: the Big Five Essay dataset (Pennebaker and King, 1999) and the MBTI Kaggle dataset (Li et al., 2018). Our results are encouraging as our models outperform existing work on the same datasets. More specifically, our models achieve improvement in classification accuracy by 2.9% on the Essay dataset and 8.28% on the Kaggle MBTI dataset. In addition, we perform ablation experiments to quantify the impact of different categories of psycholinguistic features in the respective personality prediction models.

## introduction personality is broadly defined as the combination of a person's behavior, emotions, motivation, and characteristics of thought patterns @xcite . our personality has a major impact on our lives, influencing our life choices, well-being, health, and preferences and desires @xcite . specifically, personality has been repeatedly linked to individual (e.g., happiness, physical and mental health), interpersonal (e.g., quality of relationships with peers, family, and romantic partners), and socialinstitutional outcomes (e.g., career choice, satisfaction and achievement, social engagement, political ideology) @xcite . while there are several models of human personality, the predominant and widely accepted model is the big five or five factor model @xcite . in this model, personality traits are divided into five factors: (1) extraversion (assertive, energetic, outgoing, etc.), (2) agreeableness (appreciative, generous, compassionate, etc.), (3) conscientiousness (efficient, organized, responsible, etc.), (4) neuroticism (anxious, self-pitying, worried, etc.), and (5) openness (curious, empathetic, imaginative, etc.). these five personality traits are commonly assessed by questionnaires in which a person reflects on his or her typical patterns of thinking and behavior, such as the @xcite , and the big-five inventory @xcite ; (see @xcite , for a comprehensive overview). the myers-briggs type indicator (mbti) is another widely administered questionnaire, in particular in applied settings @xcite . in contrast to the big five personality taxonomy, which conceptualizes human personality as latent trait scores, the mbti model describes personality in terms of 16 types that result from combining binary categories into four dimensions: (a) extraversion/introversion (e/i) -preference for how people direct and receive their energy, based on the external or internal world, (b) sensing/intuition (s/n) -preference for how people take in information, through the five senses or through interpretation and meanings, (c) thinking/feeling (t/f) -preference for how people make decisions, relying on logic or emotion over people and partic-ular circumstances, and (d) judgment/perception (j/p) -how people deal with the world, by ordering it or remaining open to new information. given its central importance in capturing the essential aspects of human life, increasing attention is being paid to the development of models that can leverage behavioral data to automatically predict personality. data obtained from verbal behavior is one of the key types of such data. even in the early years of psychology, a person's use of language was seen as a distillation of his or her underlying drives, emotions, and thought patterns (see @xcite , for historical overviews). early approaches to automatic personality prediction (app) -also referred to as automatic personality prediction or recognition -from textual data have relied on machine learning models based on psycholinguistic features, whereas more recent approaches to app typically draw on deep learning techniques that use pre-trained word embeddings (see vinciarelli and mohammadi, 2014, for an overview of the former) (see @xcite , for an overview of deep learning-based app). in this paper, we make a valuable contribution to this dynamic area of app research by presenting two important improvements in predicting personality traits from textual data: (1) to our knowledge, the most comprehensive set of psycholinguistic features and (2) hybrid models that integrate a pre-trained transformer language model bert and bidirectional long short-term memory (blstm) networks trained on in-text distributions ('text contours') of psycholinguistic features. since our goal is to demonstrate the utility of our modeling approach, we conduct our experiments on two widely used benchmark datasets: the big five essay dataset @xcite and the mbti-kaggle dataset @xcite , which align with the dominant personality models described above. the remainder of this paper is organized as follows: in section 2, we briefly review recent related work on these two benchmark datasets. then, in section 3, we present the two benchmark datasets and the extraction of psycholinguistic features using automated text analysis based on a sliding window approach. in section 4, we describe our modeling approach, and in section 5, we present and discuss the results. finally, we conclude with possible directions for future work in section 6. ## modeling approach our models are constructed from three components: (a) a 'contour encoder' that converts a sequence of psycholinguistic features into a hidden representation vector, (b) a pre-trained transformer-based language model, bert, that converts a sequence of tokens into a hidden representation vector, and (c) a classifier that outputs the probability of a personality feature given the hidden representation of the sample. we conduct experiments with three types of personality prediction models: (1) contour encoder + classifier, (2) hybrid models that combine the contour encoder with a transformerbased language model + classifier, and (3) a stacking model that combines ten repetitions of the best performing model. as for the contour encoder, we experiment with blstm and blstm with attention models. attention-based models have been successfully used in a variety of tasks, including machine translation @xcite , speech recognition @xcite and relation classification @xcite . in the context of personality classification, learning a scoring function gives sentence weighting to the attention mechanism and allows a model to pay more attention to the most influential sentences in a text for a personality trait. as for the hybrid models, we experiment with different strategies for applying the pre-trained language model -'feature-based' and 'fine-tuning': in the feature-based approach, we freeze model weights during training and use the pre-trained contextualized word embeddings from bert. in the 'fine-tuning' approach, we unfreeze all 12 layers and fine-tune towards the personality detection task (see @xcite . all models are implemented using pytorch @xcite . unless specifically stated otherwise, we use binary cross entropy as our loss function, 'adamw' as optimizer, a fixed learning rate of 8×10 -4 and @xmath0.1, @xmath1. the optimal network structures and values of hyperparameters were found by grid-search. the performance of the models is evaluated by 10-fold cross-validation (ten repetitions) to counter variability due to initialization of the weights. we report the results of the best performing models in comparison to the performance of the app systems presented in section 2 ## results and discussion an overview of the results of our models in comparison to those reported in the previous studies reviewed above is presented in comparing the accuracy for each personality trait for the models trained with an attention mechanism (denoted by 'attn') to the corresponding value for the models trained without this mechanism (denoted by 'blstm'), we find that accuracy on all dimensions except the mbti n/s improved when an attention mechanism was used. our results also show that approaches grounded in interpretable features can achieve competitive performance with transformer-based approaches critical industries such as healthcare, criminal justice, and news. this is due to the fact that human experts in a given application domain require both accurate and understandable models @xcite . in what follows, we present the results of the ablation experiments. feature group importance was quantified using sp-lime on the best performing model trained only on text contours of psycholinguistic features, the attn-psyling model. the results of the feature ablation experiment are presented in ## conclusion due to its central importance in capturing the essential aspects of human life, increasing attention is being paid to the modeling and predicting personality traits. in this work, we made valuable contributions to advance the state of the art in automatic prediction of personality traits from verbal behavior. we demonstrated that models trained with a comprehensive set of theory-based psycholinguistic features can compete with a transformer-based model when their within-text distribution is taken into account. moreover, we showed that hybrid models incorporating such features can improve the performance of pre-trained transformer language models, even when the latter is based on a larger model (bert-large). we also showed that different techniques for applying pre-trained language representations from the transformer model have an impact on model performance. our ablation experiments have yielded interesting insights into the interplay between theory-based psycholinguistic features and personality traits. here, we decided to focus on the two most widely used benchmark datasets. in our future work, we intend to conduct experiments with more recent, larger personality datasets such as pandora @xcite . since this dataset also includes metadata (gender, age, and location/region), it would be interesting to see how they contribute to modeling and predicting personality traits from language use. marco tulio ribeiro, sameer singh, and carlos guestrin. 2016. " why should i trust you?" explaining the predictions of any classifier. in proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 1135-1144. klaus r scherer. 2005. what are emotions? and how can they be measured? social science information, 44(4):695-729. christopher j soto. 2019. how replicable are links between personality traits and consequential life outcomes? the life outcomes of personality replication project. psychological science, 30(5):711-727. ryan a stevenson, joseph a mikels, and thomas w james. 2007. characterization of the affective norms for english words by discrete emotional categories. behavior research methods, 39(4):1020-1024. philip j stone, dexter c dunphy, and marshall s smith. 1966. the general inquirer: a computer approach to content analysis. yla r tausczik and james w pennebaker. 2010. the psychological meaning of words: liwc and computerized text analysis methods. journal of language and social psychology, 29(1):24-54. alessandro vinciarelli and gelareh mohammadi. 2014. a survey of personality computing. ieee transactions on affective computing, 5(3):273-291. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, et al. 2019. huggingface's transformers: state-ofthe-art natural language processing. arxiv preprint arxiv:1910.03771. david h wolpert. 1992. stacked generalization. neural networks, 5(2):241-259. peng zhou, wei shi, jun tian, zhenyu qi, bingchen li, hongwei hao, and bo xu. 2016. attention-based bidirectional long short-term memory networks for relation classification. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 207-212. 0 2500 5000 7500 extraversion judging sensing thinking personality trait count figure 3: distribution of labels in the kaggle mbti dataset