On the Usefulness of Personality Traits in Opinion-oriented Tasks (2021)

Abstract: We use a deep bidirectional transformer to extract the Myers-Briggs personality type from user-generated data in a multi-label and multi-class classification setting. Our dataset is large and made up of three available personality datasets of various social media platforms including Reddit, Twitter, and Personality Cafe forum. We induce personality embeddings from our transformer-based model and investigate if they can be used for downstream text classification tasks. Experimental evidence shows that personality embeddings are effective in three classification tasks including authorship verification, stance, and hyperpartisan detection. We also provide novel and interpretable analysis for the third task: hyperpartisan news classification.

## introduction the vocabulary we use in everyday language is a rich source of information about our beliefs, thoughts, and personalities @xcite . many efforts in text analysis provide compelling evidence that our everyday language carries psychological cues @xcite @xcite . with this study, we seek to determine the personality of a given text's author as defined by the myers-briggs type indicators or @xcite . myers-briggs uses four binary dimensions to classify people (introvert-extrovert, intuitive-sensing, thinking-feeling, judging-perceiving), which gives 16 different types, such as intj and entj. this work uncovers novel insights into the personality space of authors from their online writings. the personality signal carries the fingerprint of the individual's psyche and, even though noisy, can be useful (as shown in this work) for a variety of * @gmail.com † @uh.edu downstream nlp tasks, such as authorship verification, stance, and hyperpartisan detection. personality prediction does not only benefit commercial applications and psychology but also is advantageous in health care. recent works link personality types and social media behavior with depression and posttraumatic stress disorder @xcite . this is significant because it opens new avenues for prevention care as certain personality types can anticipate mental illness and schizophrenia. @xcite . this problem poses a non-trivial challenge because a good solution must be capable of capturing the complexity and the depth of the human psyche as expressed through text. anything short of that will result in task-specific pattern-matching. it follows that the main technical difficulty presented by the task at hand is the discrepancy between the corpora concerning their distributions, which results in the domain shift. this is where our work becomes relevant as it aims to bridge the gap by transfer learning and universal language understanding. the problem is also challenging because the human psyche is complex in its nature. the labels are fuzzy as the label distribution changes from population to population and the ground truth is not derived by an objective method; rather, it is a set of ideas generally agreed upon by specialists and society. furthermore, high quality curated datasets constructed by professional psychologists are difficult to obtain due to privacy reasons. models have only recently reached the capacity required. our approach uses transfer learning through language understanding for personality prediction. we create a unified dataset from the collection of user inputs of three available mbti datasets @xcite @xcite originating from social media platforms including reddit, twitter, and personality cafe forum. we investigate how transfer learning with pretrained transformers con-tributes to personality prediction under a multilabel multi-class classification strategy. we analyze the relationship between personality types with the three specific tasks of stance, authorship, and hyperpartisan news classification. the results on the unified dataset show that transfer learning along with pretrained bidirectional transformer models effectively changes the hamming loss, f1, and jaccard similarity for multi-label personality prediction. the contributions of our paper are listed below: in the following sections we introduce and evaluate the personality model, then analyze its application in the three text classification tasks (mentioned above) using transfer learning. ## related work personality prediction from text is a challenging task @xcite and many personality prediction approaches rely on crafted features which can range from simple ones, such as tf-idf of word or character n-grams to the ones produced by tools such as linguistic inquiry and word count (liwc) @xcite , which extracts anything from low-level information such as part of speech tags and topical preferences to psychological categories. these features are often supported by various psycholinguistic word lists that aim to detect emotions and sensory experiences @xcite . our work uses a bidirectional transformer to predict mbti personality types using a large collection of data obtained from three existing personality datasets. utilization of the pretrained word embeddings @xcite in many deep learning models indicates that leveraging knowledge obtained from unsupervised learning boosts the performance. recently, language models pretrained on a large amount of raw text were shown to provide representations applicable to a wide variety of tasks with minimal finetuning @xcite @xcite . these models can be effectively generalized to many downstream tasks and adapted to different domains. below are the three representative studies on utilizing online usergenerated text for personality prediction. they are annotated with self-reported mbti personality types of users. reddit9k dataset is a large-scale dataset constructed from the posts and comments of 9k reddit users. it is labeled with mbti indicators and covers a wide variety of topics @xcite . the authors extract user activity and linguistic features including word and character n-grams, liwc word categories @xcite , and two psycholinguistic dictionaries @xcite . support vector machine (svm), logistic regression (lr), and multi-layer perceptron are used to identify personality types and prove to be discriminative for personality prediction. twitter dataset is a large corpus of 1.2m tweets of 1.5k users @xcite . experiments performed by the dataset creators show that linguistic features are reliable representatives for two out of four personality dimensions. we hypothesize that the cause of the discrepancy is the difference between the distribution of personality types in social media users and the general u.s. population. finally, kaggle dataset collects the user posts of the personality cafe foot_0 forum and covers 8.6k different people with 16 mbti personality types @xcite . ## dataset we use reddit9k, twitter, and kaggle myers-briggs personality type datasets to train and evaluate our proposed model for automatic personality type prediction. in all datasets, the annotation process relies on self-reported personality types, and no questionnaire is given to the users. previously, mypersonality created from facebook user data was a questionnaire-based dataset. however, it is not available to the public anymore. we make a unified dataset from the collection of the three available mbti personality datasets and remove the non-english contents. we find that the new dataset is highly skewed towards two out of four personality dimensions. there are a few reasons for that. i) according to @xcite the distribution of personality types among the united states population is not balanced. ii) users from some specific personality types tend to participate in social media platforms and express their personality types more than others. our experiments also show that the class imbalance highly affects training, generating poor results for small classes, among evaluation methods. to alleviate the skewness of the data in training we take two (standard) steps: add class weights concerning their size in loss computation and make a balanced subset of the original dataset. we notice that the former does not improve the performance significantly, but the latter does. hence, we create a balanced version of the dataset by over-sampling the small and undersampling the large classes such that their final sizes become equal to the original average size of the 16 mbti personality types before sampling. ## personality embedding we build a general model to predict four mbti personality dimensions and to infer personality embedding. the mbti dimensions are expressed as booleans (0/1). the personality dimensions are ie, or introversion (i)/extroversion (e); ns, or intuition (n)/sensing (s); ft, or feeling (f)/thinking (t); and jp, or judging (j)/perceiving (p). under this scheme, each instance can have multiple labels with four classes. the combination of these four classes gives 2 4 = 16 mbti personality types. we consider multi-labeling classification to learn the personality dimensions together. our experiments show that sub-sampling creates a small training set with poor final results while over-sampling creates a huge dataset with hundreds of redundant examples. so, the models cannot differentiate the 16 classes (personality types) with the under-sampled small training data or they fail to predict the unseen examples of minority classes correctly in the over-sampled dataset as they get overfitted by the redundant examples. finally, we find that by converting the 16 classes into four in the multi-labeling scheme and applying over-sampling and sub-sampling simultaneously we can better overcome the class imbalance. the following section describes the proposed personality prediction model. ## transfer learning we aim to study if the knowledge gained from our personality prediction model, persbert, helps opinion-oriented problems. personality is closely connected with opinions and how people form opinions; hence, we choose three tasks, i.e., hyperpartisan news detection, authorship verification, and stance detection @xcite , that are designed around opinion mining. we create our transfer learner named docbert + persbert by connecting the pretrained personality model, persbert, to docbert (figure 2 ). empirical results show that these transformer models achieve the best results among the baselines introduced in section 4.2. we share a fully connected layer, classifier layer, between the docbert and persbert models. the layer takes the concatenation of the news dataset train dev test by-article 386 129 130 by-publisher 164,944 118,510 149, 794 ## conclusion we believe that this work lays the foundation for leveraging personality signals in a variety of opinion-oriented tasks. we first proposed a novel model, persbert, that jointly models the sentencespecific sentiment and personality information building upon the bert architecture to predict the mbti personality dimensions. our pretrained personality transformer improves bert results and other baselines in benchmark datasets on the personality task. further, our proposed model was used on different downstream nlp tasks providing major improvements showing that the subtle signals of user sentiment and their connection with personalities captured by our model are useful in real-world nlp tasks. it is worthwhile to note that the performance comes from training using only short sequences of online user posts (i.e. noisy data for personality). we believe the improvements of our model can be more pronounced if trained upon large-scale gold standard personality datasets (e.g. curated using controlled experiments which is a potential future work). we find that personality signals are more distinctive in authorship verification and stance detection than hyperpartisan news detection where the data is sourced from formal and more supervised writings. however, our personality embeddings can still be used for an effective sub-sampling even in hyperpartisan news detection. our architecture allows for novel analysis and insights that were previously unknown and have the potential to improve various other nlp tasks which we defer for future exploration.