Personality Profiling: How informative are social media profiles in predicting personal information? (2024)

Abstract: Personality profiling has been utilised by companies for targeted advertising, political campaigns and public health campaigns. However, the accuracy and versatility of such models remains relatively unknown. Here we explore the extent to which peoples’ online digital footprints can be used to profile their Myers- Briggs personality type. We analyse and compare four models: logistic regression, naive Bayes, support vector machines (SVMs) and random forests. We discover that a SVM model achieves the best accuracy of 20.95% for predicting a complete personality type. However, logistic regression models perform only marginally worse and are significantly faster to train and perform predictions. Moreover, we develop a statistical framework for assessing the importance of different sets of features in our models. We discover some features to be more informative than others in the Intuitive/Sensory (p = 0.032) and Thinking/Feeling (p = 0.019) models. Many labelled datasets present substantial class imbalances of personal characteristics on social media, including our own. We therefore highlight the need for attentive consideration when reporting model performance on such datasets and compare a number of methods to fix class-imbalance problems.

## introduction in 2023 there are over 4.59 billion social media users worldwide, constituting approximately 60% of the world's population @xcite . this enables most of the world to be connected, creating an online information environment. the huge amounts of individual-level data provided by each user is an important aspect of social media which is unique to this type of information environment. consequently, it is crucial for scholars to understand how this aspect of social media may impact society. there exists a need to quantify the extent to which social media can be weaponized by governments and other organisations for influence. every time a user enters a social media application, they leave a unique data trace -information they have posted, liked, shared, commented, even how long they have spent viewing different material on the application. we refer to this unique trace of data as a user's online digital footprint. it has been suggested that someone's online digital footprint can expose actionable information about them, including their personality profile, relationship status, political opinions and even their propensity to adopt a particular opinion or behavior @xcite @xcite @xcite . cambridge analytica was suggested to use online digital footprints to impact the result of the 2016 us election and the 2016 brexit referendum @xcite . however, the extent to which companies like cambridge analytica can determine this information from social media data is still questioned @xcite @xcite . as a result, it is of interest for individuals to understand the extent of information that is attainable from their online digital footprint. this is also of key concern for governments, who seek to maintain democracies and the ethical use of such data. we seek to determine how informative online digital footprints are in predicting myers-briggs personality types. this is a theoretical model comprised of four traits/dichotomies, based on jungian theory @xcite . modelling personal information about individuals using their online information has previously enabled researchers to understand the accuracy of such models. we extend this work by creating a new labelled dataset of myers-briggs personality types on twitter and a statistical modelling framework which can be generally applied to any labelled characteristic of online accounts. we aim to reconsider the personality profiling and political microtargetting performed by companies like cambridge analytica. first we collect a labelled dataset of accounts with self-reported myers-briggs personality types. we then collect a number of different features for these accounts including social metadata features and linguistic features: liwc @xcite ; vader @xcite ; bert @xcite ; and botometer @xcite . we then create independent logistic regression (lr), naive bayes (nb), support vector machines (svms) and random forests (rf) models on each dichotomy to model the myers-briggs personality type of the accounts. as part of this, we consider four different weighting/sampling techniques to adjust for class imbalances. lastly, we provide a statistical framework for analysing the importance of different features in these models. we consider the importance of features at an individual level and across groups of features for each dichotomy. our main contributions are: (i) a labelled dataset foot_0 of 68,958 twitter users along with their myers-briggs personality types, the largest available dataset (to our knowledge) of labelled myers-briggs personality types on twitter @xcite ; (ii) a statistical framework to combine nlp tools and mathematical models to predict online users' personality types, which can be more broadly used to model any labelled characteristics about online accounts; (iii) a comparison of machine learning models on nlp features, and a comparison of various weighting/sampling techniques to address problems with class imbalance; (iv) statistical methods which compare the importance of different features in nlp-based models at an individual level and across groups of features. ## background myers-briggs @xcite is the most well-known personality model, being applied in hiring processes, social dynamics, education and relationships @xcite @xcite . the myers-briggs type indicator (mbti) handbook illustrates a four factor model of personality where people form their 'personality type' by attaining one attribute from each of four dichotomies; extrovert/introvert, intuitive/sensory, thinking/feeling and judging/perceiving. this gives 16 different personality types where a letter from each dichotomy is taken to produce a four letter acronym, e.g., 'entj' or 'isfp'. the model has received substantial scrutiny, particularly from psychologists who question its validity and reliability @xcite . nonetheless, we utilise the myers-briggs model in our analysis for the following reasons: (i) thousands of twitter users self-report their mbti on twitter. this enables us to obtain a labelled dataset through appropriately querying for each of the 4 letter personality type acronyms that are unique to mbti. (ii) the myers-briggs model has the largest number of self-reports on twitter, enabling us to achieve the largest labelled personality dataset on twitter. (iii) we aim to develop a framework for modelling personality profiles from social media data using statistical machine learning (ml) approaches. mtbi is a test case for our framework, which can be applied to other personality models (or other labelings/characteristics of individuals on social media) more generally. open-source labelled training data with myers-briggs personality types has not existed until recently. plank and hovy @xcite modeled the mbti of twitter users through attaining a small dataset of 1,500 users and gjurković and šnajder @xcite modeled the mbti on a larger corpus of reddit users. in 2017, jolly @xcite posted a labelled mbti dataset on kaggle, constituting the only known publicly available labelled dataset used for modelling the mbti of social media users. the dataset was comprised of 8,675 users, their personality types and a section of their last 50 posts on an online forum called personalitycafe.com. this small online forum contains 153,000 members dedicated to discussing health, behavior, personality types and personality testing. the discussions are therefore quite different to those on other social media platforms, and likely a different demographic. hence, this dataset is likely not generalisable to other platforms like twitter and facebook. it is also relatively small and imbalanced, limiting which models can be utilised on various feature sets. class imbalance is considerable in all cases, and in one particular dataset some classes are up to 28 times larger than their counterpart. nevertheless, many papers apply machine learning models to such datasets without accounting for these class imbalances @xcite @xcite @xcite . consequently, the metrics reported often misrepresent model performance, and instead highlight the severity of class imbalances in the datasets. ## data collection & preprocessing we discovered a number of twitter accounts selfreport their mbti on twitter as a regular expression. we therefore formulated two methods for querying and labelling the myers-briggs person-ality type of accounts. let ω define the set of 16 acronyms for myers-briggs personality types. m1 query: {x : @xmath0. we obtained the set of users who currently self-report their personality type in their username or biography. m2 query: {(i am x) ∨ (i am a x) ∨ (i am an x) : @xmath1. we obtained the set of users who have self-reported their personality type in a tweet since twitter's creation (march @xcite . note that we only searched for self-reports in tweets, not retweets, quotes and replies -due to a number of users often not self-reporting their own mbti when referencing mbti acronyms in these forms of communication. queries were not case-sensitive. the resulting labelled dataset comprised of 68,958 users; the dataset and more details on its collection are provided in @xcite . we collected 15,986 accounts by querying usernames and biographies, and 52,972 accounts from querying tweets, with misclassification rates 1.9% and 3.4% based on random samples of 1,000 accounts from each. next we obtained account characteristics for each user, including their biography, most recent 100 tweets/quotes, as well as a set of social metadata (sm) features. the user's biography and the 100 tweets/quotes were used to generate a set of linguistic features, whereas sm features ( we removed duplicate users, then combined the biography and tweets into a combined text for every account. we then: 1. normalised the text and calculated each account's dominant language. 2. removed non-english language using the compact language detect 2 (pycld2) library. 3. calculated (language-dependent) botometer scores foot_1 . 4. converted text to lowercase, removed urls, email addresses, punctuation and numbers. 5. tokenized using the tweet tokenizer from the natural language toolkit (nltk) @xcite . 6. removed empty tokens and any instances of the 16 mbti acronyms. next, we formulated an inclusion-exclusion criteria to determine whether a personality could be profiled from a twitter account -we kept accounts with over 100 tweets/quotes, over 50% english language, botometer cap score less than 0.8, and strictly one mbti type referenced. we use the botometer cap score because we are interested in the overall bot likelihood and not the sub-category bot likelihoods. unfortunately, there is no consistency in the literature on thresholds for binary bot classification. rather, authors define their threshold based on a false positive rate in the context of their problem. for instance, wojcik et al. @xcite use a threshold of 0.43 for their political analysis of the twittersphere, whereas keller and klinger @xcite use a larger threshold of 0.76 for their analysis of social bots in election campaigns. to avoid large numbers of false positive bot classifications, we chose a high threshold of 0.8. finally, we extracted the liwc, bert and vader features from the text. the data cleaning techniques above were performed only for liwc feature extraction, whereas the bert and vader features can be extracted directly from the raw text output. thus, we calculated the liwc features on the combined text by micro-averaging the tokens present in each liwc category for every user. next, we calculated the bert features on the raw twitter output using bertweet @xcite , a pre-trained language model for english tweets. first, we averaged the embeddings for the tokens to form a single embedding vector for each tweet/quote, then averaged the embedding vectors for the tweets/quotes to create a single 768-dimensional embedding vector for each user. we calculated the vader features (sentiment, proportion of positive words and proportion of negative words) on the raw twitter output for each user and include scores for both a user's biography and their tweets. we distinguish these because of contextual differences in the language; biographies often discuss oneself and tweets often discuss one's environment. we then have a total of 866 features; these are provided in ## exploratory data analysis we performed an exploratory data analysis (eda) on the dataset to determine important information about our dataset, prior to any modelling. we acknowledge and discuss two forms of potential bias in our dataset: (i) only considering mbti types on twitter; (ii) only selecting accounts which satisfy our inclusion-exclusion criteria as well as self-report their mbti types on twitter. figure 1 demonstrates these biases through bar plots showcasing the proportions of the mbti dichotomies in our dataset. we compare with a study reporting mbti proportions on twitter @xcite , and with category features sm followers_count, friends_count, listed_count, favourites_count, geo_enabled, verified, statuses_count, default_profile, default_profile_image, profile_use_background_image, has_extended_profile botometer cap_english, english_astroturf, english_fake_follower, english_financial, english_other, english_self_declared, english_spammer liwc function, pronoun, ppron, i, we, you, shehe, they, ipron, article, prep, auxverb, adverb, conj, negate, verb, adj, compare, interrog, number, quant, affect, posemo, negemo, anx, anger, sad, social, family, friend, female, male, cogproc, insight, cause, discrep, tentat, certain, differ, percept, see, hear, feel, bio, body, health, sexual, ingest, drives, affiliation, achiev, power, reward, risk, focuspast, focuspresent, focusfuture, relativ, motion, space, time, work, leisure, home, money, relig, death, informal, swear, netspeak, assent, nonflu, filler, total_word_count bert {ei ; @xmath2, . . . , 768} vader tweets_sentiment, bio_sentiment, tweets_pos_words, bio_pos_words, tweets_neg_words, bio_neg_words the proportion of personality types in the general population @xcite . a noticeable imbalance in the intuitive/sensory dichotomy exists across all datasets in figure 1 . there are also observable imbalances in the extrovert/introvert and thinking/feeling dichotomies, whereas the judging/perceiving dichotomy is more balanced across each dataset than the other dichotomies. the imbalances in our dataset are mostly consistent with those from www.personalitycafe.com . the higher proportion of introverts in our dataset is consistent with @xcite who find that introverts tend to use social media as a primary form of communication, whereas extroverts tend to prefer communicating in-person. the larger proportion of intuitives in our dataset is consistent with schaubhut et al. @xcite who discovered that more intuitive individuals (13%) reported being active users of twitter than individuals with a preference for sensing (8%). the imbalance in the thinking/feeling dichotomy in our dataset is opposite to what we observe in the twitter dataset. however, schaubhut et al. @xcite found that people displaying the feeling trait are more likely to spend their personal time browsing, interacting and sharing information on facebook. provided the same is true for twitter users, our inclusion-exclusion condition requiring users to be active on twitter (i.e. tweet/quote at least 100 times) may bias our dataset leading to more users exerting the feelings trait. we performed a principal component analysis (pca) on the features to discover if we could significantly reduce the dimension of the feature space, and multicollinearity between the features. the first principal component explains 25.1% of the variance in the data and the first 200 principal components explain 95.4% of the variance in the data. as a result, we utilise the first 200 pca components in our machine learning models, significantly reducing both the dimension of the feature space and the multicollinearity of the features. ## model comparison we train lr, nb, svm and rf classifiers on each of the four dichotomies in our dataset, using 10fold cross validation. the class imbalances we observe for some dichotomies (particularly intuitive/sensory and extrovert/introvert), leads us to perform four different weighting/sampling tech-niques on the training data prior to model fitting: (i) weight the importance of classifying dichotomies, (ii) upsample the minority class (with replacement), (iii) perform the synthetic minority oversampling technique (smote) on the minority class, (iv) downsample the majority class. interestingly, the standard lr model most accurately predicts at least three of four user dichotomies and is only marginally worse than svm for all other metrics. the lr model is also significantly faster to train than the svms -making it the model of choice on larger datasets. the auc is important in discussions of model performance, especially for unbalanced datasets. this is because it equally weights the tpr and fpr, making it more robust for unbalanced datasets compared to accuracy. most of our aucs lie around 0.65, apart from the nb classifiers. in particular, the best performance for the macro-averaged and micro-averaged aucs is the standard svm and smote lr model, respectively. these aucs are significantly larger than for both the random and majority class classifiers, indicating a clear 'signal' in our features. we therefore perform an in-depth analysis of feature importance next. ## feature importance we perform independent upsampled lr models on each of the four mbti dichotomies because they performed well on our dataset (macro-and microaveraged aucs: 0.6676 and 0.6536). we choose an lr model because it is fast to train, and straightforward to interpret and perform feature selection on. moreover, we use an upsampled model because it does not involve creating 'synthetic' data in the same way that smote does -this is important for determining feature importance. we consider the variable importance of the descriptive features in our models; these include all features except from bert. for each dichotomy we fit the upsampled lr model and perform a stepwise feature selection to obtain a model with only significant features. in each case, we start with a null model and perform the stepwise selection algorithm on the p-values with a threshold in of 0.05 and a threshold out of 0.1. we determine the variable importance of features using the t-statistic for the parameter coefficients associated with each feature. for each dichotomy, we calculate the variable importance of each remaining feature after stepwise selection is complete, and display the absolute value of variable importance. figure 3 displays the 12 most important features for each model. we colour the bars based on the variable's preference for each class in the dichotomy. extroverts tend to be associated with more positive language, and introverts have more focus on the past. similarly, chen et al. @xcite suggested that extroverts display more positive emotion because they have a "dispositional tendency to experience positive emotions". accounts with larger favourites count (i.e. the account likes more tweets) tend to be more intuitive, whereas accounts which write more statuses tend to be more sensory. interpreting favourites as a proxy for the amount of information an account consumes, our results suggest that intuitives consume more information on twitter, whereas sensory individuals write more. this proxy is of course not perfect, because people may consume information without liking it. nonetheless, it is consistent with myers-briggs foundation definitions, which state that intuitives pay "most attention to impressions or the meaning and patterns of the information", whereas sensors pay "attention to physical reality, what i see, hear, touch, taste, and smell" @xcite . the strongest predictor for the j/p dichotomy (figure 3d ) is time; judgers are more likely to use words related to time and certainty compared to perceivers. 'end', 'until' and 'season' are examples of time-related words and 'always', 'never' are words related to certainty. this is consistent with the myers-briggs foundation, which states judgers "prefer a planned or orderly way of life, like to have things settled and organized" @xcite . next we explore how emoji usage relates to a twitter user's mbti. on twitter, emojis often have multiple meanings. for instance, the rainbow flag can indicate support for lgbtq+ social movements, the wave can symbolise a "resister" crowd of anti-trump twitter, and the okay symbol can be used by white supremacists, some of which covertly use the symbol to indicate their support for white nationalism @xcite . hence, emojis can indicate how these groups/movements interact with different personality types. we determine each emoji's frequency in a user's tweets and include these frequencies as predictors in upsampled lr models. performing the same stepwise feature selection al-gorithm as above, we display the 12 most important predictors from the remaining models in figure 4 . the rocket ship emoji is one the top 12 most important predictors across all models. an increase in this emoji's usage implies a higher likelihood of an account being introverted, intuitive, feelingsorientated and perceiving. the rocket ship emoji has been used by finance enthusiasts who use the emoji to denote a fast increase in a particular stock or cryptocurrency. hence, it is possible that we are observing crypto enthusiasts to be more introverted, intuitive, feelings-orientated and perceiving. however, this emoji has other meanings like as an actual rocket ship, so we explore created word clouds of tweets containing the rocket emoji (figure 5a ), as well as the red heart emoji (figure 5b ). the rocket ship generally appears in crypto-related tweets discussing 'projects', 'great opportunities', 'developments' and 'cryptos'. however, it also appears in tweets discussing the 'moon' and 'space'. the red heart emoji mainly appears in emotive tweets discussing 'love' and 'happiness'. a number of the emojis making an account more introverted are sad/upset emojis, whereas no sad/upset emojis make an account more extroverted. this further confirms figure 3a which suggested that extroverts prefer to display positive emotion online. next we consider the importance of different feature groups (including the bert features) and discuss whether different groups of features are more informative in our models. again, we fit an upsampled lr model to all features and perform stepwise feature selection on each model. we use the same thresholds to accept and remove features. we then measure the feature group importance using the number of remaining features in each feature group after selection. for each model, the number of features selected depends on the type of model. for instance, 243 features are selected in the n/s model, whereas only 124 features are selected in the j/p model. interestingly, the n/s model is also the most accurate and the j/p model the least accurate, implying a positive relationship between accuracy and number of features retained. this is consistent with the remark that more features are retained in a model when they are more informative about the data. moreover, the sm features are on average the most-retained across models. conversely, the botometer features have worst payoff across the four models, having the smallest proportion retained on average. the most interesting comparison is between the liwc and bert features, which both aim to describe linguistic properties about users. in each model, the bert features are more highly retained. however, only the results from the n/s model and the t/f model are significant at the 5% level. we therefore reject the null hypothesis that each feature group is equally as informative (per feature) for the n/s and t/f models. however, the chi-squared test does not alone tell us what feature groups perform significantly better, so we perform individual confidence intervals (cis) for the binomial proportions of accepting/rejecting features in each group using the wilson score interval @xcite . the cis for each feature group and model are displayed in figure 6 . for the i/s model, the 95% ci for the sm features lies completely above those for liwc and bert. this indicates that sm features are more informative (per feature) than liwc and bert features at the 5% level for this dichotomy. attributes about a user's account are therefore sometimes more important than the language they use when modelling personality. this is also validated by the results for the t/f model, where the 95% ci for the sm features and vader features lie completely above the 95% ci for the bert features. we likely observe these results because the textual features are all fairly correlated with each other. moreover, there is no evidence to suggest that bert features are more informative than liwc features in determining a myers-briggs personality type. ## conclusion this paper contributes a labelled twitter dataset of personality types and framework to model the personality types of these users. to our knowledge, this is the largest available twitter dataset of labelled myers-briggs personality types. our data collection techniques avoid the long, cumbersome questionnaires used in other research. additionally, we develop a statistical framework which combines nlp and mathematical models to model/predict users' personality type. while we applied this framework to personality types, it can model any labelled characteristics of online accounts -political opinions, psychological properties or propensity to adopt an opinion. using this framework, we analyse and compare a number of different models. since personality types in our dataset are unbalanced, we compare different weighting/sampling techniques to deal with class imbalances. we discover that class imbalances are common in these types of datasets, yet are often overlooked. because of this, we demonstrate why personality prediction models appear more accurate than they are, and demonstrate why digital footprints may be less informative of personality type than models suggest.