PANDORA Talks: Personality and Demographics on R eddit (2021)

Abstract: Personality and demographics are important variables in social sciences and computational sociolinguistics. However, datasets with both personality and demographic labels are scarce. To address this, we present PANDORA, the first dataset of Reddit comments of 10k users partially labeled with three personality models and demographics (age, gender, and location), including 1.6k users labeled with the well-established Big 5 personality model. We showcase the usefulness of this dataset on three experiments, where we leverage the more readily available data from other personality models to predict the Big 5 traits, analyze gender classification biases arising from psycho-demographic variables, and carry out a confirmatory and exploratory analysis based on psychological theories. Finally, we present benchmark prediction models for all personality and demographic variables.

## introduction personality and demographics describe differences between people at the individual and group level. this makes them important for much of social sciences research, where they may be used as either target or control variables. one field that can greatly benefit from textual datasets with personality and demographic data is computational sociolinguistics @xcite , which uses nlp methods to study language use in society. conversely, personality and demographic data can be useful in the development of nlp systems. recent advances in machine learning have brought significant improvements in nlp systems' performance across many tasks, but these typically come at the cost of more complex and less interpretable models, often susceptible to biases @xcite . biases are commonly caused by societal biases present in data, and eliminating them requires a thorough understanding of the data used to train the model. one way to do this is to consider demographic and personality variables, as language use and interpretation is affected by both. incorporating these variables into the design and analysis of nlp models can help interpret model's decisions, avoid societal biases, and control for confounders. the demographic variables of age, gender, and location have been widely used in computational sociolinguistics @xcite @xcite , while in nlp there is ample work on predicting these variables or using them in other nlp tasks. in contrast, advances in text-based personality research are lagging behind. this can be traced to the fact that (1) personality-labeled datasets are scarce and (2) personality labels are much harder to infer from text than demographic variables such as age and gender. in addition, the few existing datasets have serious limitations: a small number of authors or comments, comments of limited length, non-anonymity, or topic bias. while most of these limitations have been addressed by the recently published mbti9k reddit dataset @xcite , this dataset still has two deficiencies. firstly, it uses the myers-briggs type indicator (mbti) model @xcite , which -while popular among the general public and in business -is discredited by most personality psychologists @xcite . the alternative is the well-known five factor model (or big 5) @xcite , which, however, is less popular, and thus labels for it are harder to obtain. another deficiency of mbti9k is the lack of demographics, limiting model interpretability and use in sociolinguistics. our work seeks to address these problems by introducing a new dataset -personality and demographics of reddit authors (pandora) -the first dataset from reddit labeled with personality and demographic data. pandora comprises over 17m comments written by more than 10k reddit users, labeled with big 5 and/or two other person-ality models (mbti, enneagram), alongside age, gender, location, and language. in particular, big 5 labels are available for more than 1.6k users, who jointly produced more than 3m comments. pandora provides exciting opportunities for sociolinguistic research and development of nlp models. in this paper we showcase its usefulness through three experiments. in the first, inspired by work on domain adaptation and multitask learning, we show how the mbti and enneagram labels can be used to predict the labels from the wellestablished big 5 model. we leverage the fact that more data is available for mbti and enneagram, and exploit the correlations between the traits of the different models and their manifestations in text. in the second experiment we demonstrate how the complete psycho-demographic profile can help in pinpointing biases in gender classification. we show that a gender classifier trained on a large reddit dataset fails to predict gender for users with certain combinations of personality traits more often than for other users. finally, the third experiment showcases the usefulness of pandora in social sciences: building on existing theories from psychology, we perform a confirmatory and exploratory analysis between propensity for philosophy and certain psycho-demographic variables. we also report on baselines for personality and demographics prediction on pandora. we treat big 5 and other personality and demographics variables as targets for supervised machine learning, and evaluate a number of benchmark models with different feature sets. we make pandora available 1 for the research community, in the hope this will stimulate further research. ## pandora dataset reddit is one of the most popular websites worldwide. its users, redditors, spend most of their online time on site and have more page views than users of other websites. this, along with the fact that users are anonymous and that the website is organized in more than a million different topics (subreddits), makes reddit suitable for various kinds of sociolinguistic studies. to compile their mbti9k reddit dataset, @xcite used the pushift reddit dataset @xcite to retrieve the comments dating back to 2015. we adopt mbti9k as the starting 3 http://www.kaggle.com/datasnaek/mbti-type point for pandora. ## experiments coupling linguistic data with psycho-demographic profiles sets the stage for many interesting research questions. we showcase this with three experiments on pandora. ## prediction models in this section we describe baseline models for predicting personality and demographic variables from user comments in pandora. we consider the following sets of features: (1) n-grams: tf-idf weighted 1-3 word ngrams and 2-5 character n-grams; (2) stylistic: the counts of words, characters, and syllables, mono/polysyllable words, long words, unique words, as well as all readability metrics implemented in textacy foot_7 ; (3) dictionaries: words mapped to tf-idf categories from liwc @xcite , empath @xcite , and nrc emotion lexicon (mohammad and turney, 2013) dictionaries; (4) gender: predictions of the gender classifier from §4.2; (5) subreddit distributions: a matrix where each row is a distribution of post counts across all subreddits for a particular user, reduced using pca to 50 features per user; (6) subreddit other: counts of downs, score, gilded, ups, as well as the controversiality scores for a comment; (7) named entities: the number of named entities per comment, as extracted using spacy; foot_8 (8) part-of-speech: counts for each part-of-speech; ( 9 ) predictions (only for predicting big 5 traits): mbti/enneagram predictions obtained by a classifier built on held-out data. features (2), (4), and (6-9) are calculated at the level of individual comments and aggregated to min, max, mean, standard deviation, and median values for each user. we build six regression models (age and big 5 traits) and eight classification models (four mbti dimensions, gender, region, enneagram). we experiment with linear/logistic regression (lr) from sklearn @xcite and deep learning models (nn). we trained a separate nn model for each task. in each model, a single user is represented as a matrix, with rows representing the user's comments. the comments were encoded using 1024-dimensional vectors derived using bert @xcite . bert comment vectors are fed into convolution layers, max pooling, and several fully connected layers. hyperparameters and additional information can be found in the appendix. we evaluate the models using 5-fold crossvalidation with a separate stratified split for each target. we use regression f-tests to select top-k features, and optimize model hyperparameters and k on held-out data for each fold separately. results are shown in ## conclusion pandora dataset comprises 17m comments, personality, and demographic labels for over 10k reddit users, including 1.6k users with big 5 labels. to our knowledge, this is the first reddit dataset with big 5 traits, and also the first covering multiple personality models (big 5, mbti, enneagram). we showcased the usefulness of pandora with three experiments, showing (1) how more readily available mbti/enneagram labels can be used to estimate big 5 traits, (2) that a gender classifier trained on reddit exhibits bias on users of certain personality traits, and (3) that certain psycho-demographic variables are good predictors of propensity for philosophy of reddit users. we also trained and evaluated benchmark prediction models for all psychodemographic variables. the poor performance of deep learning baseline models, the rich set of labels, and the large number of comments per user in pandora suggest that further efforts should be directed toward efficient user representations and more advanced deep learning architectures.