Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention (2020)

Abstract: Not all documents are equally important. Language processing is increasingly finding use as a supplement for questionnaires to assess psychological attributes of consenting individuals, but most approaches neglect to consider whether all documents of an individual are equally informative. In this paper, we present a novel model that uses message-level attention to learn the relative weight of users’ social media posts for assessing their five factor personality traits. We demonstrate that models with message-level attention outperform those with word-level attention, and ultimately yield state-of-the-art accuracies for all five traits by using both word and message attention in combination with past approaches (an average increase in Pearson r of 2.5%). In addition, examination of the high-signal posts identified by our model provides insight into the relationship between language and personality, helping to inform future work.

## introduction most language-based methods for human attribute prediction assume all documents generated by a person are equally informative. however, this is not necessarily true. figure 1 gives examples of high and low signal messages for predicting extraversion -one's tendency to be energized by social interaction. the high signal messages contain words relating to social interaction (hangin out, chillin), whereas the low signal messages, while still containing social-related words, have little clear relevance to extraversion. the former examples would ideally be weighted higher by a personality prediction model than the latter. this paper applies the idea of modeling document relevance to the task of personality prediction. inferring an individual's personality traits is a fundamental task in psychology (mccrae and costa jr, 1997; @xcite , with social scientific applications ranging from public health @xcite and marketing @xcite to personalized medicine @xcite , mental health care @xcite , and even providing useful information for downstream nlp tasks @xcite . recently, researchers from both nlp and psychology have turned toward more accurately assessing personality and other human attributes via language @xcite @xcite . the idea behind "language-based assessments" @xcite is that language use patterns can supplement and, in part, replace traditional and expensive questionnaire-based human assessments. here, we present a hierarchical neural sequence model over both the words and messages of the user and correspondingly applies attention to each level. the document-level attention learns the relative importance of each social media post for predicting personality. contributions. our main contributions include: ## model architecture our goal is to encode user messages into a representation that can be used to predict the personality of the user. we can use a two-step process to produce such a representation: first encode the sequences of words in each message to form message-level representations and then encode the message-level representations to form a user-level representation. social media users write hundreds or even thousands of messages; while the messages, and the words within them, contain valuable clues to their personality, not all of it is equally valuable. an ideal representation of user text, therefore, should pay particular attention to personality-revealing portions of a user's text. hierarchical attention is a natural fit for this problem. at the message level, a word-attention model can learn to emphasize personality related words in the message representation, while at the user-level, a message attention model can learn to emphasize personality-related messages in the overall user representation. we instantiate this idea using a hierarchical sequence architecture shown in figure 2 . given a set of n messages from a user u, the first step of the model is to produce an encoding for each message m i . each word w i j in message m i is fed through a gated recurrent unit (gru) @xcite to produce a hidden state: we then apply an attention mechanism over the sequence of hidden states [h i 1 , h i 2 , ..., h i l ]: as before, the hidden states are then passed through another message-level attention mechanism: as before, e message is a learned context vector for message-level attention. the representation for a user u is thus a weighted combination of the hidden states representing that person's messages. once the user representation has been produced, u is further passed through some fully-connected layers before being used for prediction at the final layer. in this way, important words and messages don't get lost to noise and are instead carried through to later portions of the model, where they can have a greater impact on the final prediction. our model is similar in structure and motivation to the hierarchical attention network proposed by @xcite . however, our work focuses on a different level of analysis: whereas @xcite encode @xmath0, our work seeks to encode @xmath1. this idea of applying attention at a document level when modeling user-level attributes is, to the best of our knowledge, entirely novel. we hypothesize that where attention is applied is crucial and that message-level attention is of particular importance for modeling personality. ## dataset we draw our data from consenting users of a facebook application @xcite , which allowed users to take various psychological assessments and voluntarily share their data with researchers. following the work of @xcite and @xcite , the current state of the art on this dataset, we filtered the users to those who shared their facebook status posts, wrote at least 1,000 words across those statuses, provided their age and gender, and were less than 65 years old. all users completed psychological measures, ranging from 20 to 100 items, that assessed their big five personality traits @xcite : conscientiousness, agreeableness, neuroticism, openness to experience, and extraversion. each of the five dimensions is represented by a normalized, continuous score representing the degree to which that trait is exhibited. we refer to these as personality scores. the big five personality traits are described more fully in section 4. overall, our dataset contains facebook statuses and personality scores for 68,687 users. to allow for direct comparisons, we use the same test set (@xmath2,943) as @xcite . each of these test users completed a longer 100-item questionnaire, ensuring higher-quality scores. we sample an additional 4,998 for use as a development set, and leave the remaining 61,746 for training. on average, users in our dataset are 23 years old and 63% are female. users had an average of 3,619 words and 165 messages, all posted to facebook between 2009 and 2011. ## big five personality traits discovery of the "big five" personality traits began nearly a century ago with some of the first datadriven, statistical latent variable modeling techniques @xcite . the goal in this decadeslong pursuit was not very different from that of producing latent vector embeddings of words: foot_0 to use latent factor analysis to reveal underlying, stable dimensional vectors that distinguish people. however, rather than finding latent semantic dimensions of words, the models (run by hand at first) focused on how individuals answered questions about themselves. for example, modern questions include: "how much do you agree with these statements? (1) i am the life of the party; (2) i have difficulty understanding abstract ideas; (3) i like order; (4) i worry about things" @xcite . the idea behind this data-driven approach was that if such latent dimensions could be found to be stable across time and differing populations, that suggests they are fundamental to what makes each of us different. such work continued for decades, documented across thousands of studies to eventually arrive at the acceptance of five such factors being fundamental and consistent across time and populations @xcite . those fundamental human factors, the target of our human language predictive task, are described below. the big five often goes by the acronym "ocean", standing for openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. high scores for openness to experience are correlated with philosophical and free thought, as well as an interest in the arts, music, and cinema @xcite . those who score low here may be more practical, realistic, or close-minded @xcite . individuals with high conscientiousness tend to be well organized and have a lot of self-discipline, which may be expressed through discussions of work or school-related responsibilities @xcite . those who score low on this dimension may appear impulsive, disorganized, or unreliable. those with high extraversion are likely to talk about friends, social situations, and interpersonal interaction. on the other hand, those with low extraversion may be more independent and may focus more on solo activities (e.g. watching television) @xcite . agreeableness is associated with being friendly and good-natured, while those who score low may be selfish or rude. swearing is highly correlated with low agreeableness @xcite . high neuroticism is strongly linked to anxiety and depression, while low neuroticism is linked to emotional stability. 2 this dimension may be expressed through feelings such as fear, sadness, or frustration @xcite . ## evaluation in this section, we describe the method for training and evaluating our proposed model, along with the various baseline models we compared against. ## results our evaluation aims to answer the following: 1. how successful are attention-based models at predicting personality? 2. what is the distribution of high signal versus low signal messages? 3. what is the relative importance of messagelevel attention over word-level attention? ## qualitative value of identifying informative text the high-signal text identified by our attentionbased models potentially provides additional, qualitative value for researchers interested in the rela-tionship between language and personality. bag-ofwords approaches to language modeling can identify attribute-relevant words (e.g. word clouds), but this can be limiting as it lacks the context in which the words appear. by contrast, a personality researcher interested in how high extraversion, for example, manifests itself in one's language use can use our learned attention weights to identify whole messages that may warrant further study. to test whether our high-signal text might be of qualitative value to researchers, we asked two experts on personality (psychologists with past research in the area) to view 100 paired messages sets (20 per dimension) and select which set was more informative of the individual's personality. each paired set consisted of 5 messages within the top third of message weights and 5 in the bottom third for a given user. to reduce the frequency of long messages, we only selected messages whose length was at most 20 characters above or below that user's average message length. the users themselves were randomly sampled from those in the top or bottom 10th percentile of each dimension and who had at least 20 messages total. note that personality psychologists, though experts in how high ope trippy day ahead .... nothingness at last .... shutter island was good .. . they are over ... yah my phone is not working ... high con stoked on the exam schedule ! 40 % math midterm ? thank god 3/4 count . got a co-op job interview ! woo ! just had some damn good pears . note to self : buy more ? damnit . found free bag of skittles in the vending machine , jackpot . high ext at the beach with keira ! ! ! getting ready for brittany's dance recital tonight ! ! had fun at nathans barmitzvah last night ! ! ! i have made 72 cupcakes in the last 3 days ! ! ! ! lol just finished my science project :) low agr sooo excited for new school year :) going top make it awesome grudges are so ridiculous and pointless ¿ ¿ ahh shit almost 1 ! ? i need to finish this paper ! ! ! that sure was a fun ride home o.o wants to just skip to the next weekend . high neu can't believe i got that done in time ....... packing to go back to school makes me sad . losing things and is getting extremely frustrated . :( is amazed at how similar cameras are to your eyes . whhhaaa ? it's only wednesday ... in general, our proposed model seems to identify text that is informative of one's personality, both in terms of individual words and the overarching themes of the message as a whole, though this is easier for some dimensions than others. modeling document relevance is useful, then, not just as a means to boost performance but as a tool to aid those seeking to better understand language. ## related work personality modeling from language is becoming increasingly important for many social scientific @xcite found personality features to be highly predictive of depression and ptsd. lynn et al. (2017) demonstrated that the performance of document classification models can be improved by adapting to a variety of human factors, including personality. personality has also been shown to be useful for deception detection @xcite and recommendation systems @xcite . most research on personality modeling focuses on the big five, or five-factor model @xcite . personality is traditionally measured using questionnaires, but cost and scalability issues make computational methods preferable. linguistic inquiry and word count (liwc) @xcite features are popular for personality modeling @xcite @xcite , as they readily provide insight into the type of language that correlates with certain personality dimensions. however, using predefined lexica is limiting; @xcite and @xcite showed significantly improved prediction when using topics and n-grams extracted from their training set. when working with a very limited amount of data, @xcite found pre-trained word embeddings to be effective. deep learning approaches to personality prediction are limited. @xcite used a convolutional neural network (cnn) with max pooling, alongside traditional document features (e.g. word count). their best results were obtained when they filtered out sentences that did not contain strong emotion words (as determined via lexica) during preprocessing. this supports our intuition that some messages contain stronger signal than others, though our approach allows the model to identify such cases. @xcite also used cnns with max-and average-pooling to predict personality over facebook statuses. they experimented with fully-connected neural networks and bidirectional recurrent neural networks, but ultimately cnns performed best. both @xcite and @xcite used datasets that were significantly smaller than ours (@xmath3, respectively) and their problems were framed as binary classification rather than regression foot_4 . ## conclusion language-based personality prediction is an important task with many applications in social science and natural language processing. we presented a hierarchical sequence model with message-and word-level attention that learns to differentiate highand low-signal messages. our approach, which novelly models the idea that all messages are not equally valuable for psychological regression tasks, achieves new state-of-the-art results for personality prediction and provides insight into the relationship between language and personality. our analysis demonstrates that the level of abstraction at which attention is applied can have a significant impact on a model's overall performance. finally, this work highlights the critical role of document relevance as we progress with further human-centered natural language processing.