Predicting Responses to Psychological Questionnaires from Participants’ Social Media Posts and Question Text Embeddings (2020)

Abstract: Psychologists routinely assess people’s emotions and traits, such as their personality, by collecting their responses to survey questionnaires. Such assessments can be costly in terms of both time and money, and often lack generalizability, as existing data cannot be used to predict responses for new survey questions or participants. In this study, we propose a method for predicting a participant’s questionnaire response using their social media texts and the text of the survey question they are asked. Specifically, we use Natural Language Processing (NLP) tools such as BERT embeddings to represent both participants (via the text they write) and survey questions as embeddings vectors, allowing us to predict responses for out-of-sample participants and questions. Our novel approach can be used by researchers to integrate new participants or new questions into psychological studies without the constraint of costly data collection, facilitating novel practical applications and furthering the development of psychological theory. Finally, as a side contribution, the success of our model also suggests a new approach to study survey questions using NLP tools such as text embeddings rather than response data used in traditional methods.

## introduction psychologists conduct personality research in order to understand what aspects and factors consistently distinguish people from each other on an individual level. this is relevant because personality influences important life outcomes such as occupational and educational success and even physical and mental health @xcite . traditionally, psychologists measure personality through questionnaires, by having participants read and answer questions on a rating scale, for instance from "strongly disagree" to "strongly agree". however, acquiring questionnaire data in psychological research is often a tedious and costly process, as study participants must be recruited and motivated to complete questionnaires. this problem is particularly pronounced for longer surveys, which suffer from low completion rates and careless responses due to low participant motivation @xcite @xcite . therefore, the ability to predict questionnaire responses would be of great use to researchers. the main contribution of this paper is to address this issue. we propose a system that uses the participants' social media texts and the question texts to predict the participants' responses. the system extracts bert embeddings from the two input components and then trains a predictive model. after training, we can predict the responses for every new participant, new question or both, only requiring the participants' social media texts and the question texts. if our approach is successful, it will greatly reduce the costs of collecting response data for psychologists, especially when the number of new participants or questions is large. moreover, the success of our model also suggests a new approach to analyse psychological questionnaires by using natural language processing (nlp). traditionally, psychologists analyse questionnaires using only the participants' responses to the questionnaires, rather than the text and lexicon of the questions themselves @xcite . for instance, participants' responses are used to measure the similarity between two questions. however, these traditional approaches have high requirements for available response data which are costly to collect and moreover, lack the flexibility of integrating new participants or questions into their studies. in contrast, our novel approach of applying nlp into questionnaire research, as implemented in our model, offers the possibility of extending existing survey datasets and questionnaires to new subject populations and to new theoretical constructs, greatly improving the generalizability of psychological research and opening up many practical applications for personality research. ## experiments we first conducted two experiments to separately test the quality of the question and user embeddings, and then a third main experiment in which both user and question features were used together to predict the response of a user to a question. we since text embeddings of the questions for assessment questionnaires have not been explored in previous studies, the first and third tasks are novel and play a crucial role in exploring this new prediction approach. the second task, in which the users' have been characterized by their social media posts, has been explored previously. however, we will show that bert embeddings outperform the traditional latent dirichlet analysis (lda) used in prior work. our main goal is to explore the novel idea of using the text embeddings of questions and of users to predict user responses to questions. therefore, we do not focus on designing sophisticated deep learning models. instead, we chose to use simpler but powerful, widely used models: ridge regression and k-nearest neighbors. ## conclusion our study proposes a novel task: predicting responses of participants to a personality questionnaire, using their social media texts and the texts of the questions they are asked. unlike prior work, we are able to successfully make out of sample predictions for both new survey questions and new participants. our approach could potentially reduce the cost of data collection for psychologists, but more importantly our findings showcase a novel method for improving the generalizability of personality research. they also open up many novel applications that rely on existing social media and survey data to make predictions for out-of-sample participants and survey questions. finally, our results offer the promise of improving psychological research by representing survey questions with informative text embeddings, which can be used by researchers and theorists to better understand the core dimensions of personality. we look forward to future work that integrates psychological theory with novel advances in natural language processing, to better measure, predict, and understand what distinguishes humans from each other. thea f van de mortel et al. 2008. faking it: social desirability response bias in self-report research. australian journal of advanced nursing, the, 25(4):40. a susan m niessen, rob r meijer, and jorge n tendeiro. 2016. detecting careless respondents in webbased questionnaires: which method to use? journal of research in personality, 63:1-11. gregory park, h andrew schwartz, johannes c eichstaedt, margaret l kern, michal kosinski, david j stillwell, lyle h ungar, and martin ep seligman. 2015. automatic personality assessment through social media language. journal of personality and social psychology, 108(6):934. trivellore e raghunathan and james e grizzle. 1995. a split questionnaire survey design. journal of the american statistical association, 90(429):54-63. brent w roberts, nathan r kuncel, rebecca shiner, avshalom caspi, and lewis r goldberg. 2007. the power of personality: the comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes. perspectives on psychological science, 2(4):313-345. h andrew schwartz, johannes c eichstaedt, margaret l kern, lukasz dziurzynski, stephanie m ramones, megha agrawal, achal shah, michal kosinski, david stillwell, martin ep seligman, et al. 2013. personality, gender, and age in the language of social media: the open-vocabulary approach. plos one, 8(9):e73791. suvash sedhain, scott sanner, darius braziunas, lexing xie, and jordan christensen. 2014. social collaborative filtering for cold-start recommendations. in proceedings of the 8th acm conference on recommender systems, pages 345-348. chris sumner, alison byers, and matthew shearing. 2011. determining personality traits & privacy concerns from facebook activity. black hat briefings, 11(7):197-221.