Items from Psychometric Tests as Training Data for Personality Profiling Models of T witter Users (2022)

Abstract: Machine-learned models for author profiling in social media often rely on data acquired via self-reporting-based psychometric tests (questionnaires) filled out by social media users. This is an expensive but accurate data collection strategy. Another, less costly alternative, which leads to potentially more noisy and biased data, is to rely on labels inferred from publicly available information in the profiles of the users, for instance self-reported diagnoses or test results. In this paper, we explore a third strategy, namely to directly use a corpus of items from validated psychometric tests as training data. Items from psychometric tests often consist of sentences from an I-perspective (e.g., ‘I make friends easily.’). Such corpora of test items constitute ‘small data’, but their availability for many concepts is a rich resource. We investigate this approach for personality profiling, and evaluate BERT classifiers fine-tuned on such psychometric test items for the big five personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism) and analyze various augmentation strategies regarding their potential to address the challenges coming with such a small corpus. Our evaluation on a publicly available Twitter corpus shows a comparable performance to in-domain training for 4/5 personality traits with T5-based data augmentation.

## introduction the field of author profiling originally emerged from the study of stylometry @xcite and, with the rise of social media @xcite , now considers a variety of attributes, including demographic data such as age, sex, gender, nationality @xcite , personality traits @xcite , or psychological states such as emotions, or medical conditions like mental disorders @xcite . such automatic methods enable large-scale social media data analyses even for (combinations of) variables for which results from surveys are not available. therefore, personality profiling in social media helps to paint a more comprehensive, complete, and timely picture for parts of a society. state-of-the-art models reconstruct personality traits or mental health states from posts of social media users by relying on ground-truth data that links such posts to the correct annotation @xcite . the ground-truth data is typically obtained by either (1) asking participants to complete a validated survey that measures the desired variable and asking the participants to share their social media profiles, (2), by relying on self-reports of users, e.g., disclosure of a condition in the user's profile description, or (3), by having experts annotate profiles for particular properties. the quality of data obtained might therefore suffer from socialdesirability bias, from being a non-representative subsample, from a lack of validated diagnoses, or from noise stemming from the challenge that annotators do not have access to the actual characteristics of users @xcite . we explore another route for which we hypothesize that it addresses these issues, but at the cost of only having access to very small data sets: we propose to leverage the existing set of high-quality, validated, and reliable psychometric instruments to measure psychological traits directly. psychometric tests often come in the form of questionnaires which contain items, allowing a person to report variable cor. item text openness + "am interested in many things." openness -"do not like art." extraversion + "warm up quickly to others." extraversion -"am hard to get to know." thus, our contributions in this paper are that we (1) assemble a corpus from publicly available psychometric tests for the 'big five' variables of openness, conscientiousness, extraversion, agreeableness, and neuroticism @xcite , which have been shown to be principled factors of personality @xcite . based on these data, we (2) fine-tune bert @xcite and evaluate it on an existing personality trait corpus @xcite . furthermore, (3) we evaluate three data augmentation methods, namely paraphrasing with t5 @xcite , and item generation with gpt-2 @xcite and synonym replacements with easy data augmentation @xcite . our results, (4), show that the models perform en par with in-domain training for 4/5 personality trait variables. another test of personality traits would be the hexaco personality inventory-revised @xcite . it measures six factors of personality @xcite with 200 items, namely honesty-humility, emotionality, extraversion, agreeableness, conscientiousness, and openness to experience. data. psychometric tests found application in the analysis of social media user's personality in the past. an influential study has been the work by @xcite , who collected facebook data with a dedicated application @xcite in which users completed the 100item ipip-neo-100 questionnaire @xcite . the users further shared access to their status updates. this data is not available any longer. the data for the pan-author-profiling shared task in 2015 has been collected in a similar way @xcite . 3 it consists of tweets of 294 english twitter profiles (besides spanish, italian and dutch twitter profiles), which are annotated with gender, age, and the 'big five' personality traits. the personality traits were self-assessed by the twitter users with the bfi-10 ( @xcite , which is an economic psychometric test that allows the personality to be recorded with only 10 items. we use this corpus for evaluation. combining tests and social media data. an interesting combination of psychometric tests with social media posts, which is likely the one most similar to our paper, is the work by @xcite . the authors make use of social media data of users to automatically fill the ipip-neo @xcite psychometric test to predict the social media user's 'big five' personality traits. they do so by embedding sentences and items with bert into the same distributional space, followed by a k-nearestneighbor classification. this approach constitutes the opposing approach that we chose in our paper - @xcite use social media data to fill a psychometric test. we use psychometric tests to classify social media data. we refer the reader to stajner and yenikent (2020) for a more comprehensive overview of related work. ## conclusion & future work we outlined a novel methodology for automatic author profiling in social media users without a costly collection of annotated social media data. instead, we directly train on items from validated psychometric tests. this data selection procedure has some advantages: items of psychometric tests are carefully validated textual instances. such corpora of such items constitute "small data", but are available for a large number of concepts. therefore, developing a method to induce classifiers directly from psychometric tests is also a promising avenue for future research. for the tasks of developing models measuring the big five personality traits, we tested on twitter data that has been collected by asking users to fill out a (different) test. the transfer appears to be achievable, we obtain results for four out of five personality traits which are en par with in-domain models, using t5 data augmentation (except openness, which has very few test instances). an important remaining research question is how models can be obtained that show consistently good results across concepts. in a real-world setup, test data from the target domain would not be available to make model selection decisions. one way to go might be to combine various augmentation methods. another approach would be to use items as prompts in a zero-shot learning setup.