Matching Theory and Data with Personal- ITY : What a Corpus of I talian Y ou T ube Comments Reveals About Personality (2020)

Abstract: As a contribution to personality detection in languages other than English, we rely on distant supervision to create Personal-ITY, a novel corpus of YouTube comments in Italian, where authors are labelled with personality traits. The traits are derived from one of the mainstream personality theories in psychology research, named MBTI. Using personality prediction experiments, we (i) study the task of personality prediction in itself on our corpus as well as on TWISTY, a Twitter dataset also annotated with MBTI labels; (ii) carry out an extensive, in-depth analysis of the features used by the classifier, and view them specifically under the light of the original theory that we used to create the corpus in the first place. We observe that no single model is best at personality detection, and that while some traits are easier than others to detect, and also to match back to theory, for other, less frequent traits the picture is much more blurred.

## introduction human personality is a psychological construct aimed at explaining the wide variety of human behaviours in terms of a few, stable and measurable individual characteristics @xcite @xcite . research in psychology has formalised these characteristics into what are known as trait models. two are the major models widely adopted also outside of purely psychological research (see section 2.1): big five @xcite and myers-briggs type indicator (mbti) @xcite . the psychological tests commonly used to detect prevalence of traits include human judgements regarding semantic similarity and relations between adjectives that people use to describe themselves and others. this is because language is believed to be a prime carrier of personality traits @xcite . this aspect, together with the progressive increase of available user-generated data from social media, has prompted the task of personality detection, i.e., the automatic prediction of personality from written texts @xcite @xcite @xcite . personality detection can be useful in predicting life outcomes such as substance use, political attitudes and physical health. other fields of application are marketing, politics, psychological and social assessment and, in the computational domain, dialogue systems @xcite and chatbots @xcite . as a contribution to personality detection in languages other than english, we have developed personal-ity, a novel corpus of youtube comments in italian, which are annotated with mbti personality traits. the corpus creation methodology, described in detail in @xcite , makes use of a distant supervision approach that can also serve as a blueprint to develop datasets for other languages. in this work, we use personal-ity to cast light not only on the feasibility of personality detection per se on this corpus, but also on the relationship between our data and the theory it is based on. more specifically, we want to investigate if our distantly obtained labels are meaningful with respect to the psychological theory they come from, and whether language does indeed reflect the traits that should be associated with such labels. to do this, we run a series of in-and cross-dataset experiments; on top of performance analysis we conduct an in depth study on the relevant features used by the classifiers, and how they might relate to the source psychological theory. personal-ity is available at https://github.com/elisabassignana/personal-ity . ## background personality profiling is addressed both from a psychological viewpoint (traits model for the classification of personality) and from a computational perspective in the field of natural language processing. we provide relevant background from both sides, as they are intertwined in our work. ## data to run experiments on personality detection, we have created a dedicated corpus with mbti labels, exploiting distant supervision: personal-ity @xcite . here, we summarise the choices that we made regarding the source of the data and the theoretical trait model, the procedure followed to construct the corpus, and provide a description of the resulting dataset. in addition, we also partly use the existing twisty (see section 2.2 and because we deal with personal data, and because we do believe profiling is a sensitive task in general, we also provide an ethics statement. ## experiments we ran a series of initial experiments on personal-ity, which we use to peek into the relationship between labels and theory, and which can also serve as a baseline for future work on this dataset. the choices related to the experimental framework have been driven by the interest on investigating signals and linguistic cues for personality traits, analyzed through the lens of psychological studies on personality. this perspective has therefore led us not use deep learning techniques that would have made this analysis task more complex. rather, we opt for state-of-the-art approaches commonly used in author profiling, developing interpretable models that can aid the analysis process. these will make it possible to perform a linguistic and psychological analysis, though perhaps at the expense of performance. specifically, we used the sklearn @xcite implementation of a linear svm (linearsvm), with standard parameters, and tested three types of features: lexical-, stylistic-, and embeddings-based. we used four placeholders for hashtags, urls, usernames and emojis. at the lexical level, we experimented with word (1-2) and character (3-4) n-grams, both as raw counts as well as tf-idf weighted. character n-grams were tested also with a word-boundary option. considering stylistic features, we investigated the use of emojis, hashtags, pronouns, punctuation and capitalisation. lastly, we also experimented with embeddings-based representations, using more generic @xcite and youtube-specific (nieuwenhuis and nissim, 2019) pre-trained models. we created one representation per user by averaging the vectors for all words written by that user. we used 10-fold cross-validation, and assessed the models using macro f-score. we deem this way of averaging over f-scores per class appropriate, since the dataset is quite unbalanced, but we want good performance for each class. for comparison, we calculated a majority baseline (maj). to test compatibility of resources and to assess model portability, we also ran cross-domain experiments on personal-ity and twisty. we divided both corpora in fixed training and test sets with a proportion of 80/20 so that the test set stays the same for in-domain and cross-domain settings. indeed, we run the in-domain models again using this split. the models use lexical features as they are the one performing better overall the others. results are shown in we specify that in every experiments done (tables 3 4 -5) we chose to look for the best model able to predict the whole mbti personality and so we report the highest scores based on averages of the four traits. considering the four dimensions individually better results can be obtained by using specific models. ## feature analysis and linguistic cues for personality traits in this section we discuss possible correlations between linguistic cues derived from the experiments described in section 4 on the personal-ity corpus and psychological traits descriptions deriving from the field of psychology. referring to in the sensing list of words we highlighted gli ('the'), del ('of the'), di ('of') and in ('in') as those tokens are used to specify details. in the opposite column, in line with the definition, we just found the words ehm and vorrei ('i would like to'). in line with the definition of thinking we highlighted the n-grams ma io ('but i'), vero ('true'), vedo ('i see'), capire ('to understand'), perchè ('because') and questa ('this'). for their opposite we found io sono ('i am'), dentro ('inside') and trovo ('i think', 'i find'). they are more relaxed and look for liberty. in the judging column there are words such as alle ('at'), nella ('in'), tuoi ('yours'), te ('you'), metti ('put'), neanche ('neither'), da ('from', 'by') and user (which is a normalized label deriving from pre-processing; it replaces references to specific users). these can all be used to make precise plans, and fit with the description of determined people, comfortable with rules and guidelines. the more interesting word n-grams in this set, is minecraft, which is the name of a video game where planning skills are fundamental. in the opposite pole, perceiving, some distinctive words are: bene ('good'), beh, the label emoji and anche ('also'). a final consideration about the analysis above is that the correlations we found are sometime weak and not so explicit, especially for the s-n trait and we observed that this is coherent with the not so high results obtained from experiments in section 4: it is likely that the absence of strong evidences linking linguistic cues with psychological ones, makes the decision of the classifier hard. notice that a similar observation concerning the difficulty to predict the s-n trait has been reported in @xcite on an english corpus, suggesting that such trait could in general be more related to perception, with a weak linguistic signal. a second observation is that for a second qualitative analysis we used wordify foot_4 , a tool developed by bocconi university whose intent is to identify words that discriminate categories in textual data. ## conclusions we presented personali-ity, a novel youtube-based corpus for personality prediction in italian. an exploratory empirical investigation on our new corpus confirms that identifying mbti personality trait from social media texts is challenging. lexical features perform best, but they tend to be strictly related to the context in which the model is trained and so to overfit. concerning our experiments on twisty, our model outperforms the original twisty results @xcite for italian and provides a new baseline on this corpus. moreover we performed cross-domain experiments between youtube and twitter data, achieving scores above the baseline. performance drops from the in-to the cross-genre setting show limits in portability, and leave a lot of space for improvements. in general, better results could, probably, be obtained using more complex models by using neural networks (e.g. lstm) @xcite . the choice to use a simpler svm model was driven by its greater understandability and transparency. these qualities allowed the features analysis of section 5. the inherent difficulty of the task itself is confirmed and deserves further investigations, as assigning a definite personality is an extremely subjective and complex task even for humans. the distant supervision approach remains promising, also applied to youtube data. indeed, assigning an 'absolute' personality to an individual is difficult. even if, to some extent, distant supervision is inaccurate and can lead to the creation of corpus containing bias and noise, since there is no control on user statements, we cannot avoid considering that according to the literature also professional psychological tests can lead to inaccurate results. moreover, ds, despite its inaccuracy, allows the availability of a large amount of data, necessary for addressing the task by using machine learning approaches.