Why Is MBTI Personality Detection from Texts a Difficult Task? (2021)

Abstract: Automatic detection of the four MBTI personality dimensions from texts has recently attracted noticeable attention from the natural language processing and computational linguistic communities. Despite the large collections of Twitter data for training, the best systems rarely even outperform the majority-class baseline. In this paper, we discuss the theoretical reasons for such low results and present the insights from an annotation study that further shed the light on this issue.

## introduction apart from being long and requiring to be administrated by a skilled human assessor in artificial circumstances (laboratory conditions), the traditional questionnaire-based personality tests often introduce social desirability bias @xcite and the reference-group effect @xcite , which can be introduced either from the subject's or assessor's side. to avoid those biases, it was suggested that analysing person's writing provides more objective assessment of one's personality than the traditional questionnaires @xcite . ## datasets to test our hypotheses, we used two datasets, a subset of the mbti-twitter dataset @xcite , and a dataset that we collected especially for this purpose, the mbti-mturk dataset. foot_0 ## annotation as there are no studies laying out linguistic characteristics of the mbti model, we created the annotation guidelines starting from the behavioral characteristics. the mbti's theoretical and practical framework provides detailed profiles for each type ranging from general and typical characteristics to real-time depictions of those characteristics, in the contexts of relationships, marriage, work, and learning settings @xcite . we translated this behavioral information into linguistic and textual signals in general, as well as some specific signals relevant for the contexts we asked the participants to write about. to define the characteristics that would linguistically distinguish the two polarities of the ei dimension, we focused on the processes of socialization depicted in the text. as the dimension of jp is mainly related to organizational preferences, we chose the linguistic signals referring to arrangement of schedules and plans. the fact that people high on j tend to be more exact and pay more attention to details, whereas people high on p show more flexible and accidental characteristics @xcite we translate into guidelines considering sentence structures and grammar. the sn dimension identifies preferences for the characteristics of tasks and information people would like to process, either using facts and five senses (s), or using imagination and abstraction extravert introvert mention of new people (e.g. crowd, strangers) mention of closer people rather than any group of people (e.g. husband, family) mention of social activities and events that contain interaction with other groups of people (e.g. party, dancing, couchsurfing) mention of individual activities or activities that can be done without interaction with other people (e.g. by myself, spending time at home) mention of outside world and vibrant places (e.g. bars, restaurants) mention of inner world, and calm and quiet places (e.g. home, museum) we references i references use of intensifiers and exclamation marks hedging more assertive, positive, enthusiastic arguments less assertive arguments thinking feeling logical reasoning for their actions and choices (e.g. reading books for learning) emotional reasoning for their actions and choices (e.g. reading books for gateway feeling) mention of opinions, ideas, comparisons mention of people, values, feelings direct (e.g. reading is nice) tactful, indirect (e.g. reading feels nice) ## findings from the pilot rounds the annotation and mark-up obtained during the pilot rounds revealed several important characteristics of the task. ## final annotation results the annotation of the final dataset that consists of 96 instances from the mbti-twitter dataset and 96 instances from the mbti-mturk dataset revealed further particularities of each dataset, and shed some light on how feasible the task is, even for trained human annotators. ## discussion and conclusions the results of presented analyses shed some light on possible causes of the poor performances of the automatic mbti personality detection systems proposed so far. furthermore, they indicate that the linguistic cues found in short texts do not seem to directly correspond to the results of the questionnairebased results, which are commonly used as the 'gold labels' in classification experiments. this is in line with academic studies that showed the psychometric inadequacy of the questionnaire @xcite . 5 the high number of instances without enough signal and with mixed signals in the mbti-twitter dataset across all mbti dimensions, and especially the jp dimension, leads to high amounts of noise in the training datasets. therefore, it is not a surprise that the best systems trained on those datasets rarely outperform even the majority-class baseline. even in the cases where user-instances contain sufficient signals and the agreement between the human annotators is high, the agreement between the annotators and the gold label is still very low for three out of four dimensions ( finally, the results of the in-depth analyses performed during the two pilot rounds (section 4) revealed three phenomena that need to be taken into