How to Obtain Reliable Labels for MBTI Classification from Texts? (2021)

Abstract: Automatic detection of the Myers-Briggs Type Indicator (MBTI) from short posts attracted noticeable attention in the last few years. Recent studies showed that this is quite a difficult task, especially on commonly used Twitter data. Obtaining MBTI labels is also difficult, as human annotation requires trained psychologists, and automatic way of obtaining them is through long questionnaires of questionable usability for the task. In this paper, we present a method for collecting reliable MBTI labels via only four carefully selected questions that can be applied to any type of textual data.

## introduction the myers-briggs type indicator (mbti) model @xcite is one of the most widely used non-clinical psychometric models @xcite . it classifies people into two groups across four dimensions: extraversion/introversion (e/i), sensing/intuition (s/n), thinking/feeling (t/f), and judgement/perception (j/p). this leads to a total of 16 personality types. the first three dimensions are based on the theoretical work of carl @xcite , while the fourth dimension was added later by myers and @xcite . the mbti personality framework has already been used for decades in educational and industry settings, e.g. for finding jobs that best resonate with the person's preferences for information processing (s/n and t/f dimensions), finding work organization types that best resonate with the person's preferred judgement processes (j/p dimension) thus leading to better job satisfaction, and for better matching work environments with the person's preferences (e/i dimension) to lower employee turnover @xcite . the original mbti questionnaire contains 93 questions and is not freely available. 1 due to the popularity of mbti framework (it is estimated that more than 2 million us adults complete the inventory every year), 2 there is a number of freely available alternative mbti questionnaires on the internet, with the 16personalities test 3 being one of the most popular ones. according to the myers-briggs foundation 4 and the 16personality test website, 5 both questionnaires satisfy the accepted standards for test validity and reliability. nevertheless, the mbti questionnaires have received a noticeable criticism from the academic community @xcite for not relying on a scientifically proven (i.e. data-driven) background, but rather on qualitative measures such as observation and introspection. the other common criticism is the binary nature of the questionnaire as it is known that the majority of people usually lies somewhere in the middle of the scales @xcite . the questionnaire-based personality detection has several weaknesses: it requires trained human assessors; it is prone to social desirability bias @xcite and reference-group effect @xcite ; it is questionable if answering questionnaires is a natural way of showing ones personality (as opposed to free writing or behaviour "when nobody watches"). to detect mbti typologies in a more natural way and without necessity for trained human assessors, many studies have attempted at building systems for automatic detection of mbti personality types from text in the last several years. attempts have been made for automatic detection of mbti personality types from: tweets written in english @xcite , six other western european languages @xcite , and japanese @xcite ; english posts collected from personality cafe forum foot_2 available in kaggle; 7 and english reddit comments @xcite . despite being trained on large amounts of textual data (over one million), and modelled as four binary classification tasks, the best systems performed only slightly better than the random and majority-class baselines, regardless of the architecture used. some studies suggested that tweets might not contain sufficient amounts of mbti signals (even after concatenating up to 150-200 tweets per user) due to the nature of twitter posts @xcite @xcite ). another issue with all those studies and obtained results might be that the systems are supervised and were trained with gold labels obtained via mbti questionnaires that suffer from all earlier mentioned weaknesses. in our recent study @xcite , we found a low association between the mbti types obtained via questionnaires and the mbti signals found in the short texts written by participants (tweets and free texts on carefully chosen topics). at the same time, the inter-annotator agreement of two expert annotators assigning mbti types based on those free texts was quite high @xcite . ## questionnaire the whole questionnaire consisted of one optional question "you might have obtained your mbti type in the past via questionnaires. if you know your mbti type, please type it here", four compulsory demographic questions, four compulsory questions with answers on a 1-5 scale that aimed to capture the participants mbti type, and two compulsory open-end questions. demographic questions encompassed gender, age, whether or not english is their native language, and the highest level of education obtained (figure 1 ). the gender question had four possible answers: female, male, other, prefer not to specify. five age groups were offered to choose from: 18-25, 26-35, 36-45, 46-55, and over 55. after answering demographic questions, participants were provided with four questions that aimed to capture their mbti type, and were asked to provide an answer on a 1-5 points scale. those four questions are the central contribution of this study. by following the idea that aspects of leisure time represent the most natural version of personality, as it is directed by high degrees of intrinsic motivation @xcite , the questions are focussed on typical leisure time activities-hobbies and vacations. this also gave us the opportunity to utilize the previously proposed open-end questions @xcite in the validation process (section 5). in deciding the content of the questions for each individual dimension, we followed the main definitions provided by @xcite . although each mbti dimension corresponds to multiple practical and behavioral characteristics, the core theoretical focus for every dimension is consistent. the first question (for the e/i dimension) was designed with the idea of capturing whether the person prefers to be surrounded by people and social interactions, on one end of the scale (1 = extraverted), or to spend quiet and calm time by themselves, on the other end of the scale (5 = introverted). the second question (for the s/n dimension) aims to capture the characteristics of the tasks people would prefer to process, concrete or intuitive, by asking whether they prefer technical and hands-on hobbies (1 = sensing) or abstract and imaginative (5 = intuitive). the third mbti dimension (t/f) is fundamentally about how people make their decisions, whether based on rational or emotional motives. as people do not engage with strict decision-making processes during their free time, which is ultimately based on their personal interests, the question measured the preference for rational (1 = thinking) or emotional (5 = feeling) reasoning for liking a certain hobby. the fourth question aimed to capture the preference for spontaneous and flexible (1 = perceiving), or a wellplanned (5 = judging) schedule at vacations. we initially prepared two questions per each mbti dimension and performed a pilot study with 30 participants to choose those questions (figure 2 ) that better correspond to the mbti types provided by the participants, and the mbti annotations by two annotators. finally, participants were asked to answer to two open-end questions, which we previously proposed @xcite as the optimal questions for annotating mbti types from texts: the two questions were preceded by the following instructions: "the following questions aim to understand your life style preferences. while answering, please write down the first things that come to your mind without much contemplation." to be accepted, each answer needed to contain a minimum of 300 characters. ## challenges in data collection data was collected via amazon mechanical turk (amt) platform. we prepared the questionnaire as google forms and provided the link to it in the hit of the amt platform. we experimented with various setups in the platform: different values for monetary compensations, allowing only those participants with high scores on previous tasks, different times for validation of the answers and payment. the only variable that noticeably influenced the time needed for obtaining completed hits was whether or not we restrict the participants according to their performance on the previous hits. without any restrictions, we were obtaining approximately 50 completed hits per hour. the main bottleneck in the whole procedure was the need for manually checking "honesty" of the answers to the open-end questions. we manually checked all answers by particularly focusing on checking whether or not: (1) all personality questions contained the same answer (although theoretically possible that a person has the same answer to all four questions, it is very likely that this behaviour instead signifies that the amt worker just wanted to finish the task as soon as possible and get the monetary compensation); (2) the answers to the open-end questions make sense, i.e. are not just a random sequence of 300 characters; (3) the answers to the open-end questions are copied from the internet; (4) the same worker has already completed the task, and if they did, we checked whether the answers to both multiplechoice and open-end questions were similar in both completed hits. we found that approximately one third of the completed hits contained answers that were copied from the internet. those hits were disregarded and those workers did not obtain monetary compensation to prevent them from doing it again. in those cases where more than one hit was completed by the same worker, if the answers were similar, we paid the monetary compensation for all of them and maintained only one randomly chosen hit for our dataset. in those cases where the answers in the hits of the same worker had many significant differences, we paid the monetary compensation, but excluded all hits from that worker from the dataset. ## validation we performed two types of validation of our questionnaire. first, we calculated the agreement of the answers to our personality questions with the mbti labels provided by the participants, for those cases where the mbti label was provided (section 5.1). given that the mbti labels obtained by using popular questionnaires might not be reliable for mbti type detection from textual utterances @xcite , the results obtained through this validation method should be taken with the grain of salt. second, we validated our questionnaire via manual annotation of mbti types on the answers to the open-end questions (section 5.2). ## dataset statistics the total number of users whose answers we collected via amazon mechanical turk platform, after the manual quality check (section 4) was 1038. distributions of labels for each dimension separately, for the whole dataset (mturk-q) and the portion that contained mbti types entered by amt workers (mturk-mbti) is provided in figure 3 together with the distribution of labels in different populations/datasets mentioned in section 2. as can be observed, the distributions of labels for the s/n dimension vary significantly across different datasets, with our dataset being the only one that follows a similar distribution as the one found in the general us population. for the e/i dimension, distribution of labels in all datasets deviate from the distribution in the general us population. as already mentioned in some of the previous studies @xcite @xcite , this is not surprising, as it is known that introverts prefer online communication @xcite . for the t/f and j/p dimensions, all datasets roughly follow the distribution in the general us population. the distribution of mbti types across 340 amt workers that provided their previously obtained mbti type is presented in figure 4 . in our dataset, the three most frequent mbti types are infp, infj, and intj. it is interesting that those three mbti types were also reported as the most frequent ones in the mbti twitter dataset (plank and hovy, 2015), while being reported among the least frequent ones in the general us population @xcite . 8 distributions of the answers to our personality questions and demographics questions are presented in figures 5 and 6 . ## conclusions in this study, we proposed a set of four questions for quickly obtaining mbti labels that better correspond to the expert human annotations of mbti traits from short texts than the commonly used mbti labels obtained via lengthy questionnaires. apart from being faster to administer and allowing for obtaining large quantities of short texts on various topics, the proposed method also offers a more fine-grained mbti typology, overcoming thus the common objections about the binary nature of mbti questionnaires. this is particularly important for advancing research efforts in automatic mbti personality detection from texts, as