Decoding LLM Personality Measurement: Forced-Choice vs. L ikert (2025)

Abstract: Recent research has focused on investigating the psychological characteristics of Large Language Models (LLMs), emphasizing the importance of comprehending their behavioral traits. Likert scale personality questionnaires have become the primary tool for assessing these characteristics in LLMs. However, such scales can be skewed by factors such as social desirability, distorting the assessment of true personality traits. To address this issue, we firstly incorporate the forced-choice test, a method known for reducing response bias in human personality assessments, into the evaluation of LLM. Specifically, we evaluated six LLMs: Llama-3.1-8B, GLM-4-9B, GPT-3.5-turbo, GPT-4o, Claude-3.5-sonnet, and Deepseek-V3. We compared the Likert scale and forced-choice test results for LLMs’ Big Five personality scores, as well as their reliability. In addition, we looked at how temperature parameter and language affected LLM personality scores. The results show that the forced-choice test better captures differences between LLMs across various personality dimensions and is less influenced by temperature parameters. Furthermore, we found both broad trends and specific variations in personality scores across models and languages.

## introduction in recent years, large language models (llms) have demonstrated remarkable human-like cognitive abilities in dialog generation tasks. as conversational performance approaches that of humans, researchers are increasingly examining whether these models can recognize and express human traits similarly to people @xcite @xcite . interactions between llms and humans may influence users' ideolo-* corresponding author. likert scales are popular among researchers because of their ease of administration and quantitative clarity @xcite , and they are frequently used to assess personality in llms @xcite . when assessing personality traits reflected in llm's generation behavior (hereafter referred to as personality traits in llm output), prompt words indicate the task and a multiplechoice format allows the models to choose the best answers based on the descriptions provided @xcite . however, this approach has a double-edged effect: while it simplifies the process, it frequently introduces social desirability responding (sdr) @xcite , in which participants (or models) choose responses that conform to social desirability, instead of their latent personality traits. furthermore, llms may be influenced by the social values embedded in their training data, resulting in discrepancies between their "true" personality traits and performance on assessments. to reduce the effect of social desirability, forced-choice tests have emerged as an effective psychometric tool @xcite . this format of tests improves validity by including a social desirability balancing mechanism in the design. participants are asked to rank statements with similar social desirability in one item block based on their circumstances. this method effectively reduces the influence of social desirability on results, thus increasing accuracy @xcite @xcite . however, several confounding factors affect the personality test of llms @xcite . for example, the temperature parameter affects the variety of generated responses and thus affects the stability of the assessment results @xcite . cultural differences in language corpus may also lead llm's generation behavior to exhibit different personality traits across languages @xcite @xcite . to this end, we, for the first time, propose a novel generalized personality measure using forced-choice testing and systematically examine how test format, temperature parameters, and language affect assessment outcomes. our study focuses on six representative llms: llama-3.1-8b-instruct (referred to as llama-3.1-8b for brevity) 1 , glm-4-9b-chat (glm-4-9b) 2 , gpt-3.5-turbo 3 , gpt-4o-2024-11-20 (gpt-4o) 4 , claude-3.5-sonnet 5 , and deepseek-v3 6 . the ipip-neo-120 scale provides five test sets with different item block designs. to analyze the effect of item order on responses, we develop a consistency index to measure model stability. we also relate the results to the ffm and investigate secondary dimensions, or facets, within each factor (e.g., imagination and artistic interest under openness) to deepen our understanding of personality traits. the contributions of our study are as follows: ## forced-choice scale assembly forced-choice tests, unlike likert scales, require items to be organized into blocks based on their level of social desirability. therefore, in this section, we will introduce three parts: dataset, social desirability evaluation, and block assembly in test. ## discussions we would like to discuss several interesting findings from our research: (1) the forced-choice test appears to require more cognitive resources from llms than the likert scale. the likert scale requires understanding and responding to a single question, whereas the forced-choice test requires comprehending an entire item block and then ranking the items. this complexity may encourage llms to provide more honest responses, and future research could provide additional evidence for d /odpde 6fruh e *ope f *37wxuer g *37r 6fruh h &odxghvrqqhw i 'hhsvhhn9 2shqqhvv &rqvflhqwlrxvqhvv ([wudyhuvlrq @xmath0gyhqwxurxvqhvv @xmath1fklhyhphqwvwulylqj &dxwlrxvqhvv 'xwlixoqhvv 2ughuolqhvv 6hoiglvflsolqh 6hoihiilfdf\ @xmath2vvhuwlyhqhvv &khhuixoqhvv ([flwhphqwvhhnlqj )ulhqgolqhvv *uhjdulrxvqhvv @xmath3qjhu @xmath4gyhqwxurxvqhvv @xmath5fklhyhphqwvwulylqj &dxwlrxvqhvv 'xwlixoqhvv 2ughuolqhvv 6hoiglvflsolqh 6hoihiilfdf\ @xmath6vvhuwlyhqhvv &khhuixoqhvv ([flwhphqwvhhnlqj )ulhqgolqhvv *uhjdulrxvqhvv @xmath7qjhu $q[lhw\ 'hsuhvvlrq ,pprghudwlrq 6hoifrqvflrxvqhvv 9xoqhudelolw\ )dfhw h &odxghvrqqhw i 'hhsvhhn9 6fruh figure 6: personality facet results of six llms at various temperature values. this hypothesis. (2) the number of scoring levels on the likert scale may influence the final results. our study used a 7-point scale, but investigating the impact of different scoring levels on outcomes would be an interesting area of future research. (3) we used a rank-3 block assembly for the forced-choice test. however, other methods, such as pick-2 (stark et al., 2005) and mole-4 (hontangas et al., 2015), could be tested to determine which block assembly technique is best suited for measuring psychological traits in llms. (4) our investigation into how language influences llms' personality assessment results revealed potential cross-cultural differences. for example, llms had lower neuroticism scores in chinese compared to other languages, which is consistent with previous cross-cultural research on human personality (mccrae and terracciano, 2005). thus, investigating cross-cultural differences in llms is an important area for future research. ## conclusion this study examines at the personality assessment and reliability of personality scales designed for human evaluation when applied to llms from three angles. first, we created a forced-choice test based on social desirability scores and established a standard testing procedure. second, we compared the effectiveness and reliability of two formats, likert scales and forced-choice tests, for measuring personality traits in llms. finally, we investigated how temperature and language affect personality assessment and reliability in llms. the results show that the forced-choice test better captures differences between llms across personality traits and is less affected by temperature parameters. furthermore, we discovered that different languages have both shared and distinct personality traits, implying that cultural factors may influence these differences.