P sy TE x: A Knowledge-Guided Approach to Refining Text for Psychological Analysis (2025)

Abstract: LLMs are increasingly applied for tasks requiring deep interpretive abilities and psychological insights, such as identity profiling, mental health diagnostics, personalized content curation, and human resource management. However, their performance in these tasks remains inconsistent, as these characteristics are not explicitly perceptible in the text. To address this challenge, this paper introduces a novel protocol called the “Psychological Text Extraction and Refinement Framework (PsyTEx)” that leverages LLMs to isolate and amplify psychologically informative segments and evaluate LLM proficiency in interpreting complex psychological constructs from text. Using personality recognition as a case study, our extensive evaluation of five SOTA LLMs across two personality models (Big Five and Dark Triad) and two assessment levels (detection and prediction) highlights significant limitations in LLM’s ability to accurately interpret psychological traits. However, our findings show that LLMs, when used within the PsyTEx protocol, can effectively extract relevant information that closely aligns with psychological expectations, offering a structured approach to support future advancements in modeling, taxonomy construction, and text-based psychological evaluations.

## introduction large language models (llms) are transforming the field of natural language processing (nlp), performing remarkably as linguistic tools skilled in language manipulation, reasoning, explanation, and information extraction. equipped with billions of parameters, these models excel at processing and retaining vast amounts of information, reaching state-of-the-art (sota) performance in a variety of tasks including text summarization @xcite , question answering @xcite @xcite , and natural language inference (nli) @xcite @xcite , etc (yang et al. @xcite . which are evaluated against standard benchmarks designed to measure their zero-shot and few-shot capabilities in language understanding and information extraction @xcite @xcite @xcite while near-perfect performance in these tasks showcases llms' ability to "understand" language, incorporating both semantic and contextual knowledge, standard benchmarks do not typically evaluate their "interpretive" capabilities. assuming that llms can handle psychological evaluations and human categorization, preliminary studies using zero-shot prompting for tasks like authorship verification, author attribution, and psychological profiling, including the detection of implicit social signals such as sarcasm, personality, and implicit sentiment, reveal that their performance frequently borders on random chance @xcite @xcite . for example, consider a scenario (figure 1 ) where chatgpt assessed the personality trait of narcissism from a human-authored essay. it incorrectly identified the highlighted sen-tence as indicative of narcissism and assigned the essay a high score, despite its actual low score. while a human might see the sentence as an expression of gratitude, a behavior typically inconsistent with narcissism, chatgpt misinterprets it, incorrectly identifying it as evidence of the trait. these observations, alongside the findings from preliminary studies, suggest that current llms may not possess the required capabilities to effectively interpret nuanced information from text. this shortcoming is particularly critical given the potential of llms to revolutionize areas such as identity profiling, personalized advertising, mental health assessments, and human resources. highlighting the example of personality recognition where llms have shown notably poor performance, this work seeks to answer the question "can llms effectively interpret psychological characteristics from text?". most advanced approaches using llms for this purpose assume that llms can assess these cognitive characteristics and that their effectiveness can be enhanced by curating specialized prompts @xcite @xcite . several techniques have been proposed in recent literature, including zero-shot prompting, chain-of-thought (cot) prompting, and many specialized prompting methods. however, the findings remain inconsistent. while some works indicate that llms are not yet suitable for direct use as psychological evaluation tools, others present contradictory results @xcite . key factors contributing to this disparity are the reliance on lexical models for labeling that exhibit weak correlations with actual personality scores, synthetic datasets generated by llms, and questionnaire-based evaluations, where llms are artificially induced with personality traits and then assessed on their responses to personality question-naires @xcite @xcite . while effective for evaluating ai agents and chatbots, these methods lack ground truth human data, risking overestimation of llm capabilities and poor generalization to real-world populations. in contrast, our work evaluates llms on human-authored text, ensuring assessments align with natural language patterns and reinforcing both the validity and applicability of our findings for real-world psychological analysis. more importantly, existing approaches do not assess whether llms possess true "interpretive" capabilities or merely rely on superficial linguistic patterns for personality assessment. several studies suggest that llms can enhance their outputs through self-refinement, where models assess their own responses or follow self-generated checklists for structured reasoning @xcite . if llms can apply similar internal evaluation mechanisms to psychological constructs, they may be capable of more nuanced personality assessment. however, this remains largely unexplored. thus, it is crucial to deconstruct how llms might analyze psychological constructs from text to assess their interpretive capabilities. to address this, we introduce a novel protocol named psychological text extraction and refinement framework (psytex) to simulate the process by which an llm evaluates psychological characteristics. as depicted in figure 2 , this process comprehensively probes the llm's domain knowledge and its ability to extract application-specific information and integrates evaluation capabilities using the standard prompting protocol in a standalone yet explainable step-by-step fashion. furthermore, this framework is highly adaptable and can be seamlessly extended to incorporate other prompting techniques while maintaining the same foundational framework. this work makes the following contributions foot_0 : ## methodology the methodology for psytex is structured in two main steps: knowledge extraction and relevant information extraction, followed by a systematic protocol for assessing the interpretive ability of llms. ## results in this section, we evaluate the performance of llms for personality recognition under zero-shot settings. the two datasets facilitate coarse personality detection and fine-grained personality prediction. personality detection involves binary classification to differentiate between "high" or "low" trait categories, while personality prediction involves regression analysis to estimate precise trait scores. ## discussions until now, we assumed that the text segments tagged by llms are personality-relevant and contain meaningful personality cues and that the presence or absence of these segments should impact subsequent evaluations. we now shift our focus to critically examining whether the extracted text is genuinely distinct from irrelevant text and truly reflects personalityrelevant content. this investigation is essential to validate the usability and applicability of psytex in effectively isolating and identifying personalityrelevant information. to this end, we explore two key questions: firstly, are there significant linguistic differences between relevant and trivial texts? and, secondly, does the llm-extracted (tagged) text genuinely reflect personality-relevant content? this section delves into these critical questions. ## future works we plan to utilize the trait-relevant information identified in the psytex framework for downstream personality assessment in two primary ways. firstly, integrating attention mechanisms into existing personality detection models to focus on psytex-refined text segments. these models can then be fine-tuned using existing personality detection datasets for effective assessment. however, a key limitation of this approach is the potential lack of representative data across various contexts, such as different topics, genres, or domains. a strategy to overcome this limitation involves empowering llms to produce psychology-relevant insights. efforts in this direction have included the development of taxonomies through expert-llm teaming, categorizing information identified by llms into actionable insights @xcite . this method uses the precision of taxonomies with the llm's ability to detect trait-relevant text instances, refined by expert analysis. we aim to refine and expand these ideas in our future work. ## conclusion in this paper, we explore the question: can llms effectively "interpret" psychological characteristics from text? to this end, we introduce a novel evaluation protocol called "psychological text extraction and refinement framework" (psytex), designed to assess the interpretive capabilities of llms for human categorization tasks, specifically for text-based personality recognition. using the simplest and most widely used llmbased zero-shot personality evaluation, we first examine whether llms possess deep interpretive abilities. our analysis of five sota llms and two personality models -big five and dark triad, revealed that llms frequently produce random and inconsistent outcomes regardless of the presence or absence of personality-relevant text, suggesting a lack of deep interpretive abilities. this was particularly evident in their struggle with more complex task of personality prediction and traits such as dark triad that require a nuanced understanding that goes beyond basic semantic processing. these results indicate that specifically tailored benchmarks are needed to evaluate llm's interpretive abilities effectively. these benchmarks could significantly boost the efficacy of llms in areas such as mental health diagnosis, where a precise grasp of human psychology is essential. while llms cannot be directly used to eval-uate personality traits from the human-authored text in a zero-shot setting, our proposed framework enables them to extract personality-relevant information segments from the text. our findings show that psytex-refined text segments exhibit linguistic separability and capture meaningful patterns that align with personality psychology literature, validating its potential for enhancing personality assessment methodologies. moreover, psytex provides a foundation for downstream applications such as psychological modeling and taxonomy development, making it a valuable framework for text-based psychological analysis.