Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits (2025)

Abstract: Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. In this study, we systematically evaluate a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. Analyses span three temporal levels—short daily text responses about current affect, text aggregated over two-weeks, and user-level text collected over two years—allowing us to examine how each model’s strengths align with the underlying stability of different constructs. The findings show that mental health signals emerge as the most accurately predicted dimensions (r=0.6) across all temporal scales. At the daily scale, smaller models like DeBERTa and HaRT often performed better, whereas, at longer scales or with greater context, larger model like Llama3-8B performed the best. Also, aggregating text over the entire study period yielded stronger correlations for outcomes, such as age and income. Overall, these results suggest the importance of selecting appropriate model architectures and temporal aggregation techniques based on the stability and nature of the target variable.

## introduction recently, language model representations (that is, embeddings) have shown strong promise in improving psychological assessments of mental health and well-being now approaching the theoretical upper limit in accuracy for some outcomes @xcite . however, their utility in different constructs is hitherto inconsistent. a systematic evaluation is yet to be performed to determine what types of psychological attributes can best be captured in language @xcite , and by which lm. psychological variables differ by many factors, fundamentally including (a) their stability -from constructs are categorized into states (highly variable, e.g., mood), dispositions (moderately stable, e.g., stress), and traits (highly stable, e.g., personality). this figure underscores the study's focus on aligning lm architectures with psychological stability to enhance predictive performance across temporal granularities. in this study, we systematically evaluate the effectiveness of transformer-based lm representations for capturing human psychological states and traits from textual data. unlike prior work that has predominantly focused on inferring the mental states of conversational participants @xcite , our approach centers on self-expressed narratives-treating participants as the authors of their own psychological profiles. by leveraging a unique dataset consisting of language collected in bursts over daily intervals -ecological momentary assessments (ema) -and aggregated over longer periods (wave-level) across a two-year span, we directly compare the capability of various llm-based embeddings to predict standardized psychological scores. these scores span multiple domains, including affect/emotion, personality, mental health, sociodemographics, and health behaviors, thereby offering a comprehensive assessment of language models in psychological evaluation @xcite . our key contributions are: (1) we provide a systematic comparison of different llm representations' ability to capture 43 psychological variables; ## related work understanding psychological states and traits through language has been a longstanding focus of both psychology and computational linguistics. the evolution of analyzing psychological states through language has progressed from simple lexicon-based tools such as liwc @xcite to more sophisticated approaches using distributed word embeddings @xcite and ultimately to advanced contextualized models for capturing personality @xcite , emotions (al-twairesh, 2021), and substance use @xcite . concurrently, researchers have examined stable individual differences by initially employing bag-of-words and topic modeling methods to predict personality traits, gender, and socioeconomic status @xcite . modern transformer-based architectures have enhanced capturing the nuanced relationships between words and their broader context, enabling more accurate psychological modeling through language analysis @xcite . encoder-based models have emerged as a popular choice for modeling psychological constructs within natural language processing (nlp) since their bidirectional nature helps capture richer semantics @xcite , with significantly fewer parameters than decoder-only counterparts. they have been applied to identify various psychological factors ranging from mental health @xcite @xcite , cognitive distortions and thinking pat-terns @xcite , dissonance and belief changes @xcite @xcite , affective factors @xcite @xcite @xcite , substance behaviors @xcite to stable factors like personality @xcite and demographic attributes @xcite . however, previous studies have also demonstrated the effectiveness of other types of model architectures -human-level lms like hart @xcite in capturing subtle mood changes, valence, arousal, empathy and distress in text @xcite ; auto-regressive lms like gpt-2 for assessing stable traits like personality @xcite ; prompting-based methods have also been successfully applied to identify language patterns indicative of mental health, personality factors @xcite @xcite . given the high degree of variability of architectures in state-of-the-art mental health and psychology-related tasks, in this work we specifically focus on evaluating the modern llm architectures. unlike prior studies, we assess llms performance across multiple temporal granularities (daily messages, biweekly waves, and aggregated user histories) using a longitudinal dataset of self-described feelings, which are particularly suited for authors' own descriptions, as feelings are inherently subjective. ## dataset the dataset for this study was collected by recruiting service industry workers in the united states @xcite . the participant population predominantly comprises middle-aged individuals who identify as female, representing approximately 75% of the participants. their ages range from 18 to 68 years, with a mean age of 35 years (standard deviation: 9.47). these individuals were recruited through professional and online groups within the united states and are known to be at high risk for excessive alcohol use @xcite . this demographic choice is particularly relevant for studies focusing on occupational stressors prevalent within the service industry. over a two-year period, data were collected in six 14-day "waves" via a custom smartphone application. the participants were prompted three times a day for ecological momentary assessments (emas), where they provided a textual response (with minimum of 200 characters) in english to the prompt eliciting the current affective state: "please describe in 2 to 3 sentences how you are currently feeling." (see table a1 in the appendix for additional illustrative ema responses). each wave thus yielded up to 42 responses per participant, totaling a potential 42 x 6 = 252 responses across all six waves. to ensure data quality, only individuals who completed at least two waves with a minimum of two responses per wave were included, resulting in 10,108 emas from 120 participants across 406 total user waves. (see ## methodology this research explores three hierarchical levels of data analysis: the message-level, which focuses on individual ema responses; the wave-level, which involves analysis of aggregated ema responses over 14-day periods; and the user-level, which aggregates ema data across all waves to examine long-term psychological trends. ## conclusion this study systematically evaluated the capabilities of many transformer-based language models, spanning bert-style autoencoders, as well as generative large lms, for capturing human factors across different levels of temporal stability-low, medium, and high as well as different psychological dimensions. the findings reveal that model performance is highly influenced by the temporal granularity of the data, the stability of the outcomes, and the constructs being modeled. while aggregation strategies proved instrumental for enhancing predictive reliability for stable constructs (traits), low-stability constructs that undergo a lot of dynamic fluctuations on the daily might not be best represented through averages over time, an effect we specifically observe for some states. additionally, we introduced a framework that determines the preferred temporal granularities at which these constructs should be analyzed. this framework not only aids improving models of psychological constructs but it also has practical implications for data collection design in future studies. by identifying the optimal data collection frequency for such experiments, it offers the potential to eliminate the need for costly daily surveys when evaluating llms' capabilities in psychological assessment. together, these insights emphasize the importance of tailoring model selection, data aggregation strategies, and experimental design to align with the unique temporal characteristics and stability of psychological constructs, paving the way for more efficient and reliable llm-based approaches to psychological evaluation.