The Consistent Lack of Variance of Psychological Factors Expressed by LLM s and Spambots (2025)

Abstract: In recent years, the proliferation of chatbots like ChatGPT and Claude has led to an increasing volume of AI-generated text. While the text itself is convincingly coherent and human-like, the variety of expressed of human attributes may still be limited. Using theoretical individual differences, the fundamental psychological traits which distinguish people, this study reveals a distinctive characteristic of such content: AI-generations exhibit remarkably limited variation in inferrable psychological traits compared to human-authored texts. We present a review and study across multiple datasets spanning various domains. We find that AI-generated text consistently models the authorship of an “average” human with such little variation that, on aggregate, it is clearly distinguishable from human-written texts using unsupervised methods (i.e., without using ground truth labels). Our results show that (1) fundamental human traits are able to accurately distinguish human- and machine-generated text and (2) current generation capabilities fail to capture a diverse range of human traits

## introduction modern large language models (llms; e.g., llama and gpt4) can produce coherent, grammatically sound, and human-like text. these models can also take on human personas @xcite , reproduce human-like biases @xcite , and may be able to pass a turing test @xcite . as such, these models are being deployed in real-world situations, such as tutoring @xcite , serving as synthetic patients for training therapists @xcite , and replacing humans in crowdsourcing tasks @xcite . these advances have also driven an increase in machine-generated text. while llms can be used * equal contribution figure 1 : humans express a range of psychological traits (or human factors) through language. while llms and spambots produce fluent text, the psychological traits they express tend to average out across all dimensions, which is uncharacteristic of humans. for innocuous tasks (generating a cover letter for an employment application) they can also be used with malicious intent, such as for phishing attacks, spamming, and disinformation @xcite . thus, machine-generated text presents a significant problem for cybersecurity and other social and political contexts. despite their human-like generations, there is mounting evidence that llms express a limited range of humanness. llms have been shown to reflect western norms @xcite , lean politically left @xcite , and fail to reflect opinions of many sociodemographic groups @xcite . in particular, these models are known to generate text according to the average of their training data (i.e., predict the most probable next token), and thus reflect average values and beliefs @xcite . against this backdrop, the current work leverages the limited diversity in human-like expressions to identify machine-generated text. this is done through the lens of individual differences (which we call human factors), or fundamental psychological traits (such as personality) known to distinguish people and their outcomes @xcite ., gpt4: movie reviews gpt4o, llama3.1: academic essays gpt 3.5: hotel reviews november 2022 march 2023 -present gpt-2: movie reviews february 2019 gpt4: books social media spambots pre -2019 human ai @xmath0.48** @xmath1.24** @xmath2.28** @xmath3.16** @xmath4.19** figure 2: kernel density estimate plot for agreeableness, a big-5 personality trait across various nlp models over the years over multiple domains. although recent developments seem to have improved ai-generated texts' capacity to display more variability in agreeableness trait, ai-generated texts are still distinguishable from human-generated texts when analyzing multiple such human traits together. ks: two-sample kolmogorov-smirnov statistic for a two-sided test; ** : p-@xmath5.001. ## related work recent llm research has extensively focused on distinguishing machine-generated text from human writing. some studies have considered linguistic patterns such as sentence lengths, lexical variations, and richness of vocabulary @xcite . conversely, some prior works focused on emotions @xcite , cultural variations @xcite , and psychological factors such as personality @xcite , and psychometric inventories @xcite . llms have been shown to exhibit an ecological fallacy by treating individual text sequences as independent samples rather than considering the broader context of authorship @xcite , resulting in an averaged representation of writing styles @xcite and personalities @xcite from their training data. prior work has leveraged this lack of variance in llms-generated text in tasks like authorship attribution in the realm of human versus machine generated texts @xcite @xcite , differentiating human versus bot language @xcite . in this study, we further build on past works to show that psychological features can help identify machine generated text. ## data we estimate human factors across four datasets of human/machine text, which span a range of domains and llms. two of the datasets have been used in past work incorporating human factors (summarized here) and the remaining two applications are novel. all datasets used were collected from previous works and our contribution is the application of our methods to these domains. twitter spambots this dataset consists of 2,913 genuine (human) twitter accounts and 2,913 spambots originally collected by @xcite and analyzed for human traits by @xcite . 1 these spambots are known as social spambots and differ from traditional bots in that they intentionally try to emulate real humans @xcite . 2 hotel reviews this dataset consists of 400 human and 400 machine generated hotel reviews from 1 unsupervised classification results using human factors can be found in @xcite this dataset is included here to summarize previous work and show how the human factors of machine generated text has evolved over time. the human reviews were collected from tripadvisor and the machine reviews were generated by gpt4. the human dataset was collected by @xcite and all texts were analyzed for human traits by @xcite . academic essays this dataset consists of 3,722 english academic essays and 2,369 arabic academic essays written by humans and machines @xcite . for machine language, seven different open and closed llms were used. for this dataset, both human and llm english essays were provided alongside arabic essays. before running human trait inference all arabic essays were translated into english using the google translate api. raid this is a benchmark dataset for machinegenerated text detection, which includes 6 million generations across 11 models and 11 domains @xcite . because our human factor models were trained on social media data, we dropped domains that we believed were least similar to social media language: recipes, poetry, and code. we also dropped non-english texts. due to space limitations, we only consider gpt4, with a greedy decoding strategy and no repetition penalty. ## methods we proceed in three steps: (1) estimate human factors from text, (2) visualize the human factor distributions, and (3) cluster the human factors using unsupervised methods (i.e., clustering with no ground truth) to assign human/machine labels. the dlatk package @xcite is used for both human factor estimation and clustering. ## conclusions we have shown that individual differences -fundamental psychological traits that distinguish humans -can also distinguish humans from machines. unlike human traits, the values for these dimensions are so consistently average for machines, that it is unusual for a person to have them. specifically, across multiple bots and generative llms, datasets, and domains, machine-generated text exhibits smaller variations in expressed human factors than human-generated text. this enables unsupervised classifiers using a handful of interpretable features (those that can theoretically distinguish people) to distinguish bots from people well beyond baseline models.these results also give insight into how current generation methods, such as llms aligned with rlhf, generate human-like text that nonetheless lacks a diverse range of human traits. this dovetails with a growing line of research showing that llms fail to generate diverse cultural values, beliefs, and attitudes @xcite . these weaknesses underscore limitations in training data quality and generation methods as well as the opportunities for integrating psychological theories of individual differences to improve llms.