GPT -who: An Information Density-based Machine-Generated Text Detector (2024)

Abstract: The Uniform Information Density (UID) principle posits that humans prefer to spread information evenly during language production. We examine if this UID principle can help capture differences between Large Language Models (LLMs)-generated and human-generated texts. We propose GPT-who, the first psycholinguistically-inspired domain-agnostic statistical detector. This detector employs UID-based featuresto model the unique statistical signature of each LLM and human author for accurate detection. We evaluate our method using 4 large-scale benchmark datasets and find that GPT-who outperforms state-of-the-art detectors (both statistical- & non-statistical) such as GLTR, GPTZero, DetectGPT, OpenAI detector, and ZeroGPT by over 20% across domains.In addition to better performance, it is computationally inexpensive and utilizes an interpretable representation of text articles. We find that GPT-who can distinguish texts generated by very sophisticated LLMs, even when the overlying text is indiscernible.UID-based measures for all datasets and code are available at https://github.com/saranya-venkatraman/gpt-who.

## introduction the recent ubiquity of large language models (llms) has led to more assessments of their potential risks. these risks include its capability to generate misinformation @xcite , memorized content @xcite , plagiarized content @xcite , toxic speech @xcite , and hallucinated content @xcite . to mitigate these issues, researchers have proposed automatic and human-based approaches to distinguish llm-generated texts (i.e., machinegenerated) from human-written texts @xcite @xcite . billionaire investor bill ackman recently suffered a massive blow, as his holdings in valeant pharmaceuticals and herbalife plummeted. during pershing square's second quarter call, ackman said that results for his various funds fell between 14 percent and 18 percent. automatically detecting machine-generated texts occurs in two settings-turing test (tt) which is the binary detection of human vs. machine; and authorship attribution (aa) which is the multiclass detection of human vs. several machines (e.g., gpt-3.5 vs. llama vs. falcon) @xcite . while the tt problem is more rigorously studied, due to the wide usage of different llms, in the future, it will be imperative to build models for the aa tasks to determine which llms are more likely to be misused. this knowledge will be needed by policymakers when they inevitably institute laws to guard the usage of llms. to that end, we propose gpt-who, the first psycholinguistically-inspired supervised domainagnostic task-independent multi-class statisticalbased detector. gpt-who calculates interpretable uniform information density (uid) based features from the statistical distribution of a piece of text and automatically learns the threshold (using logistic regression) between different authors. such features are calculated using the surprisals of tokens in a text, for example, the variance of surprisals, the mean, and so on (elaborated in section 3.1). to showcase the detection capabilities of gpt-who, we use 4 large llm benchmark datasets: turing-bench @xcite , gpabenchmark @xcite , argugpt @xcite , and deepfake text in-the-wild @xcite . we find that gpt-who outperforms state-of-theart statistical detectors and is at par with task and domain-specific fine-tuned lms. this performative gain is consistent across benchmark datasets, types of llms, writing tasks, and domains. in addition to improved detection performance, gpt-who is computationally inexpensive as it eliminates the need for any llm fine-tuning. it utilizes a freely available off-the-shelf lm to compute token probabilities, followed by logistic regression using a small set of carefully crafted and theoretically motivated uid features. gpt-who also provides a means to interpret and understand its prediction behaviors due to the rich feature space it learns from. uid-based features enable observable distinctions in the surprisal patterns of texts, which help in understanding gpt-who's decisionmaking on authorship (figure 1 ). we also analyze the uid feature distributions of different llms and human-generated texts for the 4 datasets and find that humans distribute information more unevenly and diversely than models. in addition, uid features are reflective of differences in llm architectures or families such that models that share architectures have similar uid distributions within but not outside their category. we find that uid-based features are a consistent predictor of authorship. even when there are no glaring differences between uniform and non-uniform text, the differences in uid distributions are easily detectable and a powerful predictor of authorship, since they capture patterns that go beyond the lexical, semantic, or syntactic properties of text. our work indicates that psycholinguistically-inspired tools can hold their ground in the age of llms and a simpler theoretically-motivated approach can outperform complex and expensive uninterpretable black-box approaches for machine text detection. ## our proposal: gpt-who we propose a psycholinguistically-motivated statistical-based machine-generated text detector gpt-who that uses a gpt-based language model to predict who the author of an article is. gptwho works by exploiting a densely informationrich feature space motivated by the uid principle. uid-based representations are sensitive to intricate "fluctuations" as well as "smoothness" in the text. specifically, operationalizations of uid are aimed at capturing the evenness or smoothness of the distribution of surprisal per linguistic unit (tokens, words), as stated by the uid principle. for example, in figure 2 , we show sequences of tokens that correspond to the highest and lowest uid score spans within an article. here, the differences between the two segments of texts might not be obvious at the linguistic level to a reader, but when mapped to their surprisal distributions, the two segments have noticeably distinct surprisal spreads as can be seen by the peaks and troughs i.e. variance of token surprisals along the y-axis about the mean (dotted line). most approximations of this notion of "smoothness" of information spread and uid, thus, formulate it as the variance of surprisal or as a measure of the difference of surprisals between consecutive linguistic units @xcite @xcite . in measuring the distribution of surprisal of tokens, uid-based features can capture and amplify subtle information distribution patterns that constitute distinct information profiles of authors. using just an off-the-shelf language model to calculate uid-based features, gpt-who learns to predict authorship by means of a simple classifier using uid representations. in addition, as these features can be directly mapped to their linguistic token equivalents, gpt-who offers a more interpretable representation of its detection behavior, unlike current black-box statistical detectors, as illustrated in figure 2 . the use of a psycholinguistically motivated representation also enables us to better interpret the resulting representation space. it can capture "every coin has two sides "and it is also the case to the problem whether children should be taught to compete or to cooperate ………. it is better to have a child who is competitive and cooperative at the same time, rather than having him compete and cooperate at different times in his life. surprisal distributions indicative of and commonly occurring in human-written or machine-generated text. gpt-who is one of the first text detectors that focus on informing a simple classifier with theoretically motivated and intuitive features, as it only requires a fixed-length uid-based representation of length 44 and learns to predict authorship based on just these features, without the need for the full text or any lm fine-tuning in the process (see gpt-who's complete pipeline in figure 3 ). ## empirical validation we use @xcite 's implementation of uid-based scores foot_0 and use the publicly available off-the-shelf pre-trained gpt2-xl language model foot_1 to obtain conditional probabilities. for all our experiments, we calculate the uid features for the publically released train and test splits of all ## discussion we turn to the uid principle, which states that humans prefer to spread information evenly in language, to automatically extract features that measure the spread and flow of information content or surprisal in texts. our uid-based features are formulated to capture how surprisal is distributed in an article as they measure the local and global variance, mean, and most uniform and non-uniform segments of a text. this rich and succinct representation space drives the predictive capability of our proposed detector and the interpretability of its representations. analysis of this feature space reveals that human-written text tends to be more nonuniform in comparison to machine-generated text. hence, machines seem to be spreading information more evenly or smoothly than humans who are more likely to have fluctuations in their surprisal distributions. however, this finding does not imply that humans are not producing uniform text. it is important to note that our work cannot provide support for or refute the uid hypothesis which comes from psycholinguistic studies such as those in section 2.1. our work shows that, given our operationalization of uid based on prior works, machine text is relatively more uniform than human-written text. while this might seem contradictory to uid theory, it does not still disprove that humans are uniform in their language production. we conjecture that this unexpected finding is because we use gpt-2's probability distribution to calculate surprisal, which is potentially a poor approximation of the "human" probability distribution. a closer-to-human probability distribution might (or might not) show humans to be more uniform than machines, though this determination is not within the scope of this work. it is crucial to note that uniformity is relative, and while machines are more uniform under this operationalization, it would still be true that human text is uniform as per the human's probability distribution (that we do not have access to and can only approximate using some lm distribution, for example, gpt2-xl in our case). uid theory does not make any predictions on where machine-generated text might lie in the uniform to non-uniform spectrum but only indicates that humans are arranging utterances evenly as per their own language distribution. irrespective of its alignment with what theory suggests, we find that the uid-based features are very useful in distinguishing authors, which is the focus of this work. this is an important consideration and helps disentangle the utility of uidinspired features from the cognitive plausibility of those feature calculations or uid approximations. thus, this operationalization of uid does not imply that humans are "less human" or machines are "more human" as it is an approximation of a theory that states that humans are uniform as per their language distribution. it does not have any further implications for machine-generated text and is unable to predict what happens in the case of machines. we find that uid-based features can capture differences between text generated by not only humans and models but also capture differences between multiple models and lm families. our main contribution is a psycholinguisticallyinspired domain-agnostic statistical-based machinegenerated text detector, gpt-who, that: while our detector may not significantly outperform fine-tuned transformers-based models, it is essential to highlight its independence from finetuning, offering nearly comparable performance at significantly lower computational costs and remains one of the only statistical-based detectors that can operate in multi-author settings beyond the turing test. these findings indicate that approaches rooted in psycholinguistic theories that delineate indicators of "human-like" language use hold enormous and untapped potential in tackling the fast catapulting and ever-changing llm landscape. this work has implications for cognitively plausible and explainable solutions to complex challenges arising from ever-growing automated text generators. ## conclusion we propose gpt-who, a statistical-based machinegenerated text detector that utilizes features inspired by the uniform information density (uid) principle rooted in the observation that humans tend to evenly distribute information or surprisal in language. we leverage uid-based features, including variance of surprisals and minimally/maximally uniform sub-sequences extracted from the surprisal sequence generated by an off-the-shelf lm. we demonstrate that these features are highly effective in discerning machine-generated text from humangenerated text as they capture nuances in how models and humans distribute surprisal in their texts. our findings have implications for enhanced text authenticity assessment.