Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood (2024)

Abstract: Human and model-generated texts can be distinguished by examining the magnitude of likelihood in language. However, it is becoming increasingly difficult as language model’s capabilities of generating human-like texts keep evolving. This study provides a new perspective by using the relative likelihood values instead of absolute ones, and extracting useful features from the spectrum-view of likelihood for the human-model text detection task. We propose a detection procedure with two classification methods, supervised and heuristic-based, respectively, which results in competitive performances with previous zero-shot detection methods and a new state-of-the-art on short-text detection. Our method can also reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies.

## introduction a recent focus in natural language generation is to develop effective methods of detecting modelgenerated texts from real human texts. this endeavor gives human users an opportunity to know whether a given text, e.g., an explanation, is either generated by language model or written by human, allowing them to decide how much trust to place in the explanation. the current most effective methods for this task utilize the likelihood information in text data (e.g.,see @xcite . more specifically, most existing methods rely on the observation that the absolute likelihood values of texts naturally distribute differently, depending on their sources. examples include the early work like gltr @xcite and more recent ones like detectgpt @xcite and fast-detectgpt @xcite . however, existing methods have the following limitations: first, most work estimates likelihood as a static property of language, but overlooks the fact that human language processing is a dynamic process during which the likelihood of information under processing is bound to certain linguistic and cognitive constraints. for example, the tradeoff between processing effort and likelihood of words @xcite , limited and selective context access @xcite , activation decay @xcite and so on. second, merely using the absolute likelihood values to distinguish generated and human texts is a tricky "cat and mouse game" -as models' capability of mimicking human language keeps growing, their productions would eventually become hardly distinguishable @xcite . third, current methods are not computationally economical, because most of them need to run at least one time of inference on text with a fairly large language model. we propose a human-model text detection approach that addresses the aforementioned issues and achieves better or comparable performance with existing methods. our approach extracts features from the spectrum view of relative likelihood scores of texts, to capture the dynamic changes of likelihood in language. these features are used to design two types of classifiers, a supervised learning-based one and a heuristic-based zero-shot one, both of which reach impressive performances. the core idea is to obtain the spectrum of likelihood using the fourier transform, which summarizes the complex patterns of likelihood change in time domain into a much more compact view that magnifies the subtle differences between different texts. it has a theoretical basis in psycholinguistic studies on surprisal (likelihood) distribution in natural language. further, our method is still effective when likelihood scores are estimated by naïve n-gram models, which places much less computa-tional cost. we name our approach fouriergpt, inspired by existing methods like detectgpt and fast-detectgpt. recent works develop more advanced statistics based on deeper insights into the distributional difference between human-created and modelgenerated languages in log-likelihood space. for example, @xcite finds that the probability distribution of model-generated text tends to lie under the areas of negative curvature of the log-likelihood function, and in contrast, human text tends not. based on this finding, they propose detectgpt, a zero-shot detection method that measures perturbation discrepency, the gap between an original text and its rewritten variant that maintains the same meaning. the assumption is that human text presents smaller gaps than model text. @xcite make substantial methodological improvements to detectgpt and propose fast-detectgpt by replacing the probability curvature with conditional probability curvature, which broadly improves the detection accuracy and greatly shorten the computational time. therefore, fast-detectgpt is the main state-of-the-art method compared with in this study. in nature, both detect-gpt and fast-detectgpt find an empirical threshold for the variance of absolute likelihood values, which depend on the choice of the inference model. two other likelihood-based methods are also compared within this study: normalized log-rank perturbation (npr) @xcite and divergence between multiple completions of a truncated passage (dna-gpt) @xcite . both rely on estimation of absolute likelihood to some extent. ## method the procedure of fouriergpt consists of three steps: 1) estimate and normalize likelihood scores; 2) carry out fourier transform to get the spectrum view; 3) conduct classification on the spectrum. the procedure is illustrated in figure 1 with an example. details of each step are described below. ## conclusions in this study, we propose a new text detection method fouriergpt, which draws information from the spectrum view of relative likelihood scores in language, as the basis for distinguishing human and model texts. our approach reaches better or competitive performances with state-ofthe-art methods on typical zero-shot detection tasks, and particularly better on short text detection tasks. our method has the following strengths: first, it utilizes the relative likelihood (z-scores) rather than absolute values as used by most previous methods, which means it can capture likelihood patterns in language that are less dependent on the expressiveness of the generation model. we consider this as an advantage because the lm's capability of producing more "likely" texts inevitably grows, and thus, detection methods relying on the absolute "thresholds" of likelihood will also eventually fail. secondly, we take a novel spectrum view of likelihood, which goes beyond the static view that simply aggregates likelihood at multiple time steps into a single value, but instead, characterizes the dynamic features how likelihood changes in time. this spectrum view draws inspiration from cognitive characteristics of language production revealed in the psycholinguistics literature, such as uid, periodicity of surprisal etc. the likehood spectrum can reflect subtle differences in human and model languages that are otherwise indetectable. thirdly, our method places a relatively low requirement on how accurate the likelihood scores need be estimated. a gpt-2 level model or even ngram model suffices to provide likelihood features to reach a decent detection performance. it suggests that how the likelihood of human language distributes in time is a subtle process, which may not be easily mimicked by language models trained via maximum likelihood estimation. llms that try hard to squeeze out the gap between every single prediction and ground-truth token may still lack the ability to produce human-like language. for future work, we will address the limitations with focus on: building stronger supervised classifiers by better utilizing the circularized spectrum; collecting larger datasets from broader domains and multiple languages; looking for more concrete linguistic cases (such as the "yes"/"no" example in section 5.1) to provide richer interpretations for the spectrum-view of likelihood.