The Two Paradigms of LLM Detection: Authorship Attribution vs. Authorship Verification (2025)

Abstract: The detection of texts generated by LLMs has quickly become an important research problem. Many supervised and zero-shot detectors have already been proposed, yet their effectiveness and precision remain disputed. Current research therefore focuses on making detectors robust against domain shifts and on building corresponding benchmarks. In this paper, we show that the actual limitations hindering progress in LLM detection lie elsewhere: LLM detection is often implicitly modeled as an authorship attribution task, while its true nature is that of authorship verification. We systematically analyze the current research with respect to this misunderstanding, conduct an in-depth comparative analysis of the benchmarks, and validate our claim using state-of-the-art LLM detectors. Our contributions open the realm of authorship analysis technology for understanding and tackling the problem of LLM detection.

## introduction generative ai is everywhere. from writing assistants to the generation of complete documents, texts curated or written by large language models (llms) can now be found in practically all social structures: llms are used in online media @xcite , in education (adeshola and adepoju, 2024), for scientific publications @xcite @xcite , for drafting @xcite , and more. owing to the many risks of llm text generation @xcite , research into llm detectors has sparked great interest. so far, it seems difficult to ascertain the realworld effectiveness of current detectors. on the one hand, new detectors quite regularly "beat" the latest benchmarks @xcite @xcite , leading us to believe the problem is easily solved, at least within fixed domains. on the other hand, serious concerns are voiced about the readiness of the technology. perhaps most strikingly, openai shut down their own llm detector just six months after its launch, citing a lack of accuracy @xcite . in this paper, we show that the conflicting reports of success or failure are due to a mix-up of two paradigms of the related research field of authorship analytics: authorship attribution and authorship verification @xcite . as the authorship analytics and llm detection communities have developed almost independently until now, neither could learn from the other. framed as authorship attribution, llm detection considers llms and humans as two collective "authors," each possessing their own distinct writing style. llm detection is thereby cast as a closedset binary classification problem, relying on both classes to be sufficiently discriminative. we show that llm detection is more realistically cast as an authorship verification problem-i.e., as an openset one-class classification problem @xcite . however, our literature review and data analyses show that llm detectors are often developed with attribution in mind, but are evaluated under much broader assumptions on inadequate benchmarks, leading to unnecessary type ii errors in evaluations. our main contributions are as follows: (1) we reconcile authorship analytics and show that llm detection is an authorship verification task (section 3). (2) we conduct a corpus analysis on how llm text differs from human text today (section 4). humans as detectors humans' ability to recognize llm text depends on their prior experience. as llms advanced, the basic heuristics applied by humans became inaccurate @xcite @xcite so that llm text can even be perceived more human-like than human text @xcite . tools like gltr @xcite and scarecrow @xcite can significantly improve average human performance, but @xcite found that experienced chatgpt users have near-perfect detection rates, outperforming automatic detectors. supervised llm detection a popular method for llm detection is to train supervised binary classifiers. an early approach was openai's robertabased gpt-2 detector @xcite . this, as well as more recent approaches, are usually found to be highly accurate in-domain, but do not generalize across domains, nor to newer llm families @xcite @xcite @xcite , nor to language learner texts @xcite . fine-tuned versions of bert, t5, or smaller llms are frequently cited as state-of-the-art detectors in the literature @xcite , e.g., ghostbuster @xcite and gpt-sentinel @xcite . besides classifying entire documents, more fine-grained approaches are investigated as well @xcite . detectors with fewer parameters tend to generalize better to unseen llm families @xcite , and even symbolic approaches can achieve state-of-the-art results @xcite . zero-shot llm detection many zero-shot detectors use reference llms to measure token perplexities: detectgpt @xcite measures perplexity degradation due to text perturbations. likely unbeknownst to the authors, it bears a resemblance to @xcite unmasking. fast-detectgpt @xcite reduces detectgpt's computational costs by estimating perturbations, and detectllm @xcite adapts it to use log ranks for accuracy and speed. llmdet @xcite calculates perplexity from n-gram profiles for each llm. binoculars @xcite addresses the limited generalization capabilities of previous models by calibrating cross-entropy scores with a second llm. other approaches use gans @xcite , token compression ratios @xcite )similar to compression models in authorship analytics @xcite )-, they prompt llms directly @xcite , or measure the similarity of responses to reverse-engineered queries @xcite or paraphrasing prompts @xcite with reference texts. robustness besides the mentioned domain adaptation issues, detectors are also vulnerable to adversarial attacks. paraphrasing attacks @xcite or basic synonym replacements @xcite are effec-tive. search for adversarial (soft) prompts @xcite or downstream "humanizer" models @xcite can further reduce the detection accuracy. these attacks against detectors are very similar to authorship obfuscation approaches @xcite , a fact outlined only by @xcite so far. it is known that even manual style obfuscation by untrained writers is quite effective @xcite . however, obfuscations leave detectable traces @xcite , which have been observed for llm obfuscators, too @xcite . ## llm detection as a verification problem llm detection, albeit a relatively new task, shares many theoretical foundations with the analysis of human authorship and writing style @xcite @xcite . our work aims to bridge the gap between the two communities. ## properties of llm text we know that llm attribution is considered (relatively) effective today, so we use this section to illustrate how texts from llms and humans differ at the surface level. we highlight certain properties of recent models that render them increasingly human-like. these observations align with our theory from the previous section that llm detection is becoming an authorship verification task. however, more research will be necessary to compile further evidence in support of our hypothesis. ## llm detector evaluation we analyze the behavior of three state-of-the-art detector models: (1) a re-implementation of the binoculars (hans et al., 2024) zero-shot detector, (2) a set of svms trained on 800 tf-idf word unigrams, and (3) a lora-fine-tuned mistral-7b. 3 all three were used in the top submissions to pan'24 @xcite . we trained multiple svms on several balanced splits of the pan dataset, each containing randomly sampled texts from humans and one llm family. we tested the models both in-domain, on identically constructed holdout splits, and outof-domain, on mixed random samples from all datasets (with an equal number of samples from small and large datasets). we trained two versions of the mistral model for binary classification on un- model family (test) claude gpt-3 gpt-4 llama2 mistral o1 (a) svm tested on all datasets false positive rate false negative rate g p t -3 g p t -4 g e m in i l la m a 2 m is t r a l q w e n o 1 p a l m 2 g p t -3 g p t -4 g e m in i l la m a 2 m is t r a l q w e n o 1 p a l m 2 model family (training) model family (test) gemini gpt-3 gpt-4 llama2 mistral o1 qwen palm2 (b) svm tested on pan g p t -3 g p t -4 g e m in i l la m a 2 m is t r a l q w e n o 1 p a l m 2 model family (training) l o w f n r l o w f p r a c c u r a c y threshold (d) mistral tested on pan and raid false positive rate false negative rate test dataset training dataset hd gb essays gb reuters m4 mage raid (no attacks) raid pan p a n r a i d p a n r a i d (c) binoculars t. on pan false negative rate false negative rate figure 4: (a) fnr and fpr of svm-based detectors with tf-idf features trained on class-balanced samples of individual model families from the pan dataset and tested on mixed samples from all other datasets. fpr variations within columns are due to different per-model sample sizes (ranging from 60-8,576). (b) fpr of the same detectors tested on a pan holdout set. (d) fpr of binoculars on pan. (d) fnr and fpr of mistral-7b, fine-tuned for llm detection on pan or raid. we see a similar picture for the mixed dataset but at worse detection rates. detectors trained on the same model fare slightly better, but o1 is practically not recognized by any non-o1 detector. fprs (figure 4a ) are 10-20 % for all but the o1 detector, which is less likely to misclassify human text than any other detector (t(38) = 4.4, @xmath0.001, @xmath1.1). however, the lower fpr translates into a higher fnr (t(38) = 2.2, @xmath2.034, @xmath3.1). binoculars manages to keep its fpr close to zero on all datasets, but suffers in terms of recall (tpr), especially for gpt-3 and 4 (figure 4c ). the detection fails completely on the o1 texts at both operating points (accuracy and low fpr). we tried a third threshold ("low fnr") of 0.97, which brings the fnr down to zero at the cost of a 40 % fpr. the svm-based detectors are biased towards the human class with lower fpr. the mistral-based detector seems to work the other way round and prioritizes a low fnr (figure 4d ). otherwise, the results are very similar, which shows that also the difficult raid dataset can be learned extremely well (at least if we remove the shortest and lowestquality texts). training on pan seems to generalize better to other datasets than training on raid, but the two seem to be incompatible. ## discussion the results from our exploratory data analysis in section 4 and model evaluation in section 5 bring us to an important debate about the quality and appropriateness of the data and the conclusions we can draw from models trained and evaluated on it. ## conclusion in this work, we delineated the parallels between authorship analytics and llm detection. we explicated the two authorship analytics paradigms of attribution and verification, how they are different, and how they apply to llm detection. through critical analysis of, and insights from the related work, and our own experiments, we showed how llm detection is often implicitly framed as authorship attribution. detectors are built under this paradigm but are evaluated on broad benchmarks that are in part unsolvable and in part too complex to be approached by means of attribution. we showed how this modeling approach misses the key aspect of llm detection being a one-class problem and how it fails to scale to better, more diverse, and linguistically more consistent llms already today.