Your Large Language Models are Leaving Fingerprints (2025)

Abstract: It has been shown that fine-tuned transformers and other supervised detectors are effective for distinguishing between human and machine-generated texts in non-adversarial settings, but we find that even simple classifiers on top of n-gram and part-of-speech features can achieve very robust performance on both in- and out-of-domain data. To understand how this is possible, we analyze machine-generated output text in four datasets, finding that LLMs possess unique fingerprints that manifest as slight differences in the frequency of certain lexical and morphosyntactic features. We show how to visualize such fingerprints, describe how they can be used to detect machine-generated text and find that they are even robust across text domains. We find that fingerprints are often persistent across models in the same model family (e.g. 13B parameter LLaMA’s fingerprint is similar to that of 65B parameter LLaMA) and that while a detector trained on text from one model can easily recognize text generated by a model in the same family, it struggles to detect text generated by an unrelated model.

## introduction large language models (llms) produce text often indistinguishable from human-authored text to human judges @xcite . this unfortunately allows potential misuses such as academic plagiarism @xcite and the dissemination of disinformation @xcite , which has therefore prompted interest in machine-generated text detection (mgt). we conduct linguistic analysis on four popular published datasets for mgt, showing that the machine-generated content in each shows linguistic markers in aggregate which make it relatively easy to separate it from human content. these discrepancies, which we call a model's "fingerprint", are consistent enough across domains and within model families that we find we can treat each llm as if it were a unique author with a distinct writing style. to do so, we use a wellfounded method from the field of author identification (aid) for a closed set of authors: using handcrafted n-gram features and training a simple machine learning classifier on those features. best reported model n-gram (ours) paper f1 auroc f1 auroc deepfake -99.0 94.7 94.3 hc3 99.8 -96.7 99.6 ghostbuster 99.9 100.0 98.0 98.0 outfox 96.9 -98.7 98. 7 as shown in in this paper, we empirically uncover and characterize the fingerprints of individual and families of llms through a series of comprehensive analyses, and present a new perspective of llm-content detection as authorship identification. noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 davinci-002 noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 davinci-003 noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 gpt-3.5-turbo openai noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 gpt-j noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 gpt-neox noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym empty eleuther noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 llama-13b noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 llama-30b noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 llama-65b llama noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 glm-130b noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym empty noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym empty glm noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 t5-small noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 t5-large noun punct verb pron adp det aux adj adv propn part cconj sconj num x intj sym 0 0.06666666666666667 0.13333333333333333 0.2 t5-xxl flan figure 1: visualization of the fingerprints. we plot frequencies of each part-of-speech (pos) class from the output of several models, sorted by model family. within each family, the shapes (distributions) look mostly similar regardless of model size. each radial plot is shown at the same 0% to 20% frequency scale, with pos tags sorted from most to least common among human-written outputs. jagged/bumpy shapes indicate the fingerprint is more distinct from human distributions. pos is just one component of the full 'fingerprint' we investigate. ## experiments we conduct a series of analyses of llm fingerprints, finding (1) they are predictive of which model authored a text, (2) consistent across domains, and (3) relatively consistent within model families. ## related work a common approach to machine-generated text detection is to train a supervised binary classifier on labeled data @xcite @xcite . @xcite proposed a variety of classification testbeds, finding that pre-trained language models perform the best. while n-gram frequencies have often been used for author identification, only a few recent works examine hand-crafted features or stylometrics in machine-generated text detection @xcite . one example is @xcite : a unique system that uses the top-k words to highlight text spans to visually aid humans in the task of spotting ai-written text themselves. @xcite finds a combination of fine-tuned neural features and hand-crafted linguistic features effective for mgt on the m4 dataset as part of the semeval2024 task on machinegenerated text detection @xcite . @xcite analyze their corpus deepfake-textdetect across linguistic feature axes, but report differences across pos-tag distributions between human and machine data when considering all models and domains in aggregate as insignificant; however, they do find these distributions begin to diverge when considering a subset of models or domains. we demonstrate that these differences extend to every publicly available machine text detection dataset, prove largely consistent within model families, and are very powerful features for training a robust machine-generated text detection classifier. while linguistic-feature-based approaches have shown promise, other state-of-the-art (sota) methods, such as @xcite ; @xcite ; tian and cui (2023), adopt probabilistic and statistical modeling approaches to detect machine-generated text in a training-free setting. we focus purely on manually extracted linguistic features rather than probability curvatures. ## conclusion we demonstrate that in four popular datasets for machine-generated text detection, n-gram features are highly effective for mgt. we uncover that llms have unique writing styles that can be captured in lexical and syntactic features, which we characterize as "fingerprints", and show may be effectively harnassed for text-detection in a variety of settings.