SENTRA : Selected-Next-Token Transformer for LLM Text Detection (2025)

Abstract: LLMs are becoming increasingly capable and widespread. Consequently, the potential and reality of their misuse is also growing. In this work, we address the problem of detecting LLM-generated text that is not explicitly declared as such. We present a novel, general-purpose, and supervised LLM text detector, SElected-Next-Token tRAnsformer (SENTRA) . SENTRA is a Transformer-based encoder leveraging selected-next-token-probability sequences and utilizing contrastive pre-training on large amounts of unlabeled data. Our experiments on three popular public datasets across 24 domains of text demonstrate SENTRA is a general-purpose classifier that significantly outperforms popular baselines in the out-of-domain setting.

## introduction the problem of determining whether a text has been generated by an llm or written by a human has been widely studied in both academia @xcite and industry. several commerciallevel automated text detection systems have been developed, including gptzero @xcite , originality (originality.ai, 2025), sapling @xcite , and reality defender @xcite . although significant progress has been made in detecting llm-generated text over the past several years, these systems remain far from perfect and are often unreliable. a major limitation is their brittleness: they can perform well on certain types of llm-generated text but fail catastrophically in other cases @xcite . this issue is particularly pronounced when operating in a real world scenario, where models must handle out-of-domain (ood) data, different llm generators, or various llm "attacks" @xcite † work performed at mozilla corporation. software: https://github.com/firefox-ai/sentra/ @xcite . therefore, it is crucial to develop more generalizable methods that deliver reliable performance across these settings. the probability of a document under an llm's model can be measured by auto-regressively feeding the document's tokens into the llm and observing the predicted probabilities for each token. this process produces a sequence of values called selected-next-token-probabilities (sntp) that has been extensively used in prior work on llmgenerated text detection @xcite @xcite . these prior works primarily rely on either heuristics (handcrafted functions) applied to sntp sequences or linear models trained on expert-derived features @xcite . in contrast, our proposed approach encodes sntp sequences using a transformer model pre-trained on unlabeled data, leveraging the expressivity of transformers to directly learn a representation of the probability that a single or multiple llms assign to tokens in a document. more specifically, we propose selected-next-token transformer (sentra), a method for detecting llm-generated text that directly learns a detection function in a supervised manner from sntp sequences. this method utilizes a novel transformer-based architecture with a contrastive pre-training mechanism. the learned representation can be fine-tuned on labeled data to create a supervised model that distinguishes llm-generated texts from human-written texts. for the llm-text-detection task, supervised detectors have been shown to generalize poorly outside the training distribution @xcite . prior supervised methods typically leverage raw tokens as input and tend to overfit to token selections in a document. heuristic or linear models on sntp input have been shown to generalize well, but these simple models lack the expressivity to fully exploit the information in the sntp sequences. our sen-tra network addresses this issue by learning gen-eralizable functions on sntp. we show empirically that the supervised method presented in this paper generalizes to unseen domains better than both supervised and unsupervised baselines by leveraging our proposed transformer-based architecture, thus demonstrating greater generalization to distribution shifts. in this paper, we demonstrate the following: because of the number of possible domains, improving out-of-domain generalization is the most important task to achieve llm generated text detection in the wild. ## related work with the rise of llms, significant research has been conducted on accurately detecting text generated by these models @xcite . at a high level, these detectors can be categorized into three approaches: watermarking, unsupervised (or zeroshot) detection, and supervised detection. watermarking generally relies on the llm deliberately embedding identifiable traces in its output @xcite . in this work, we focus on the general detection problem, including cases involving noncooperative llms; therefore, we do not consider watermarking as a point of comparison. unsupervised methods typically leverage metrics computed by an llm on the target document. these methods can be further divided into white-box detection, where the candidate llm is known @xcite , and black-box detection, where the candidate llm is unknown @xcite . given our focus on the general detection problem, we prioritize black-box detection methods. supervised methods, on the other hand, involve collecting a corpus of human-written and llm-generated text samples, which are then used to train the detection models @xcite . selected-next-token-probabilities (sntp) have been widely used for llm detection in both white and black box settings @xcite @xcite . perplexity @xcite is a commonly used metric to evaluate an llm's ability to model a given dataset. in the context of ai detection, a lower perplexity score on a document indicates an llm "fits" a document and this indicates a higher likelihood the document was llm-generated. conversely, a higher perplexity score suggests the llm's probability model does not fit or accurately represent the candidate text, implying a lower likelihood that the text was generated by the llm @xcite . some detectors use multiple sequences of sntp for the detection task @xcite . @xcite leveraged sntps from two markov models, along with an llm's sntp, extracted features, and a forward feature selection scheme as inputs to a linear classifier. in contrast to @xcite , @xcite argued that relying solely on the perplexity score for llm-generated content detection can be misleading. although human-authored text generally results in higher perplexity, prompts can significantly influence perplexity values. the authors highlighted the "capybara problem", where the absence of a prompt can cause an llm-generated response to have higher perplexity, leading to false detections. they addressed this issue by introducing cross-perplexity as a normalizing factor to calibrate for prompts that yield high perplexity. detectgpt is an unsupervised method based on the idea that texts generated by llms tend to "occupy negative curvature regions of the model's log probability function" @xcite . the method generates perturbations of the sample text using a smaller model and compares the log probability of the sample text to that of the perturbations. fast-detectgpt replaces the perturbations in de-tectgpt with a more efficient sampling step @xcite . @xcite observed that the similarity between a sample and its counterpart generation is notably higher than the similarity between the counterpart and another independent regeneration. they demonstrated that this difference in similarity is useful for detection. the most common supervised baseline for llmgenerated text detection is a roberta classifier @xcite trained on a corpus of labeled text, where each document is marked as either human-written or llm-generated. several studies have expanded on this approach to supervised text-based classification. @xcite trained a feed-forward classifier with two hidden layers using intrinsic features derived from transformer hidden states, determined via kl-divergence. tian et al. (2024) address the challenge of detecting short texts by treating short samples in the training corpus as partially "unlabeled". hu et al. (2023) employed adversarial learning to improve the robustness of their roberta-based classifier against paraphrase attacks. several publications have explored contrastive training for the llm detection task @xcite @xcite . these studies use contrastive pre-training for a text transformer, which is chosen to be roberta @xcite in many cases, to guide the network toward a representation more useful for llm-generated text detection. furthermore, many prior contrastive training strategies focus on identifying stylometric features @xcite , while other studies extract stylometric features directly and train classifiers using those features @xcite . rather than focusing on text representations, our method is mainly designed to produce useful sntp representations and, thus, proposes a different contrastive pre-training scheme that compares textual representations with those of the sntp transformer. however, sntp and supervised methods have been shown, both intuitively and empirically, to struggle with generalization to unseen domains @xcite . for instance, @xcite applied adaptive ensemble algorithms to enhance model performance in ood scenario. meanwhile, @xcite and @xcite , recognizing the limited number of widely adopted general-purpose ai assistants, proposed to train an embedding model to learn the writing style of llms, and thereby improving the detection accuracy. prior work has shown sntp to be an effective in-put for identifying llm generated text @xcite @xcite ), but they rely on relatively simple metrics or heuristics. in this paper, we propose a transformer-based sentra model that learns a representation of sntp sequences used for more effective training of detection models that better generalize to unseen domains. ## conclusions in this paper, we proposed a novel general purpose supervised llm text detector method sen-tra that is a transformer-based encoder leveraging sntp sequences and utilizing contrastive pre-training on large amounts of unlabeled data. we show this supervised method acting on sntp input outperforms previously considered heuristic functions and other methods that rely on text input. since supervised detectors tend to perform better on data that is similar to their training distributions @xcite , it is essential to include a wide variety of domains when testing such general-purpose detectors. therefore, we tested the performance of sentra on three public datasets raid, m4gt and mage containing a broad range of different domains (24 in total) across various experimental settings and compared its performance with eight popular baselines. we also evaluated sentra and the baselines on a out-of-llm evaluation. we empirically demonstrated that sentra significantly outperformed all baselines in our studied experimental settings. on our three evaluation datasets, sentra outperformed all eight popular baselines for the average and the worst-case ood scenarios. these results show that sentra is a strong method for training llm text detectors that can generalize well to unseen domains and llm generators. our ablation study showed performance of sentra increases when two frozen llms are used instead of one frozen llm. we also demonstrated our contrastive pre-training strategy increased the performance of sentra on all outof-domain evaluations. because sentra is better able to handle these critical out-of-domain and out-of-llm settings, these results demonstrate sentra is a generalpurpose encoder that can serve as a foundation for the llm text detector models.