nakas/gpttrace-v5-1b

GPTTrace text detector: Ettin-1B (v5)

A binary classifier that tells human writing apart from machine-generated text. It is GPTTrace's RAID leaderboard entry: liamdugan/raid#223.

• Base model: jhu-clsp/ettin-encoder-1b (MIT)
• Outputs: two logits, [human, machine]. The score is softmax[...,1].
• Weights: full precision (model/model.safetensors).
• Code: the training and evaluation code is at github.com/andrewnakas/gpttrace (eval/train/text/).
• Smaller version: a browser-sized 150M student distilled from this model is in progress.

RAID: official test-set results

These come from the RAID leaderboard CI on the hidden test set (672,000 texts).

Setting	AUROC	TPR @ FPR 5%	TPR @ FPR 1%
All generators, domains, decodings and adversarial attacks	99.94	99.90	99.42
Without adversarial attacks	99.97	99.95	99.77

Other measured results

Test	Result
MAGE out-of-distribution, unseen GPT-4 generations	AUROC 0.990, TPR@5%FPR 94.8%
MAGE out-of-distribution, paraphrased	AUROC 0.811, TPR@5%FPR 45.4% (a known hard case)
Held-out human text, flagged as machine at logit 0	arXiv 0.24%, Wikipedia 0.32%, PMC 0.0%, Gutenberg 0.0% (1,250 each)
RAID train, held-out titles (our split)	TPR@5%FPR 99.91%, AUROC 0.9995

Training

Data, one epoch, 842,830 texts, 33% human:

• RAID train (MIT), with paraphrase-attacked rows oversampled ×2. No RAID test data was used.

• 200k rows of MAGE train (Apache-2.0).

• 100k human texts:

  • Wikipedia (CC BY-SA)
  • PMC open-access sections, commercial subset (CC BY)
  • arXiv abstracts (CC0 metadata)
  • Project Gutenberg (public domain)

  Any text sharing 8-grams with our evaluation sets was dropped.

• About 16k topic-matched human/AI pairs:

  • AI side written by Qwen2.5-7B, Qwen3-8B and Phi-4-mini (open weights).
  • A third rewritten with paraphrase or "humanizer" instructions by a different model.

Setup:

• bf16, learning rate 2e-5 with cosine decay and 5% warmup, global batch 64, gradient checkpointing.
• One A100-40GB, 5.4 h.

Preprocessing: before tokenizing, use normalize.py (included). It removes zero-width characters and maps Cyrillic/Greek homoglyphs back to Latin, which counters RAID's homoglyph and zero-width attacks.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from normalize import normalize_text            # included in this repo

repo = "nakas/gpttrace-v5-1b"
tok = AutoTokenizer.from_pretrained(repo, subfolder="model")
model = AutoModelForSequenceClassification.from_pretrained(repo, subfolder="model").eval()
enc = tok(normalize_text(text), truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    p_machine = model(**enc).logits.softmax(-1)[0, 1].item()

Licence and limits

• Licence: CC BY-SA 4.0. Part of the human training text is share-alike (Wikipedia), so the weights are released under the same terms. The base model is MIT.
• A detector score is evidence, not proof. Short texts (under ~150 words) are unreliable, and paraphrased AI text is caught far less often (see MAGE above). Never use a score alone to accuse someone.