mesmertech/mesmer-ai-detector-4b

Mesmer AI Detector 4B

Tells you whether English text is human-written, AI-written or AI-edited (human writing that an AI reworded or polished), and labels each sentence. Fine-tuned from Qwen3.5-4B-Base. A bigger, more accurate sibling: mesmer-ai-detector-9b.

Try it without a GPU: Hugging Face Space · mesmer.tools

Fun fact: this whole project was generated with Claude. Claude Opus collected the data, trained and evaluated these models autonomously, and wrote this card, so the card doubles as a test sample. Run on this card, the 4B says AI-written (p ≈ 0.99 AI, 0.01 AI-edited, 0.00 human) and flags every sentence.

Benchmarks

The best open-weight detector on every benchmark we could compare. It catches AI text about as well as the closed Pangram models (better than Pangram 3.x, a little behind Pangram 4 on style imitation), but it flags more human writing (see Limitations).

Open Pangram suite

These are Pangram's own eval sets and protocol from the Open Pangram release. Ternary means human vs AI vs AI-edited. Llama-family text was kept out of our training, so the Llama-3.3-70B set is an unseen generator for us too.

	Pangram 3.2 (closed)	Open Pangram (Llama-3.2-3B)	Mesmer 4B	Mesmer 9B
In-domain, ternary accuracy ↑	0.920	0.895	0.918	0.923
Enron emails (unseen domain), ternary accuracy ↑	0.905	0.863	0.912	0.911
Llama-3.3-70B (unseen generator), ternary accuracy ↑	0.952	0.921	0.929	0.944
Non-native English essays (TOEFL), FPR ↓	0.000	0.055	0.044	0.011
Human Detectors, F1 ↑ / FPR ↓	1.000 / 0.000	0.987 / 0.027	0.973 / 0.053	0.997 / 0.007
RAID 10k sample, F1 ↑ / FNR ↓	0.992 / 0.007	0.930 / 0.062	0.994 / 0.003	0.995 / 0.003

Pangram's numbers are from its blog. Ours use the bf16 weights; the int8 weights are within 0.007 on the ternary rows.

Other public benchmarks (zero-shot)

	Pangram 4 (closed)	Pangram 3.3.2 (closed)	Open Pangram	MELD (open)	Mesmer 4B	Mesmer 9B
MELD-eval: % of AI caught at 1% FPR (4 unseen 2026 models) ↑	99.99	99.96	61.7	99.7	99.98	100.00
MELD-eval: worst of 6 character/word attacks ↑	99.96	99.72			99.90	100.00
DetectRL AUROC: multi-domain ↑	97.02	99.75	99.48		99.81	99.89
DetectRL AUROC: multi-LLM ↑	97.32	99.85	99.48		99.81	99.92
DetectRL AUROC: multi-attack ↑	97.22	99.72	99.26		99.64	99.76
DetectRL AUROC: human writing ↑	93.80	95.67	96.44		97.63	97.59
Epoch AI style imitation: AI missed of 594 ↓	17	30			25	24
Epoch AI: human texts flagged of 495 ↓	0	0			4	3

Other detectors' numbers are as published by Pangram, the Pangram 4 report, the MELD paper and Epoch AI (GPTZero on Epoch: 34 missed, 0 flagged). Ours use the int8 weights. Epoch counts strictly: only an "AI" verdict counts as caught, and an "AI-edited" verdict on human text counts as flagged. We dropped 140 benchmark human texts that overlapped our training data.

Quickstart

pip install torch numpy tokenizers safetensors emoji huggingface_hub
pip install fla-core   # optional: faster GPU kernels (otherwise pure PyTorch; CPU works too, but slowly)
hf download mesmertech/mesmer-ai-detector-4b mesmer_detector.py --local-dir .
python mesmer_detector.py --model mesmertech/mesmer-ai-detector-4b "Paste the text to check here."

from mesmer_detector import Detector

det = Detector.from_pretrained("mesmertech/mesmer-ai-detector-4b")   # int8=True: 8-bit weights, about half the memory
r = det.analyze(text)       # any length; long texts are scored in windows
r["verdict"]                # "human" | "ai" | "ai_edited"
r["probs"]                  # {"human": ..., "ai": ..., "ai_edited": ...}
r["sentences"]              # [{"text", "label", "p"}, ...] per sentence
det.predict(texts)          # batch scoring: [N, 3] probabilities from each text's first 1,023 tokens

Weights: bf16 model.safetensors 8.4 GB, or int8 int8/model.safetensors 4.2 GB (4.9 GB once loaded). On the GPU, add 1–3 GB for activations, or pass a smaller tokens_per_batch (default 16384). The default verdict thresholds (policy.json) were set on held-out human texts so that at most 0.7% are called AI-written and at most 1% get any flag.

Limitations

• English only. Below about 50 words the results are unreliable (the hosted tool refuses texts under 40 words).
• It flags more human writing than Pangram. At the default thresholds it flags about 0.3% of MELD-eval's human texts and 1% of DetectRL's. Most of these are academic abstracts, news articles and poetry.
• AI text written to imitate a specific person's style is the main way through, especially in scientific writing.
• No detector is proof. Never use a result as the only evidence against anyone.

License

CC BY-NC-SA 4.0: non-commercial use, with attribution and share-alike.