level20tauren/aeacus-s-1

Aeacus S 1

Aeacus S 1 is a token-level human-vs-AI writing classifier: a fine-tune of ModernBERT-base that labels every token human (0) or ai (1). It can show which parts of a text are likely to be AI-generated, and it can handle AI-generated text mixed with human-written text, even within the same paragraph.

Try it in your browser: the q4f16 export below, running entirely on your own device.

No training data is redistributed with these weights.

Files

file	size	notes
model.safetensors	571 MiB	fp32 PyTorch weights
config.json	2 KB	id2label maps 0 → human, 1 → ai
tokenizer.json	3.4 MiB	byte-level BPE, with offsets
onnx/model.onnx	573 MiB	fp32
onnx/model_fp16.onnx	288 MiB	fp16, no measurable change vs fp32
onnx/model_quantized.onnx	146 MiB	int8 dynamic, per-channel weights
onnx/model_q4.onnx	91 MiB	4-bit weights, blocks of 32, embedding included
onnx/model_q4f16.onnx	82 MiB	4-bit weights in fp16, smallest and fastest
thresholds.json	5 KB	calibration map and cutoffs, see below

The ONNX files are named the way transformers.js expects for each dtype, so they can be loaded directly with AutoModelForTokenClassification.from_pretrained(id, { dtype: "q4f16" }).

Usage

ModernBERT needs transformers >= 4.48:

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

name = "level20tauren/aeacus-s-1"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name).eval()

text = "Some text to check."
enc = tok(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    probs = torch.softmax(model(**enc).logits, dim=-1)[0, :, 1]

for piece, p in zip(tok.convert_ids_to_tokens(enc["input_ids"][0]), probs):
    print(f"{piece:>12}  {p:.3f}")

The model was trained on windows of 512 tokens with a stride of 128. Feeding longer inputs is not advised. Split long texts into overlapping windows and average each token's probability across the windows that cover it.

Calibration

Raw probabilities are overconfident: a token scoring 0.90 is not AI 90% of the time. We calibrate the model on our held-out test set, including models it was never trained on, to map the scores more accurately.

thresholds.json holds that calibration curve (raw score → estimated AI rate) and the two cutoffs.

Evaluation

Token-level results on data the model never trained on (full sets, PyTorch weights):

set	accuracy	AI F1	AUROC
test	0.982	0.973	0.996
held out (generators never trained on)	0.972	0.958	0.993
recent (human text from 2021–2022)	0.975	0.968	0.995

test is human written text from sources before 2021, and AI-generated texts from AI models we trained on.

held out is texts from AI models we never trained on. These results show that Aeacus is able to generalize to models and model families it has not seen before.

recent only consists of human written text from sources between 2021 to 2022, and AI-generated texts made to match similar topics. These results show that the model does not simply learn to differentiate old texts from new texts.

It is important to highlight that these scores are per-token. Whole-document scoring creates stronger results.

Quantization

We separately tested each quantization of the model against a small subset of documents to confirm how quantization effects the model.

variant	size	test acc	held out acc	AUROC (test / held out)	token labels identical to fp32
fp32	573 MiB	0.984	0.972	0.997 / 0.991	—
fp16	288 MiB	0.984	0.972	0.997 / 0.991	100% / 99.99%
int8	146 MiB	0.984	0.972	0.997 / 0.991	99.86% / 99.83%
4-bit	91 MiB	0.983	0.972	0.997 / 0.991	99.53% / 99.35%
4-bit fp16	82 MiB	0.982	0.972	0.997 / 0.991	99.53% / 99.35%

Quantization costs essentially nothing in accuracy, though scores do shift: the 4-bit exports disagree with fp32 on about 0.5% of token labels.

Intended use and limits

This model is intended for research, education, and entertainment purposes. Results of this model are not evidence.

Aeacus is deliberately conservative with the calibration: we choose to report only the top 5% of scores on either direction as human or AI. Still, AI-writing detectors can be unreliable. The results of Aeacus should not be used as the sole basis of any accusation, grade, hiring decision, misconduct finding, or any other kind of enforcement.

The model was not trained on adversarial evasion (paraphrase attacks, deliberate typo injection, homoglyphs).

License

The model weights and quantizations in this repository is licensed under MIT. Please see LICENSE.

Base model is ModernBERT-base, licensed under Apache-2.0, Copyright (c) Answer.AI and contributors. Aeacus S 1 is a derivative of it; the ModernBERT license and attribution are retained here.