noumenon-labs/Firebird-ModernBERT-512

Firebird — ModernBERT 512

Firebird is the model series powering experiments for Earlybird V2, an AI-generated text detection project.

This checkpoint is the first controlled Firebird baseline:

• Architecture: ModernBERT-base
• Maximum context: 512 tokens
• Task: binary HUMAN / AI text classification
• Training examples: 705,329
• Epochs: 1
• Effective batch size: 32
• Learning rate: 2e-5

Labels

• 0 — HUMAN
• 1 — AI

Evaluation

Earlybird V2 — ID

Metric	Result
Accuracy	98.45%
Macro F1	0.9813
MCC	0.9626
AUROC	0.9987
Human FPR	2.66%

Earlybird V2 — OOD

Metric	Result
Accuracy	82.17%
Macro F1	0.8216
MCC	0.6970
AUROC	0.9588
Human FPR	29.88%

PAN held-out validation

Metric	Result
Accuracy	99.05%
Macro F1	0.9897
MCC	0.9793
AUROC	0.9991
Human FPR	1.25%

PAN validation is held out from training, but PAN training data was included in the training corpus. It should therefore not be interpreted as a completely unseen-domain benchmark.

Known limitations

Performance varies substantially across domains.

A diagnostic OOD evaluation found particularly high false-positive rates on short procedural human fragments. Formatting artifacts such as [title], [step], and [substeps] were also found to inflate AI confidence.

This checkpoint should not be used as sole evidence for academic misconduct, authorship attribution, or other high-stakes decisions.

Earlybird V2

The public Earlybird V2 dataset combines material derived from MAGE, HC3, SemEval 2024 Task 8, TuringBench, and RAID.

Additional PAN training material was used locally for this checkpoint and is not redistributed with the Earlybird V2 dataset.

Series

Firebird checkpoints are controlled experiments exploring architecture, context length, training distribution, and robustness for AI-generated text detection.

Firebird ModernBERT 512 is the 512-token baseline.