noumenon-labs/Firebird-ModernBERT-512-RW

Firebird ModernBERT 512 — Reward-Weighted Exp001

Firebird-ModernBERT-512-RW is an experimental post-trained variant of noumenon-labs/Firebird-ModernBERT-512.

It is a ~149M parameter ModernBERT binary classifier for distinguishing:

• 0 — HUMAN
• 1 — AI

The maximum sequence length is 512 tokens.

What is different?

This checkpoint was produced through reward-weighted classifier post-training.

The original Firebird checkpoint was kept frozen as a reference model. Training examples were scored by the original classifier, difficult examples received larger loss weights, and the post-trained model was constrained against the frozen reference using a KL penalty.

The objective was approximately:

L = weighted_cross_entropy
    + beta * KL(reference || policy)

Exp001 configuration:

KL beta:               0.10
Learning rate:         5e-6
Epochs:                1
Training examples:     53,598
Optimizer steps:       1,675
Batch size:            32
Gradient accumulation: 1
GPU:                   RTX 4090 24 GB
Runtime:               ~17 minutes

Hard human examples received greater weighting than ordinary examples because one goal of the experiment was reducing false AI detections.

Is this RL?

Not in the conventional PPO/GRPO/RLHF sense.

Firebird is a discriminative encoder classifier rather than a generative policy. Exp001 directly optimized a differentiable weighted classification objective with a frozen-reference KL constraint.

For that reason, reward-weighted post-training or reward-guided classifier post-training is a more precise description.

---

Evaluation

Earlybird distribution-shift stress test

Dataset:

noumenon-labs/Earlybird-V2
split: validation_ood
N = 26,705

At threshold 0.50:

Metric	Original Firebird	Reward-Weighted
Accuracy	80.51%	84.01%
Balanced accuracy	83.63%	86.53%
Macro F1	80.51%	83.99%
MCC	0.6729	0.7223
AUROC	0.9825	0.9805
Human FPR	32.58%	26.57%
Human recall	67.42%	73.43%
AI recall	99.84%	99.63%
AI precision	67.48%	71.75%

Paired comparison:

Original mistakes corrected:    959
New mistakes introduced:         24
Net corrections:               +935

Human false positives fixed:    959
Correct humans broken:            1

Correct AI examples broken:      23

The primary improvement at the default operating point came from substantially fewer human false positives while preserving very high AI recall.

---

AI Text Detection Pile Cleaned

Dataset:

srikanthgali/ai-text-detection-pile-cleaned
split: validation
N = 72,162

At threshold 0.50:

Metric	Original Firebird	Reward-Weighted
Accuracy	82.26%	81.19%
Balanced accuracy	82.26%	81.20%
MCC	0.6459	0.6250
AUROC	0.8941	0.8984
Human FPR	20.19%	15.78%
Human recall	79.81%	84.22%
AI recall	84.70%	78.17%
AI precision	80.80%	83.25%

The unchanged 0.50 threshold exposes an important calibration shift: Exp001 generally assigns lower AI probabilities than the original checkpoint.

Because of this, matched-operating-point analysis was also performed.

Matched AI recall

AI recall	Original FPR	Reward-Weighted FPR
70%	13.19%	12.51%
75%	15.22%	14.40%
80%	17.47%	16.66%
85%	20.38%	19.38%
90%	24.38%	23.51%
95%	31.93%	30.85%

At each tested AI-recall target, Exp001 produced a lower human false-positive rate.

Matched human false-positive rate

Human FPR	Original AI recall	Reward-Weighted AI recall
5%	38.82%	39.31%
10%	60.21%	61.84%
15%	74.60%	76.41%
20%	84.49%	85.90%
25%	90.61%	91.25%
30%	94.03%	94.50%

At each tested human-FPR target, Exp001 produced higher AI recall.

Across sampled operating points from 0.1% through 40% human FPR, Exp001 produced higher AI recall at approximately 99.9% of sampled points.

The mean AI-recall improvement at matched human FPR was approximately:

+1.014 percentage points

Pile AUROC:

Original: 0.894066
Exp001:   0.898377
Delta:   +0.004311

Best MCC found in the diagnostic threshold sweep:

Original:
    MCC       = 0.664386
    threshold = 0.302

Exp001:
    MCC       = 0.673120
    threshold = 0.125

These threshold values were selected retrospectively on the benchmark and therefore should be treated as diagnostic results, not deployment thresholds.

---

HellaSwag human stress test

HellaSwag is not an AI-text detection benchmark.

Its human contexts were used here only as a difficult distribution-shift stress test.

After removing HellaSwag-style bracket metadata:

Original human FPR: 88.00%
Exp001 human FPR:   81.86%

The result remains poor in absolute terms.

This exposes an important remaining weakness, especially for unusual, short, fragmentary, and procedural human text.

The HellaSwag result should not be interpreted as performance on a standard AI-detection benchmark.

---

Interpretation

Exp001 investigated whether reward-weighted hard-example post-training could reduce Firebird's tendency to classify difficult human text as AI-generated without completely sacrificing AI detection.

The experiment produced:

• substantially lower human FPR on the Earlybird OOD stress test;
• lower clean HellaSwag human FPR;
• a calibration shift toward HUMAN predictions;
• modestly higher Pile AUROC;
• a modestly improved Pile ROC operating frontier.

The results therefore suggest that Exp001 did more than merely move the default classification threshold.

However, this checkpoint does not solve AI-text detection.

Distribution shift remains a major weakness.

---

Intended use

This checkpoint is intended for:

• research;
• benchmarking;
• classifier post-training experiments;
• AI-text detector robustness research;
• false-positive analysis;
• calibration research;
• reward-weighted classifier experiments.

It should not be used as proof that a person used generative AI.

Detector outputs are probabilistic predictions and may be incorrect, particularly for out-of-distribution human writing.

---

Labels

0 = HUMAN
1 = AI

---

Basic usage

import torch

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

repo = "noumenon-labs/Firebird-ModernBERT-512-RW"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = AutoModelForSequenceClassification.from_pretrained(repo)

text = "Your text goes here."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

with torch.no_grad():
    logits = model(**inputs).logits

probabilities = torch.softmax(
    logits,
    dim=-1,
)[0]

p_human = probabilities[0].item()
p_ai = probabilities[1].item()

print({
    "p_human": p_human,
    "p_ai": p_ai,
})

---

Relationship to Firebird

Firebird-ModernBERT-512
         |
         |
         +-- hard-example mining
         |
         +-- reward weighting
         |
         +-- frozen reference model
         |
         +-- KL reference constraint
         |
         v
Firebird-ModernBERT-512-RW

Exp001 is preserved as a separate experimental checkpoint rather than silently replacing the original Firebird release.

---

Reproducibility note

Some historical Firebird evaluation numbers were produced using an earlier evaluation pipeline. Later same-run comparisons exposed differences between historical and current evaluation outputs.

For this reason, the comparisons reported on this model card prioritize same-run Original-vs-Exp001 evaluations where both checkpoints were evaluated through the same inference harness.

Exact evaluation-pipeline reproducibility remains an area for further audit.

---

Noumenon Labs

Experimental AI-text detection and robustness research.