inbox-triage-encoder (v4)

A ModernBERT-base classifier that makes an inbox assistant's one decision per incoming message: reply, ask_human, wait or no_reply. It is the encoder gate in gonogo's inbox-triage example, where it is compared with Claude and TypeSafe's Jev on the same 870 Enron messages. It was fine-tuned with thomas and temperature-scaled on a held-out calibration split. The eval and holdout messages, with their labels, are the gated dataset keppy/inbox-triage-enron.

It is a worked example for measuring a gate, not a production triage model.

Read before using the numbers

• The labels are a reference policy written by two model families (DeepSeek and GLM, agreement only, plus DeepSeek alone on extra training rows), not by people. A blind human check of a sample is pending.
• The eval and holdout messages come from four Enron inboxes (two lawyers, a government-affairs lead and an operations lead) from 2000–2001; the training rows add other mailboxes, with any thread that appears in eval or holdout dropped (the FERC public release, CMU's May 2015 copy). In those inboxes the assistant should rarely reply on its own, so reply is rare and ask_human is common. Expect a different mix in your mail.

Results (gonogo 0.4, from the example's reports)

set	pass rate (95% interval)	calibration error
eval, enriched (541)	80.4% [76.9, 83.5]	0.05
eval, natural mix (344)	86.0% [82.0, 89.3]	0.04
holdout (329)	78.1% [73.3, 82.2]	0.06

The pass rate hides the kind of mistake. On the holdout, the encoder leaves 42 of the 159 messages that needed the owner with no reply and no hand-off; Claude leaves 5 and Jev 3 on the same cases. It drafts a reply to 10 of those 159. Under a declared cost rule (a missed owner message costs 4× an unneeded escalation), its cost per 100 messages is about twice Claude's or Jev's. The review threshold chosen on eval did not hold on the holdout at a 90% precision target.

Use

The artifact follows thomas's contract: label2id.json maps labels to logit indices and temperature.json holds the temperature to divide the logits by before the softmax.

import json, torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

d = snapshot_download("keppy/inbox-triage-encoder")
tok = AutoTokenizer.from_pretrained(d)
model = AutoModelForSequenceClassification.from_pretrained(d).eval()
id2label = {v: k for k, v in json.load(open(f"{d}/label2id.json")).items()}
T = json.load(open(f"{d}/temperature.json"))["temperature"]

text = "Mailbox owner: Shackleton\nFrom: ...\nSubject: ...\n\n..."   # same layout as the example
with torch.no_grad():
    logits = model(**tok(text, truncation=True, max_length=512, return_tensors="pt")).logits[0]
probs = torch.softmax(logits / T, dim=-1)
print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs)})

Messages were formatted as Mailbox owner / From / To / Cc / Date / Subject, a blank line, then the body cut at 2,000 characters (see gates.message_text in the example).