samjohnson11/sms-spam-distilbert

SMS Spam DistilBERT

An English SMS spam/ham classifier fine-tuned from distilbert-base-uncased on the UCI SMS Spam Collection (5,574 messages).

Uploaded checkpoint: MLflow subset demo

These weights are from the MLflow demo, not the full-data fine-tune. Run 2b09015f87d94c65870560c98e8346a3: 1,000 training messages, one epoch, 300 held-out test messages, maximum length 64 tokens, batch size 16, learning rate 5e-5, seed 42; stratified 80/20 split followed by stratified subsets.

Verified demo results: 98% accuracy, spam F1 0.921 (approximately 0.92), precision 0.972, recall 0.875. Labels: 0 = ham, 1 = spam.

Project results (separate experiments)

The full fine-tune used a held-out test set of 1,115 messages and reported 99.3% accuracy and 0.973 spam F1, compared with a TF-IDF logistic regression baseline at 0.912 spam F1 that let 24 spam messages through. These are project-reported results, not results for the checkpoint uploaded here. The full fine-tune checkpoint was not available in the local project artifacts.

Independent Claude LLM-as-judge comparisons on the same 300 messages (40 spam / 260 ham), same prompt and 80-token cap. These are separate model comparisons, not measurements of these DistilBERT weights.

Judge	Correct	Accuracy	Precision	Recall	Spam F1	Cost
Claude Opus 4.6 (claude-opus-4-6, temperature 0)	297/300	0.990	0.974	0.950	0.962	~$0.63
Claude Haiku 5.5 (claude-haiku-5-5, temperature omitted, effort low)	299/300	0.997	1.000	0.975	0.987	~$0.018

Haiku 5.5: no replies truncated (all 300 end_turn, longest 72 tokens, no thinking tokens); 103,781 input / 14,351 output tokens at $0.10/$0.50 per MTok.

Usage

from transformers import pipeline
classifier = pipeline("text-classification", model="samjohnson11/sms-spam-distilbert")
print(classifier("Hey, are we still meeting for lunch tomorrow?", truncation=True, max_length=64))

Limitations

Educational benchmark/demo, not a production spam filter. Short English SMS data may not represent modern scams, other languages, or longer messages. Inputs are truncated to 64 tokens for demo inference. False negatives and false positives occur; confidence scores are not calibrated guarantees. For example, a synthetic cash-prize message was incorrectly labeled ham during a smoke test. Do not submit sensitive personal messages to the public demo.

Links and licensing

• Project source and experiments
• UCI SMS Spam Collection

Model weights inherit the Apache-2.0 license of the base model. No training messages are distributed here; consult UCI for dataset licensing and attribution.