DistilBERT for SMS Spam Detection

This model classifies English SMS messages as ham or spam. It is a Transformers-based follow-up to my earlier study of TF-IDF logistic regression and a GloVe BiLSTM. The follow-up asks whether pretrained representations improve spam-class performance enough to justify their additional computational cost.

Research question

Does fine-tuning a pretrained Transformer materially improve spam-class performance over a strong TF-IDF logistic-regression baseline on the same fixed evaluation split?

Model description

Base model: distilbert/distilbert-base-uncased
Architecture: DistilBERT with a binary sequence-classification head
Task: English SMS text classification
Labels: ham (0), spam (1)
Primary metric: spam-class F1
Model license: Apache-2.0, following the base model

Dataset

The experiment uses all 5,574 messages in the UCI SMS Spam Collection, accessed through the ucirvine/sms_spam Hugging Face dataset loader. The original UCI record is the authoritative source for the dataset license.

The dataset is licensed under CC BY 4.0. It is not covered by the model's Apache-2.0 license.

Almeida, T. & Hidalgo, J. (2011). SMS Spam Collection [Dataset]. UCI Machine Learning Repository. DOI: 10.24432/C5CC84.

Experimental protocol

Deterministic, stratified 80/20 split with seed 42
4,459 training messages and 1,115 held-out evaluation messages
Identical evaluation examples for DistilBERT and the baseline
Decision threshold of 0.5 for both models
Best DistilBERT epoch selected by spam-class F1

Because the held-out split was also used for epoch selection, it is more accurately an evaluation split than a fully untouched final test set. A stronger follow-up would introduce a separate validation split or nested cross-validation.

Training configuration

Setting	Value
Epochs	4
Batch size	16
Learning rate	2e-5
Weight decay	0.01
Maximum sequence length	192 tokens
Random seed	42
Selected checkpoint	Epoch 1

The run used Python 3.11.16, Transformers 5.17.0, PyTorch 2.14.0, Datasets 5.0.1, and scikit-learn 1.9.1.

Baseline

The comparison model is logistic regression (class_weight="balanced", seed 42) over lowercased TF-IDF word unigrams and bigrams (min_df=2, sublinear term frequency). It is trained and evaluated on exactly the same examples as DistilBERT.

Results

Metric	DistilBERT	TF-IDF + Logistic Regression
Spam F1	0.9655	0.9488
Spam precision	0.9929	0.9653
Spam recall	0.9396	0.9329
Macro F1	0.9802	0.9705
Accuracy	0.9910	0.9865
ROC AUC	0.9975	0.9847
PR AUC	0.9886	0.9690

On this split, DistilBERT improved spam F1 by 0.0167 absolute while also improving every secondary metric. The sparse baseline remains competitive and is substantially cheaper to train and serve, so the Transformer gain should be weighed against latency, memory, and operational complexity rather than treated as evidence that Transformers are universally better.

Error analysis

I inspected every DistilBERT error on the 1,115-message evaluation split: nine false negatives and one false positive. There were fewer than ten examples in each category, so inspecting ten of each was not possible.

False negatives clustered around conversational or joke-like wording with no clear call to action, implicit or truncated promotions, and older ringtone or short-code marketing language. Several are semantically ambiguous enough to highlight possible label noise. The sole false positive was an informal personal message containing a phone number and an imperative to call, features that also occur frequently in spam.

In an additional seven-message qualitative sanity check, all three ordinary personal messages and three of four synthetic spam messages were classified as expected. A vague prize claim without an actual phone number, link, shortcode, or prize amount was incorrectly classified as ham, reinforcing the domain and wording-sensitivity limitation.

This small error set is descriptive, not a statistically stable taxonomy.

Intended use

Educational and research demonstrations of binary SMS spam classification
Reproducing a controlled comparison between a sparse linear baseline and a pretrained Transformer
Prototyping where predictions remain subject to human review

This model is not a production anti-abuse system.

Limitations

The dataset is small and old relative to modern messaging patterns.
The data is English-only and represents a narrow SMS domain.
Results come from one deterministic split and do not quantify variability across seeds.
Epoch selection used the evaluation split; no untouched final test set was retained.
A 0.5 decision threshold may not reflect the operational cost of false positives and false negatives.
Output scores have not been calibrated and must not be interpreted as calibrated confidence.
Performance may degrade on contemporary scams, other languages, longer messages, adversarial spelling, or data from different regions.

Ethical and practical considerations

False positives can suppress legitimate communication, while false negatives can expose users to abuse or fraud. Deployment would require representative current data, subgroup and drift analysis, threshold selection tied to explicit costs, monitoring, an appeal/recovery path, and privacy-preserving data handling. Do not use this model as the sole basis for punitive or high-impact decisions.

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="omarash/sms-spam-distilbert",
)

print(classifier("URGENT! You have won a $1000 cash prize. Reply WIN now."))

Reproducibility

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python train.py --output_dir ./distilbert-sms-spam

train.py recreates the fixed split, trains both models, selects the best DistilBERT epoch by spam F1, and writes metrics.json. Exact full-precision results are included in that file.

Relationship to the original SMS study

This model is a new, separately labelled extension. DistilBERT was not one of the models evaluated in the earlier coursework study, which compared TF-IDF logistic regression with a fine-tuned GloVe BiLSTM.

Canonical earlier write-up: SMS Spam Detection study

Citation and acknowledgements

The dataset should be cited as:

@misc{almeida_hidalgo_2011_sms_spam,
  author       = {Almeida, Tiago and Hidalgo, Jos\'e Mar\'ia G\'omez},
  title        = {SMS Spam Collection},
  year         = {2011},
  publisher    = {UCI Machine Learning Repository},
  doi          = {10.24432/C5CC84}
}

The model is based on distilbert/distilbert-base-uncased and uses the Hugging Face Transformers and Datasets libraries.