nhis-project7-distilbert

A DistilBERT model fine-tuned to decide whether an English tweet reports a real disaster. It was built for NHIS Project 7 (Disaster Tweet Classification) and is the model behind the project's Streamlit dashboard.

• Task: binary text classification. LABEL_1 = real disaster, LABEL_0 = not a disaster.
• Base model: distilbert-base-uncased
• Code and notebook: [add the GitHub repository link]

Intended use and limitations

This is a learning project, not an official alert system. Do not use it for emergency decisions.

• Figures of speech can fool it. Disaster words are mostly used for real events in the training tweets, so the model can treat everyday uses as emergencies. For example, "The student set the fire on the stage with their performance" received a 98% disaster score.
• Overconfident probabilities. Very short messages (such as "Heavy rain") can receive scores of 96–99%. Treat the probability as the model's estimate, not a guarantee.
• English tweets only. Other languages and long texts were not tested.
• The lead over simpler models is small. On the test set, DistilBERT led the next-best model (Logistic Regression) by 0.010 Accuracy, which is within sampling noise.

Training and evaluation data

A hand-labelled set of 7,613 disaster/non-disaster tweets; 7,490 were used after data cleaning [add the dataset source and licence here]. The tweets were split 80/20 (5,992 training, 1,498 held-out test). Within the training part, 85% was used for fine-tuning and 15% for validation. The test set was untouched until final evaluation.

Input format

The model was trained on cleaned text, so inputs must be cleaned the same way: HTML entities decoded, links and @mentions removed, "#" removed, everything except letters removed, lowercased, common English words (stop words) removed, and words reduced to their base form (lemmatised). Raw tweets will give worse results.

from transformers import pipeline

clf = pipeline("text-classification", model="ysakkaf/nhis-project7-distilbert")
clf("wildfire spreading fast near highway evacuation ordered")   # cleaned text

Results

On the 1,498 held-out test tweets:

Metric	Value
Accuracy	0.8144 (95% interval 0.7937–0.8344)
F1	0.7718
Precision	0.8103
Recall	0.7367
ROC AUC	0.8749

Always predicting the larger class would score an Accuracy of 0.574.

Training procedure

Fine-tuned with the Hugging Face Trainer. The saved model is the epoch-2 checkpoint, which had the best validation Accuracy (0.8209).

• learning_rate: 2e-05, weight_decay: 0.01
• train_batch_size: 8, eval_batch_size: 8
• num_epochs: up to 4 (early stopping, patience 2), max_length: 128
• seed: 42, optimizer: AdamW, lr_scheduler: linear

Validation results by epoch

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
0.4615	1.0	637	0.4160	0.8165	0.8539	0.6867	0.7612
0.358	2.0	1274	0.5059	0.8209	0.8227	0.7389	0.7785
0.2816	3.0	1911	0.6596	0.8009	0.7698	0.7598	0.7648
0.2163	4.0	2548	0.7222	0.8031	0.7711	0.7650	0.7680

Framework versions

• Transformers 4.57.6
• Pytorch 2.11.0+cpu
• Datasets 4.8.5
• Tokenizers 0.22.2