gdsrAbhi/cyber-url-slm

Cyber URL SLM — Malicious URL Detector

A fine-tuned DistilBERT (~67M params) that classifies URLs as benign or malicious. My first language model — trained in 2 hours on Kaggle as a learning exercise.

What it does

Input: a URL (e.g. paypa1-secure-login.com/verify) Output: BENIGN or MALICIOUS with a confidence score.

Training details

• Base model: distilbert-base-uncased
• Dataset: Malicious URLs Dataset by sid321axn (~651K URLs)
• Subset used: 20,000 URLs, balanced 50/50 (benign/malicious), with 4 malicious classes collapsed into one
• Preprocessing: Lowercased, stripped http://, https://, and www. prefixes from all URLs (training and inference) to prevent the model from learning protocol-presence as a spurious shortcut
• Hyperparameters: 2 epochs, batch size 32, learning rate 2e-5, AdamW optimizer, 100 warmup steps, weight decay 0.01
• Hardware: Kaggle T4 GPU, ~4 minutes training time

Results

Metric	Value
Accuracy	90.1%
F1	0.902
Precision	0.897
Recall	0.907

A note on the metrics

An earlier version of this model hit 97% accuracy — but only because the training data had a spurious correlation between URL protocol prefix (http:// vs bare domain) and label. The model had learned to detect the prefix, not the threat. After normalizing the protocol on both classes, accuracy "dropped" to 90% — but the model now generalizes to actual phishing patterns. The 90% number is the real one.

Limitations

• Famous short domains can be misclassified. Bare domains like google.com are rare in the benign training set (which mostly contains long path-heavy URLs), so the model can flag them as malicious. Not a model bug — a data coverage issue.
• Trained on URL strings only. No domain age, WHOIS data, page content, or SSL info.
• Dataset is from 2021. Phishing patterns evolve; performance on today's threats may be lower.
• Not for production use. This is a learning project, not a security tool.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "gdsrAbhi/cyber-url-slm"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

def normalize_url(url):
    url = str(url).lower().strip()
    for prefix in ['https://', 'http://']:
        if url.startswith(prefix):
            url = url[len(prefix):]
    if url.startswith('www.'):
        url = url[4:]
    return url

def predict(url):
    cleaned = normalize_url(url)
    inputs = tokenizer(cleaned, return_tensors='pt', padding='max_length', truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=1)[0]
    label = 'MALICIOUS' if probs[1] > probs[0] else 'BENIGN'
    return label, max(probs).item()

print(predict("http://paypa1-secure-login.com/verify"))

What I learned building this

This project taught me more about ML by failing than it would have by working. The 7% accuracy gap between v1 and v2 of this model is a perfect demonstration of how dataset bias can make a useless model look great. I'd rather ship the honest 90% model than the inflated 97% one.