🎫 Support Ticket Classifier
Automatically classifies customer support tickets by category and urgency using a fine-tuned SetFit model trained on 30,000+ real support tickets.
What It Does
Input: Raw support ticket text
Output: Category + confidence score + urgency level
{
"category": "billing",
"confidence": 0.79,
"urgency": "high"
}
Categories
| Category | Example ticket |
|---|---|
billing |
"I was charged twice for my subscription" |
technical |
"My account keeps logging me out" |
complaint |
"This service is completely unacceptable" |
refund |
"I want to cancel and get my money back" |
Urgency Levels
| Level | When assigned |
|---|---|
high |
Fraud, service down, unauthorized charges, locked out |
medium |
General issues, standard requests |
low |
General questions, curiosity, minor changes |
Performance
| Metric | Score |
|---|---|
| Weighted F1 | 82% |
| Complaint F1 | 92% |
| Technical F1 | 82% |
| Billing F1 | 79% |
| Refund F1 | 68% |
Trained on 30,571 labeled tickets from Kaggle + HuggingFace datasets.
Evaluated on a held-out test set of 3,058 tickets.
Why Use This Instead of an LLM?
- ✅ 100x cheaper per call than GPT-4 at volume
- ✅ Fast — under 200ms per ticket
- ✅ Private — runs on your own server, data never leaves your infrastructure
- ✅ No vendor lock-in — no API key, no per-token billing
- ✅ GDPR friendly — fully on-premise capable
Quick Start
Install dependencies
pip install setfit==1.0.3 sentence-transformers==2.7.0 transformers==4.40.2 huggingface_hub==0.23.5 scikit-learn numpy
Run predictions
from predict import predict_ticket
result = predict_ticket("I was charged twice and need a refund immediately")
print(result)
# {"category": "billing", "confidence": 0.79, "urgency": "high"}
Files in This Repo
| File | Description |
|---|---|
predict.py |
Ready-to-run prediction script |
requirements.txt |
Pinned dependencies |
category_model/ |
Fine-tuned SetFit classifier |
calibration.pkl |
Platt scaling confidence calibration |
label_mappings.pkl |
Label encoders |
Tech Stack
- Model: SetFit (Sentence Transformers fine-tuning)
- Base model:
paraphrase-MiniLM-L3-v2 - Training data: 30,571 labeled support tickets
- Confidence calibration: Platt scaling on held-out validation set
- Urgency: Keyword-rule layer (transparent and auditable)
Get the Full Docker API Version
Want a production-ready REST API you can deploy to your own server in minutes?
The Docker version includes:
- FastAPI wrapper (
POST /predictendpoint) - Dockerfile — one command to deploy anywhere
- Full setup guide
👉 Get the Docker API version on Gumroad