saketgarodia1/bert-it-ticket-student

📘 DistilBERT IT Ticket Classifier (Student Model – Knowledge Distilled)

This is a DistilBERT student model fine-tuned via knowledge distillation from the teacher model:

👉 saketgarodia1/bert-IT-ticket-classifier-full

The goal is to create a lighter, faster IT ticket classifier while retaining high performance.

---

🔥 Results (Teacher vs Student)

⚠️ Replace the X values below once you finalize the test metrics.

Model	Accuracy	Macro F1	Notes
Teacher (BERT Base)	0.9374	0.9362	~110M params
Student (DistilBERT)	0.93X	0.93X	~67M params, ~40% smaller & faster

---

🧠 Distillation Setup

The student is trained using a combined loss:

[ \mathcal{L} = \alpha \cdot \text{CE}(\text{student}, \text{labels}) ;+; (1 - \alpha) \cdot \text{KL}(\text{student}, \text{teacher}) ]

Key settings:

• α (alpha): 0.3 → more weight on hard labels
• Temperature (T): 2 → KL on softened logits
• Loss components:
  • Cross-entropy (CE) between student logits and ground-truth labels
  • KL divergence between softened student logits and teacher logits
• Warmup ratio: 10%
• Optimizer: AdamW
  • Learning rate: 2e-5
  • Weight decay: 0.01
• Epochs: 3
• Batch size: 16 / 32
• Teacher: frozen; used only to generate logits (no gradient updates)

---

🧰 Intended Use

• Classifying IT support tickets into 8 categories
• Suitable for smaller GPUs / CPUs
• Real-time or near real-time inference scenarios
• Internal IT help desk routing, analytics, or triage dashboards

---

🚫 Limitations

• Inherits potential biases from both BERT and DistilBERT
• Assumes English-only input text
• Not designed for long documents (> 512 tokens)
• Domain-specific: trained on IT ticket text, may not generalize perfectly to other domains

---

📦 How to Use the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "saketgarodia1/bert-it-ticket-student"

model = AutoModelForSequenceClassification.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

text = "VPN not connecting to corporate WiFi"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

pred_class_id = logits.argmax(dim=-1).item()
print("Predicted class id:", pred_class_id)