khizar7786/feedback-triage-distilbert

Feedback Triage DistilBERT

A fine-tuned DistilBERT classifier for English customer feedback and support tickets. The model predicts one of six categories, including a Non-Qualifying category for messages that should be excluded from further ticket analysis.

This model was developed as an internship proof of concept and learning project. It is a full sequence-classification checkpoint based on distilbert/distilbert-base-uncased.

Evaluation context: the reported results come from a small synthetic, assistant-reviewed dataset. They do not establish accuracy on real customer traffic.

Categories

Label ID	Category	Meaning
0	Bug Report	Existing functionality is broken or behaves incorrectly.
1	Feature Request	A request for new functionality or an enhancement.
2	Billing Issue	A charge, invoice, subscription, refund, or billing-account issue.
3	General Inquiry	A genuine question about using or understanding a product.
4	Customer Feedback	Praise, criticism, or an opinion without a more specific intent.
5	Non-Qualifying	Greeting-only text, spam, promotional broadcasts, or irrelevant content.

For mixed messages, the classifier predicts one main category. Category decisions should be reviewed when the message is ambiguous.

Intended use and application role

The intended use is local classification within a customer-feedback triage demonstration. The application first applies conservative Python rules to blank messages and exact simple greetings. Other messages reach this classifier. A Non-Qualifying prediction stops subsequent sentiment and generative analysis.

This checkpoint predicts category and semantic qualification only. In the full application, a separate pretrained RoBERTa model predicts sentiment, and a separate API-based LLM assigns priority, extracts the mentioned product or feature, and drafts a reply. Those outputs are not produced by this DistilBERT checkpoint.

Predictions support human review. Automatically discarding customer messages in production would require further evaluation and safeguards.

Application source: AI-Powered Customer Feedback & Support Ticket Analyzer PoC.

Dataset and review provenance

The project corpus contains 600 synthetic English messages, balanced across the six categories. Labels were reviewed and corrected by an assistant under explicit user delegation. They were not independently human-validated and are not real customer records.

Split	Messages	Purpose
Training	420	Update model weights.
Validation	90	Select the checkpoint using macro-F1.
Held-out test	90	Evaluate the selected checkpoint after choices were fixed.

The development and test pools were kept separate. Dataset preparation tracked provenance and related scenario groups, with reproducible group-aware splitting of the development pool. The test set contains 15 messages per category.

The held-out set was evaluated once after model selection. Saved results were reused for subsequent reporting; the test results were not used to tune the model. Assistant-authored and assistant-reviewed scenarios are not guaranteed to represent independent real-world customer traffic.

Dataset fingerprint (SHA-256):

c8f76e80e95337bdfd874780bcad16483a2db02a9539f05ab44794b8a2ae15bf

Training procedure

The pretrained DistilBERT encoder and six-class classification head were fine-tuned using PyTorch and Hugging Face Transformers. Only message text was used as model input; category labels supplied the training targets. IDs, provenance, review status, and scenario groups were excluded from model features.

Setting	Value
Base model revision	12040accade4e8a0f71eabdb258fecc2e7e948be
Epochs	3
Learning rate	2e-5
Optimizer	AdamW (PyTorch)
Weight decay	0.01
Maximum input length	128 tokens, including special tokens
Training / evaluation batch size	2 / 2
Gradient accumulation	4 steps; nominal effective batch size 8
Padding	Dynamic
Scheduler	Linear, with 10 warmup steps
Random seed	42
Training hardware	NVIDIA GeForce RTX 3050 Ti Laptop GPU, approximately 4 GB VRAM
Training precision	FP16
Checkpoint selection	Highest validation macro-F1; epoch 3 selected

Recorded environment: Python 3.12.5, PyTorch 2.11.0+cu128, Transformers 4.57.6, Datasets 4.8.5, and Accelerate 1.15.0. CPU inference was also exercised.

Evaluation results

The final held-out evaluation was performed on 2026-10-26 using CPU inference.

Measurement	Result
Selected checkpoint validation macro-F1	0.9305
Held-out test accuracy	98.89% (89/90 correct)
Held-out test macro-F1	0.9889
Held-out messages with successful predictions	90/90
Qualifying test messages incorrectly classified as Non-Qualifying	0/75
Non-Qualifying test messages incorrectly accepted	0/15

Category	Test precision	Test recall	Test F1	Support
Bug Report	1.0000	1.0000	1.0000	15
Feature Request	1.0000	0.9333	0.9655	15
Billing Issue	1.0000	1.0000	1.0000	15
General Inquiry	0.9375	1.0000	0.9677	15
Customer Feedback	1.0000	1.0000	1.0000	15
Non-Qualifying	1.0000	1.0000	1.0000	15

The one held-out classification error was a Feature Request predicted as General Inquiry. Non-Qualifying recall on the earlier validation set was 0.7333 (11/15), illustrating that perfect results on a small test subset do not establish reliable filtering in other settings.

These measurements evaluate category classification only. They do not measure sentiment accuracy, priority quality, product extraction, reply quality, or end-to-end application accuracy. The test set is small and balanced, and the scores should not be presented as production performance guarantees.

Local inference

Download the model files from this repository into models/ticket_classifier/ under the application project. After installing the project dependencies, the following example runs entirely from the local checkpoint:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_dir = "models/ticket_classifier"
tokenizer = AutoTokenizer.from_pretrained(
    model_dir, local_files_only=True, trust_remote_code=False
)
model = AutoModelForSequenceClassification.from_pretrained(
    model_dir,
    local_files_only=True,
    trust_remote_code=False,
    use_safetensors=True,
)
model.eval()

message = "The payment page crashes when I click Checkout."
normalized_message = " ".join(message.split())
inputs = tokenizer(
    normalized_message,
    return_tensors="pt",
    truncation=True,
    max_length=128,
)

with torch.inference_mode():
    probabilities = model(**inputs).logits.softmax(dim=-1)

label_id = int(probabilities.argmax(dim=-1).item())
print({
    "category": model.config.id2label[label_id],
    "confidence": float(probabilities[0, label_id].item()),
})

The example prints the actual prediction; no expected output is hardcoded. Softmax confidence is a model score, not a calibrated guarantee of correctness. The application separately records input truncation and retains the full original message.

Files required by the application

Upload these files at the root of the Hugging Face model repository:

README.md
model.safetensors
config.json
tokenizer.json
tokenizer_config.json
special_tokens_map.json
vocab.txt
label_mappings.json
training_summary.json

training_summary.json is required by the application's checkpoint checks, in addition to the standard model and tokenizer files. Intermediate training checkpoints and training_args.bin are not required for inference.

Model weights SHA-256:

993d236aa4473be875c004a9a1c8256271286349b7c929eb67bbb7aa5c087b06

Limitations

• Training and evaluation use a small synthetic corpus with assistant review, not independent human validation or real customer traffic.
• Performance on unfamiliar products, writing styles, typos, languages, and actual spam may differ substantially. Non-English use was not evaluated.
• Only the first 128 tokens, including special tokens, are used. An issue appearing later in a long message may be missed.
• The model predicts one category, so messages containing several different requests can be ambiguous.
• Confidence calibration, demographic fairness, and adversarial robustness were not evaluated. The checkpoint may retain biases from its base model and synthetic training examples.
• A mistaken Non-Qualifying prediction can hide a genuine customer issue. Human review is appropriate before consequential filtering decisions.

Evidence and base model attribution

The results and settings above were read from the project's saved training_summary.json, reports/intent_metrics.json, and training/evaluation documentation. No additional training or held-out evaluation was performed to write this card.

The base model's documentation and license information are available in the DistilBERT base model card.