nrr26/yelp-complaint-tagger

Yelp complaint tagger

Tags why a restaurant got a bad review. Give it a 1-2 star review and it returns one or more of 8 complaint types: wait_time, staff_attitude, food_quality, order_accuracy, price_value, hygiene_safety, ambience, billing_policy.

A small (66M parameters) DistilBERT model, fine-tuned on about 2,000 Yelp reviews labelled by an open LLM. Built as a learning project.

Try it without code: Restaurant Complaint Tagger Space · Code and notebook: GitHub

Use it in Claude or ChatGPT: add a custom MCP connector with https://nrr26-complaint-tagger.hf.space/gradio_api/mcp/

Code and notebook: https://github.com/namsarena/yelp-complaint-tagger

Quick start

Each label has its own threshold, stored in config.label_thresholds. Use them; plain 0.5 gives worse results.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "nrr26/yelp-complaint-tagger"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
th = model.config.label_thresholds

text = "We waited 45 minutes and the burger was cold."
with torch.no_grad():
    probs = torch.sigmoid(model(**tok(text, return_tensors="pt", truncation=True)).logits)[0]
print({label: round(float(p), 2) for label, p in zip(model.config.id2label.values(), probs) if p > th[label]})

Intended use

• Use it on reviews already known to be negative (1-2 stars) about restaurants, bars, cafes and similar places.
• Good for spotting patterns across many reviews, such as "30% of our complaints are about wait time".
• Not for: positive or mixed reviews (it was never trained on them and will invent complaints), non-restaurant businesses, languages other than English, or decisions about individual staff members.

Labels and results

Test set: 354 held-out reviews labelled by the same LLM. F1 is between 0 and 1; higher is better. The baseline is TF-IDF + logistic regression.

Label	Threshold	Baseline F1	This model F1	Test examples
wait_time	0.60	0.72	0.79	109
staff_attitude	0.30	0.74	0.78	121
food_quality	0.65	0.86	0.87	239
order_accuracy	0.35	0.25	0.60	39
price_value	0.40	0.67	0.73	97
hygiene_safety	0.35	0.52	0.71	42
ambience	0.45	0.42	0.64	57
billing_policy	0.60	0.46	0.61	24

Average	Baseline	This model
micro-F1	0.71	0.77
macro-F1	0.58	0.71

How it was built

1. Data: 3,500 random 1-2 star reviews (200-1,500 characters) from Yelp/yelp_review_full, seed 42.
2. Labels (silver): Qwen/Qwen3-4B-Instruct-2507 labelled each review with a written guide (8 labels plus rules) and 5 invented few-shot examples. It dropped 1,138 non-restaurant reviews, leaving 2,361.
   • Checked against 10 hand-labelled reviews: 70% precision and 70% recall per label, 9/10 on restaurant-or-not.
   • Spot check of 20 fresh reviews: about 18/20 judged correct.
3. Training: DistilBERT, 8 epochs, learning rate 5e-5, weighted loss (rare labels up to 10x), best epoch chosen by v7).
   • Split: 1,800 train / 200 validation / 354 test.
   • Per-label thresholds tuned on validation.

Limitations

• It copies its teacher. The training and test labels come from an LLM, so the scores above measure agreement with tOn 6 hand-labelled reviews it matched exactly on 2 and repeated the LLM's habits: tagging praised ambience, reading slowor sloppy service as rude, and under-tagging price_value.
• Negation: "the staff was not rude" can still be tagged staff_attitude.
• Positive reviews: it assumes every text is a complaint.
• staff_attitude over-fires with weak scores (about 0.3); its threshold was tuned on only 200 validation reviews.
• Secondary complaints (sold-out items, "had to file a dispute") are sometimes missed.
• Rare labels (billing_policy, order_accuracy) have only 24-39 test examples, so their scores are noisy.
• Process note: the test set was looked at once before the weighted-loss retrain. All later choices used the validation set only.

License and credits

• Model weights: CC BY-NC 4.0 (non-commercial), because the training text comes from Yelp, whose dataset terms allow non-commercial use. The training reviews are not redistributed.
• Base model: DistilBERT (Apache 2.0). Labelling model: Qwen3-4B-Instruct (Apache 2.0).