diggiiiiiii/review-intel-sentiment

Customer review sentiment (negative / neutral / positive)

DistilBERT fine-tuned to read a customer review and say whether it is negative, neutral or positive. It powers the sentiment part of the Customer Review Intelligence Tool, a student project that summarises what a shop's customers like and dislike.

from transformers import pipeline

classify = pipeline("text-classification", model="diggiiiiiii/review-intel-sentiment")
classify("The food was great but we waited an hour for a table.")

Results

Tested on 14,996 reviews from shops the model never saw in training. Macro F1 weighs the three labels equally, so the rare neutral label counts as much as the common positive one.

Model	Macro F1	Negative F1	Neutral F1	Positive F1
DistilBERT (this model)	0.761	0.837	0.492	0.954
TF-IDF + logistic regression	0.732	0.812	0.445	0.941

Accuracy 88.7%. By shop type:

Shop type	Macro F1
appliances	0.738
grocery	0.746
restaurant	0.776

A shop type it never saw. Retrained without any appliance reviews, it still scores 0.699 on appliances, above the baseline trained with them (0.688).

Training

• Fine-tuned from distilbert-base-uncased: learning rate 5e-05, batch size 32, up to 256 word-pieces per review, at most 3 epochs.
• The best epoch was kept: epoch 2 of 3 kept (validation macro F1 0.766).
• Training run 9501c6c56b68: 27 minutes on a Tesla T4 (Kaggle's free GPU).
• The loss counts rare labels (neutral) for more, so the model doesn't just learn "positive".
• Labels come from star ratings: 1-2 stars negative, 3 neutral, 4-5 positive. Whole shops go to one split, so test shops were never seen in training.

Data

About 150,000 English reviews, 50,000 from each of:

• Yelp Open Dataset, restaurants (academic, non-commercial licence);
• Amazon Reviews 2023 (McAuley Lab), Grocery and Gourmet Food and Appliances (research use).

No review text is published with this model: only the weights and their scores.

Limits

• Neutral is hard (F1 0.49): 3-star reviews are often mixed, and many are really positive or negative.
• Star labels are noisy: a 5-star review can describe a fault.
• English only. Trained on restaurants, groceries and appliances; other shop types may score lower.
• Research and learning only. The weights come from Apache-2.0 distilbert-base-uncased, but the training data's licences don't allow commercial use, so neither does this model.