twnar/azerbaijani_sentiment_analysis

Azerbaijani Sentiment Analysis (XLM-RoBERTa)

xlm-roberta-base fine-tuned for 3-class sentiment classification (negative / neutral / positive) on Azerbaijani Google Play Store reviews.

Model Details

• Developed by: twnarx
• Model type: XLM-RoBERTa encoder with a sequence classification head
• Language: Azerbaijani (az)
• License: MIT
• Finetuned from: FacebookAI/xlm-roberta-base
• Labels: negative, neutral, positive

Intended Use

Classifying the sentiment of short, informal Azerbaijani text, especially app and service reviews (banking, taxi, telecom, shopping apps).

Out-of-Scope Use

• Other domains such as news, political text, or formal writing were not tested and may perform worse.
• Not suitable for automated decisions that affect people.
• Text in other languages or heavily code-switched text.

How to Use

from transformers import pipeline

clf = pipeline("text-classification", model="twnarx/az-sentiment-xlmr", top_k=None)
print(clf("Bu tətbiq çox yaxşıdır!"))

Training Data

hajili/azerbaijani_review_sentiment_classification (MIT), about 160K Azerbaijani Play Store reviews with 1-5 star ratings.

Star ratings were mapped to labels: 1-2 → negative, 3 → neutral, 4-5 → positive.

The original data is heavily imbalanced (about 85% positive, 13% negative, 2% neutral), so balanced sampling was used:

• Train: 12,763 examples (5,000 negative, 2,763 neutral, 5,000 positive)
• Test: 1,800 examples (600 per class), sampled from the dataset's test split

Training Procedure

Setting	Value
Epochs	3
Learning rate	2e-5
Batch size	16
Max sequence length	128
Precision	fp16 mixed precision
Hardware	Google Colab, NVIDIA T4
Training time	about 15 minutes

No text cleaning was applied beyond removing empty reviews.

Evaluation

Evaluated on the balanced test set (1,800 examples):

Class	Precision	Recall	F1
negative	0.613	0.813	0.699
neutral	0.690	0.352	0.466
positive	0.741	0.862	0.797

Accuracy: 0.676 | Macro-F1: 0.654

Bias, Risks, and Limitations

• Noisy labels: Labels come from star ratings, not manual annotation. Some clearly positive reviews (e.g. "əla") carry 1-2 stars, so part of the measured error is label noise.
• Weak neutral class: 3-star reviews are mostly complaints or questions, so the model often predicts them as negative (neutral recall 0.35).
• Domain bias: Trained only on app reviews. Performance on other domains is unknown.
• Spelling variation: Reviews mix Azerbaijani letters and ASCII spellings (əla / ela). The model handles common variants, but unusual spellings, slang, and emoji-only text are less reliable.
• Results were measured on a balanced test set; real-world class distributions differ and will give different precision values.

Environmental Impact

• Hardware: NVIDIA T4 (Google Colab)
• Hours used: about 0.25