klohdfhk/amharic-sentiment-ai

Amharic Sentiment AI

A three-class Amharic sentiment classifier trained on the official AfriSenti Amharic benchmark. The release contains model files, code for inference, and reproducibility metadata; it does not contain source tweets. Source code: GitHub repository.

Labels

• positive (ID 0)
• neutral (ID 1)
• negative (ID 2)

Training and selection

• Selected approach: fine-tuned-transformer (transformer)
• Selection: highest validation macro-F1 across candidates; test results did not choose the released model.
• Final fit: fresh refit on train+validation using settings or epoch count selected on validation; no test use during selection or refitting.
• Seed: 42
• Base model: [Davlan/afro-xlmr-mini] (https://huggingface.co/Davlan/afro-xlmr-mini)
• Base revision: bc04038b969667884bd83cdd37bed9559c2c3d9c
• Dataset: AfriSenti Amharic, pinned commit 5aec3cfcf87b59f885e98641f72e52182776429d; license CC BY 4.0.
• Source SHA-256 and split-overlap audit: data_audit.json.

Held-out test results

The official test split contains 1,999 posts. Macro-F1 is emphasized because class frequencies are imbalanced.

Metric	Score
Accuracy	0.460
Macro-F1	0.466
Weighted-F1	0.521
Balanced accuracy	0.569
Macro-F1 95% bootstrap interval	0.444–0.487

Class	Precision	Recall	F1	Support
positive	0.718	0.477	0.573	438
neutral	0.167	0.839	0.279	224
negative	0.894	0.391	0.544	1337

The test labels are distribution-shifted: 66.9% negative and 11.2% neutral, versus about 25.9% negative and 51.9% neutral in train/validation. Classical and neural test scores were also observed in an earlier interrupted run with identical settings; no hyperparameters were changed from those scores. The bootstrap interval reflects test-row resampling for this fixed model, not variation across training seeds.

Example inference

With Python 3.12, install the pinned dependencies, then download this model repo:

pip install -r requirements.lock --extra-index-url https://download.pytorch.org/whl/cpu

from pathlib import Path
import sys
from huggingface_hub import snapshot_download
repo_id = "klohdfhk/amharic-sentiment-ai"
model_type = "transformer"
model_dir = Path(snapshot_download(repo_id=repo_id))
sys.path.insert(0, str(model_dir / 'src'))
from amharic_sentiment.inference import SentimentPredictor
if model_type == 'transformer':
    artifact = model_dir
else:
    filename = 'model.joblib' if model_type == 'classical' else 'neural_weights.npz'
    artifact = model_dir / filename
predictor = SentimentPredictor(model_type, artifact)
print(predictor.predict_label("ይህ በጣም ጥሩ ነው"))

Demo availability

Try the free browser-only Static Space. It runs the quantized ONNX model on your device, without an inference server or API token.

Browser-native INT8 variant

The Static Space uses a dynamically quantized, per-channel INT8 ONNX variant (118.5 MB) with Transformers.js. The model runs in the visitor's browser; no inference token or server is used. First use downloads about 141 MB of model and tokenizer files.

Variant	Validation macro-F1	Test accuracy	Test macro-F1	Test label agreement with full model
Full-precision transformer	0.536	0.460	0.466	—
Browser INT8 (per-channel)	0.659	0.442	0.456	94.5%

The INT8 variant was selected using validation results only. Its test results are reported separately because quantization changes some individual outputs; the browser demo shows only the predicted class, not an uncalibrated score. See browser_model_evaluation.json for full per-class metrics.

Intended use and limitations

This model is for research and educational demonstrations of Amharic sentiment classification. Social-media sentiment may not generalize across domains or time. Sarcasm, dialects, spelling variation, code-switching, cultural context, and annotation disagreement can affect results. Predictions are not calibrated probabilities and must not be used for high-impact decisions or profiling.

Data, attribution, and licensing

AfriSenti was created for SemEval-2023 Task 12; the upstream repository states CC BY 4.0. Please cite and attribute the dataset authors; see Muhammad et al., EMNLP 2023. This release does not bundle tweets. The base encoder's license and attribution remain applicable.

Original project code is MIT licensed. Fine-tuned weights are released under CC BY 4.0 with attribution. This is a project release choice, not legal advice.

Reproducibility

This card is generated from the benchmark results and the accompanying training config and audit. It reports one fixed-seed run; it does not claim multi-seed statistical significance.