ezechinnabugwu/pidgin-sentiment-model

Pidgin Sentiment Model (Group 28 ML/DL Project)

AfriBERTa-large fine-tuned to classify Nigerian Pidgin text as negative, neutral or positive.

Built by Group 28 (TechCrush ML/DL Project, supervised by Mr Lamzey):

• Ezechinyere Nnabugwu Kingsley (project lead): sourced the round-1 training data and the NaijaSynCor data; data pipeline, model training and evaluation, web app, deployment

• Oguche Charles Arome: researched and wrote Pidgin training sentences; graded slang as positive, negative or neutral

• Caleb Bassey Bassey: researched and wrote Pidgin training sentences; graded slang as positive, negative or neutral

• Live demo: https://pidgin-sentiment.streamlit.app

• Code, notebook and full evaluation: https://github.com/Ezechis/pidgin-sentiment

Training data

• NaijaSenti, Nigerian Pidgin (pcm) tweets (Muhammad et al., 2022, CC BY 4.0). Duplicates and every dev/test tweet that also appears in training were removed.
• 626 extra training sentences, added to training only:
  • 280 sentences (BBC News Pidgin and AI-drafted customer messages) checked by the three group members;
  • 87 sentences researched, written and labelled by group members Oguche Charles Arome and Caleb Bassey Bassey;
  • 260 spoken sentences from the NaijaSynCor treebank (UD_Naija-NSC, CC BY-SA 4.0), labelled from their English translations.

Evaluation

The cleaned NaijaSenti test set (3,228 tweets) was split in half, stratified by label: one half chose the model, the other half (1,614 tweets) was used only for the scores below.

Model	Macro-F1	Neutral F1
This model (round 2, best of three seeds on the selection half)	0.528	0.32
Round 1 (280 extra sentences), mean of three seeds	0.481	0.20

Use

from transformers import pipeline

clf = pipeline("text-classification", model="ezechinnabugwu/pidgin-sentiment-model",
               revision="e9f06b70eff07313906da006b547c2003d8ec7b7", top_k=None)
clf("abeg wetin be una opening hours")

The training tweets were lower-cased with URLs, handles, digits, punctuation and emoji removed, so normalise input the same way for the best results.

Limitations

• Neutral is still the weakest class.
• Some complaints phrased as requests ("make una send my token code, e no dey drop") and curses ("una go see shege") are read as neutral.
• Trained on tweets; performance on chats, reviews or voice transcripts is untested.