You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

AraNeoBERT v3 (dialect-augmented)

A ~250M-parameter modern Arabic encoder (NeoBERT/ModernBERT-style) pretrained with MLM on an AraMix web backbone plus MSA news/books, augmented with ~17% dialectal data (FineWeb-2 dialect slices: Egyptian/Moroccan/Najdi/Levantine/Iraqi + filtered tweets). 28 layers, hidden 768, 12 heads; alternating local/global attention (window 128, global every 3 layers), QK-Norm, RoPE (dual theta 10k local / 160k global), StableAdamW + WSD, three-phase length curriculum to 4096. 64k SentencePiece Unigram tokenizer with web_canonical normalization baked into the fast tokenizer.

Held-out AraMix MLM pseudo-perplexity: 16.82 @ seq 4096.

Usage

This model uses custom modeling code — load with trust_remote_code=True:

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("miweru/AraNeoBERT-v3-dialect")
model = AutoModelForMaskedLM.from_pretrained("miweru/AraNeoBERT-v3-dialect", trust_remote_code=True)

AutoModel, AutoModelForMaskedLM, AutoModelForSequenceClassification and AutoModelForTokenClassification heads are registered.

Downstream benchmark (finetuned, identical recipe across models)

Task	AraNeoBERT-v3	AraBERT v0.2	AraModernBERT	MARBERTv2
ANERcorp NER (entity-F1)	0.815	0.823	0.714	0.807
ASTD sentiment (macro-F1)	0.561	0.568	0.540	0.637
ArSarcasm sentiment (macro-F1)	0.709	0.735	0.705	0.752
AJGT sentiment (acc)	0.911	0.942	0.917	0.950

Strongest on MSA/NER (beats AraModernBERT decisively, edges MARBERTv2); dialectal social-media sentiment is competitive but trails the dialect specialists.

Research artifact; private. Built in cooperation with SinaLab for Computational Linguistics and Artificial Intelligence.

Downloads last month: -

Safetensors

Model size

0.3B params

Tensor type

C64

F32