mahwizzzz/avey-b-ur

Avey-B Urdu 25M

A compact, fast Urdu encoder trained from scratch for masked-language modeling.

Highlights

• 78.53% ± 0.51 sentiment accuracy, essentially level with 278M-parameter XLM-R base in this setup.
• 81.27 WikiANN Urdu entity F1 with only 24.87M parameters.
• 1.71 ms encoder latency at batch 1 / length 128 and 148 MiB peak VRAM at batch 16 / length 512.
• 14.29 held-out MLM perplexity, 50.46% top-1, and 70.39% top-5 masked-token accuracy.
• Compact Urdu tokenization: 4.19 characters/token on held out data.

Quick start

This model contains a custom Avey architecture, so trust_remote_code=True is required.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "mahwizzzz/avey-b-ur"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True).eval()

text = "پاکستان کا دارالحکومت [MASK] ہے۔"
inputs = tokenizer(text, return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits

mask_index = (inputs.input_ids[0] == tokenizer.mask_token_id).nonzero()[0, 0]
top_ids = logits[0, mask_index].topk(5).indices
print(tokenizer.convert_ids_to_tokens(top_ids.tolist()))

Extract contextual token representations with AutoModel:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mahwizzzz/avey-b-ur", trust_remote_code=True)
encoder = AutoModel.from_pretrained("mahwizzzz/avey-b-ur", trust_remote_code=True)
hidden = encoder(**tokenizer("یہ ایک اردو جملہ ہے۔", return_tensors="pt")).last_hidden_state
print(hidden.shape)  # [batch, tokens, 384]

Evaluation

Model	Parameters	Sentiment accuracy	WikiANN Urdu NER F1
Avey-B Urdu	24.87M	78.53% ± 0.51	81.27
UrNova	94.72M	81.83% ± 0.42	87.79
HPLT Urdu BERT	124.36M	83.93% ± 1.27	90.92
XLM-R base	278.04M	78.80% ± 0.52	84.89

Training

Item	Value
Corpus	HPLT 3.0 Urdu (urd_Arab), quality bins 10 and 9
Clean data	31,297 train docs, 151 validation docs, 541,165,369 characters
Sequence length	512
Token batch	16,384 tokens/step
Training	20,000 steps, BF16, fused AdamW
Peak learning rate	7e-4
Hardware/runtime	NVIDIA RTX 4060 8GB, approximately 20 minutes
Final train loss	2.52

Intended use and limitations

Use this checkpoint for Urdu classification, token classification, representation extraction, or continued pretraining. It is an encoder, not a chat model or free-text generator. Benchmark coverage is limited to masked-token prediction, machine-translated movie-review sentiment, and Wikipedia-derived NER; evaluate on native, domain-specific Urdu before production use.

HPLT packages its dataset under CC0 but does not own the underlying web text. The Apache-2.0 model license does not grant rights to third-party source documents that may be reproduced by the model.

Architecture and citation

Based on Avey-B, an attention-free bidirectional encoder architecture.

@inproceedings{2026aveyb,
  title={Avey-B},
  author={Acharya, Devang and Hammoud, Mohammad},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026}
}

Training provenance is available in MANIFEST.json. Model code is included for reproducible loading.