meutsabdahal/nepali-ner-roberta

A Nepali named-entity recognition model: IRIISNEPAL/RoBERTa_Nepali_110M fine-tuned on EverestNER. Published as one row of a six-encoder ablation comparing base models under an identical pipeline and protocol. See the benchmark repo for the full comparison, method, and error analysis.

Model description

• Base model: IRIISNEPAL/RoBERTa_Nepali_110M
• Task: token classification (NER), 5 entity types: Person, Location, Organization, Date, Event (IOB2 scheme, 11 labels including O)
• Training data: EverestNER (~15.8k sentences of Nepali news text)
• Checkpoint: seed 13 of 3 trained seeds (13, 21, 42); this seed was selected as the one closest to the across-seed mean micro-F1, so it is representative rather than a best-of-N pick

Intended uses and limitations

Intended for named-entity recognition on Nepali text, particularly news-domain text similar to EverestNER's source material (political, geographic, and organizational names skew are expected to be well covered; other domains may see degraded performance).

EVENT is unreliable and should not be trusted as a ranking signal. The EverestNER test set has only 59 Event instances, so its F1 (see table below) carries a wide margin of error and swung between 0.25 and 0.61 across the six models compared in the parent benchmark. Do not build a claim on this model's Event performance specifically.

This model is a research/portfolio artifact, not a production service. It has not been evaluated outside the EverestNER domain.

Training procedure

Identical protocol across every model in the ablation (no per-model hyperparameter tuning, so comparisons between base models are not confounded by which one got a luckier learning rate):

hyperparameter	value
learning rate	5e-05
epochs	4
batch size	16
weight decay	0.01
warmup ratio	0.1
max sequence length	256
checkpoint selection	best dev micro-F1 (seqeval, strict IOB2)
seeds	13, 21, 42

Evaluation results

EverestNER test set (1,950 sentences), seqeval strict IOB2. Mean +- std across 3 seeds (optimization variance):

metric	value
micro-F1	0.9185 +- 0.0020
macro-F1	0.8557 +- 0.0024

Per-entity (mean across seeds):

entity	precision	recall	f1
PER	0.957	0.938	0.947
LOC	0.916	0.906	0.911
ORG	0.895	0.904	0.900
DATE	0.944	0.923	0.933
EVENT	0.562	0.616	0.587

On this seed's own checkpoint, a sentence-level bootstrap over the test set gives a 95% CI of [0.9093, 0.9282] on micro-F1 (test-sampling variance; distinct from the seed mean/std above, which is optimization variance; the two are not combined).

How to use

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("meutsabdahal/nepali-ner-roberta")
model = AutoModelForTokenClassification.from_pretrained("meutsabdahal/nepali-ner-roberta")
ner = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
ner("काठमाडौं महानगरपालिकाले नयाँ योजना ल्यायो।")

License

EverestNER is released for non-commercial use only. This model is a derivative fine-tuned on that data and inherits the same restriction; it is not licensed for commercial use. The training/evaluation code for this project is separately licensed under Apache-2.0; that code license does not extend to this model's weights. See the dataset's own repository for the exact terms.

Citation

If you use this model, please credit EverestNER and the base encoder:

• EverestNER: https://github.com/nowalab/everest-ner
• Base model: IRIISNEPAL/RoBERTa_Nepali_110M