Indic-Lite: Unified Indic NLP Suite

Indic-Lite is a compact, high-efficiency Indic NLP toolkit designed for edge, mobile, and server environments. It provides lightweight models and text utilities:

Indic-LID: 28-Language Identification (covering Northeast languages, all 22 Eighth Schedule languages, regional dialects, and English)
Indic-Sentiment: 12-Language Sentiment Classifier
Indic-Topic: 6-Class Regional News & Headline Classifier
Indic-Clean: Rule-based Unicode NFC normalization, Indic numeral translation (e.g. १२३ → 123), and PII scrubbing.

Supported Languages

1. `Indic-LID` (Language Identification — 28 Languages)

Northeast Indian Languages (6)

Assamese (as) — Assam (Bengali-Assamese script)
Bodo (brx) — Bodoland / Assam (Devanagari script)
Khasi (kha) — Meghalaya (Latin script)
Manipuri / Meitei (mni) — Manipur (Meetei Mayek & Bengali script)
Mizo (lus) — Mizoram (Latin script)
Santali (sat) — Assam, Tripura, Odisha (Ol Chiki script)

Scheduled National Languages (15)

Bengali (bn) — West Bengal, Tripura
Gujarati (gu) — Gujarat
Hindi (hi) — Central & Northern India
Kannada (kn) — Karnataka
Konkani (gom) — Goa, Maharashtra, Karnataka
Maithili (mai) — Bihar, Jharkhand
Malayalam (ml) — Kerala
Marathi (mr) — Maharashtra
Nepali (ne) — Sikkim, West Bengal
Odia (or) — Odisha
Punjabi (pa) — Punjab
Sanskrit (sa) — Pan-India
Tamil (ta) — Tamil Nadu
Telugu (te) — Andhra Pradesh, Telangana
Urdu (ur) — Pan-India

Regional Dialects & Literary Varieties (6)

Awadhi (awa) — Uttar Pradesh
Bhojpuri (bho) — Bihar, Uttar Pradesh
Chhattisgarhi (hne) — Chhattisgarh
Garhwali (gbm) — Uttarakhand
Haryanvi (bgc) — Haryana
Marwari (mwr) — Rajasthan

Associate Official Language (1)

English (en) — India-wide

2. `Indic-Sentiment` (Sentiment Analysis — 12 Languages)

Assamese (as), Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (kn), Malayalam (ml), Marathi (mr), Odia (or), Punjabi (pa), Tamil (ta), Telugu (te), Urdu (ur).

3. `Indic-Topic` (News & Headline Categorization — 6 Languages)

Gujarati (gu), Hindi (hi), Malayalam (ml), Marathi (mr), Tamil (ta), Telugu (te).
Categories (6): entertainment, business, tech, sports, state, spirituality.

Benchmark Evaluation

Evaluated on standardized held-out test splits:

Model	Task / Coverage	Test Set Size	Accuracy	Macro F1	Joblib Size	ONNX Size	Latency / Item
Indic-LID	Language ID (28 languages)	27,312	99.53%	99.54%	10.83 MB	4.27 MB	0.155 ms
Indic-Sentiment	Sentiment (12 languages)	10,129	75.84%	75.84%	1.64 MB	0.19 MB	0.164 ms
Indic-Topic	Topic (6 categories)	3,244	90.23%	89.35%	3.41 MB	1.03 MB	0.135 ms

Quickstart

Installation

pip install joblib onnxruntime scikit-learn

1. Python Usage

import joblib

# Load models
lid = joblib.load("indic_lid_model.joblib")['model']
sentiment = joblib.load("indic_sentiment_model.joblib")['model']
topic = joblib.load("indic_topic_model.joblib")['model']

# Language Detection (28 languages including Northeast)
print(lid.predict(["Chibai, eng nge i an?"]))   # ['lus'] (Mizo)
print(lid.predict(["Khublei shibun!"]))        # ['kha'] (Khasi)
print(lid.predict(["வணக்கம் நண்பா"]))          # ['ta'] (Tamil)

# Sentiment Analysis
sent = sentiment.predict(["यह उत्पाद बहुत ही शानदार है!"])[0]
print("Sentiment:", "Positive" if sent == 1 else "Negative")

# News Topic Classification
top = topic.predict(["विराट कोहलीने झळकावले शतक"])[0]
print("Topic:", top)  # 'sports'

2. ONNX Runtime

Every model has a matching .onnx checkpoint inside onnx/ allowing execution across platforms (C++, Python, Go, Rust, mobile):

onnx/indic_lid.onnx (4.27 MB)
onnx/indic_sentiment.onnx (0.19 MB)
onnx/indic_topic.onnx (1.03 MB)

Provenance

Indic-LID: Sourced from google/IndicGenBench_flores_in, robzchhangte/mizo-corpus-passages, and damerajee/khasi-datasets.
Indic-Sentiment: Sourced from mteb/IndicSentiment and IIT Patna reviews (ai4bharat/indic_glue).
Indic-Topic: Sourced from India Today Regional News (inltkh.*) and BBC Hindi (bbca.hi).

License

Creative Commons Attribution 4.0 International (CC-BY-4.0).

Downloads last month: -; Downloads are not tracked for this model. How to track