skyblanket/tamil-800m-base

Tamil-800M (base)

A from-scratch 800M-parameter Tamil language model — not a fine-tune of an existing multilingual model. Trained on 4B tokens of Tamil text with a custom 32k Tamil BPE tokenizer, on 2× RTX PRO 6000 Blackwell.

Headline result: on held-out Tamil text, this 800M model compresses Tamil better than Sarvam-1 (2.5B), a strong multilingual Indic baseline — at ~3× fewer parameters.

Model	Params	Tamil BPB ↓	Tamil PPL ↓	bytes/token
Tamil-800M (this model)	653M	0.2791	2.77	5.27
Sarvam-1	2.5B	0.3346	12.37	10.84

BPB = bits per byte on held-out Tamil (Aya split); lower is better. A dedicated Tamil tokenizer and Tamil-only training budget go a long way at small scale.

Architecture

Llama-style decoder, 24 layers, hidden 1536, 12 heads (4 KV, GQA), SwiGLU, context 2048, tied embeddings, vocab 32k (custom Tamil BPE). bfloat16.

Training

• 4.0B Tamil tokens, WSD schedule
• Corpus: CulturaX (ta), AI4Bharat Sangraha, Tamil Wikipedia/Wikisource/Wiktionary
• Single stage-1 pretraining run (this is stage 1 of an ongoing scaling series; 1B/3B in progress)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("skyblanket/tamil-800m-base")
model = AutoModelForCausalLM.from_pretrained("skyblanket/tamil-800m-base", dtype="bfloat16", device_map="auto")
ids = tok("தமிழ்நாடு என்பது", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**ids, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.2)[0]))

Limitations

• Base model — continues text, does not follow instructions (see the instruct version).
• Fluent Tamil, but factual reliability is weak at this scale — do not use for factual QA without retrieval.
• 2048-token context. Tamil only.

Personal research project by skyblanket.