Maithili Causal Language Model (109.5M)
A decoder-only transformer pretrained from scratch on the Maithili language corpus.
Architecture
| Component | Detail |
|---|---|
| Type | Decoder-only causal transformer |
| Parameters | ~109.5M |
| Layers | 12 |
| Hidden dim | 768 |
| Attention heads | 12 |
| Head dim | 64 |
| Position encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Feed-forward | SwiGLU |
| Vocab size | 32,000 |
| Max sequence length | 1024 |
| Tokenizer | Byte-level BPE (purpose-built for Maithili) |
Training
| Detail | Value |
|---|---|
| Training data | AiventraLab/maithili-corpus (Maithili gold corpus) |
| Total training tokens | ~983,040,000 |
| Steps | 30000 |
| Effective batch size | 32,768 tokens |
| Precision | fp16 mixed precision |
| Optimizer | AdamW (betas=0.9/0.95, weight_decay=0.1) |
| LR schedule | Cosine with linear warmup (500 steps) |
| Peak LR | 3e-4 |
| Gradient checkpointing | Yes |
| Hardware | 1x NVIDIA T4 (16GB) |
Evaluation
| Metric | Value |
|---|---|
| Final validation loss | 3.1215 |
| Final validation perplexity | 22.68 |
Validation Loss Over Training
[EVAL] step 26000: val_loss=3.1731, val_ppl=23.88 [EVAL] step 27000: val_loss=3.2049, val_ppl=24.65 [EVAL] step 28000: val_loss=3.2059, val_ppl=24.68 [EVAL] step 29000: val_loss=3.2060, val_ppl=24.68 [EVAL] step 30000: val_loss=3.1897, val_ppl=24.28
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support