LiLM1-230M-base

LiLM1-230M-base is the pretrained LiLM1 checkpoint. It has 233,897,728 parameters and a 4,096-token context. The public name uses the nearest 10M parameter class.

This is a base model. It is not instruction-tuned.

Model details

Field	Value
Developer	Akshay Gautam
Model repository	glouriousgautam/LiLM1-230M-base
Parameters	233,897,728
Context length	4,096 tokens
Architecture	Hybrid decoder: 10 gated causal-convolution blocks and 6 GQA blocks
Pretraining	40B token exposures
Standardized training compute	75.1 to 75.6 GPU-hours on one RTX PRO 6000 Blackwell
Tokenizer	SmolLM2 135M tokenizer, revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2
Source checkpoint SHA-256	a0c7627df310a21ac1ceeedbce3694084a6c8e36deb35bbd2252fb682550d963
Safetensors SHA-256	2964530e470d8e1b942e5c9ad8d67bd3387a07f358831e1fdf785adfef151908
License	Apache-2.0

The hybrid stack uses width 1,024, SwiGLU intermediate size 2,560, and tied input and output embeddings. Convolution layers use kernel size 3. Attention layers sit at depths 2, 5, 8, 11, 13, and 15. Attention uses 16 query heads, 4 key/value heads, head dimension 64, and Q/K RMSNorm. Position encoding is RoPE with base theta 10,000.

Training

The base model used 40B token exposures at 4,096 context. Training used BF16 on one NVIDIA RTX PRO 6000 Blackwell GPU. The recorded steady rate was 147K to 148K tokens per second. This gives a standardized pretraining estimate of 75.1 to 75.6 GPU-hours.

The public pretraining corpus is not available.

Evaluation

The 40B base checkpoint scores 45.22% on an 11-task zero-shot aggregate. WikiText word perplexity is 26.71. WikiText bits per byte is 0.886.

Same-scale public references remain ahead on this aggregate. SmolLM2 135M scores 47.84%. LFM2.5 230M scores 48.73%. SmolLM2 360M scores 54.02%. LiLM1 used 50 times fewer pretraining tokens than SmolLM2 135M.

The base checkpoint is a next-token model. It is not a tool-use or instruction-following model.

Load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "glouriousgautam/LiLM1-230M-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)

The architecture is custom. Loading requires trust_remote_code=True.

Intended use

Use LiLM1-230M-base for research on compact language models. It can serve as the parent for further training. Validate any downstream use before deployment.

Do not use this checkpoint as a chat or tool-calling model. Do not use it with context longer than 4,096 tokens. Do not use it for medical, legal, financial, or security decisions.

Limitations

• English is the only evaluated language.
• The model has limited math, retrieval, and instruction-following ability.
• The evaluation does not cover every downstream task.
• No quantized release has passed final validation.

License

Apache License 2.0.

Citation

@misc{lilm1_base_2026,
  title        = {LiLM1-230M-base},
  author       = {Akshay Gautam},
  year         = {2026},
  howpublished = {https://huggingface.co/glouriousgautam/LiLM1-230M-base}
}