Micro-Llama for Vicharak Shrike-Lite

Ultra-small Llama-style language model trained on TinyStories for bare-metal inference on the Vicharak Shrike-Lite (RP2040 + Renesas FPGA).

Weights are sized to live in QSPI Flash via XIP; activations + KV cache fit in the RP2040's 264KB SRAM.

Model details

Field	Value
Parameters	213,312
`d_model`	64
MLP hidden	128
Layers	2
Heads	2
Context	128
Vocab (BPE)	1024
Architecture	RoPE · RMSNorm · SwiGLU
Framework	JAX / Flax / Optax
Train device	gpu
Train steps	5,000
Dataset	`roneneldan/TinyStories`

W&B run: https://wandb.ai/dotslasha/vicharak-micro-llama/runs/t2mx7owg

Files

File	Description
`micro_llama.bin`	llama2.c-style float32 weight blob (Phase-3 firmware)
`micro_llama.json`	Export metadata / layout notes
`latest_params.npz`	Flax parameter tree (NumPy object array)
`latest_meta.json`	Train + model config used for this checkpoint
`tinystories_bpe_1024.model`	SentencePiece BPE model
`tinystories_bpe_1024.vocab`	SentencePiece vocabulary

Training summary

Optimizer: AdamW (β₁=0.9, β₂=0.95), cosine decay after warmup
Batch size: 32
Peak LR: 0.0003
Weight decay: 0.1
Grad clip: 1.0
Final eval loss: 3.2735
Final eval perplexity: 26.4023
Card generated: 2026-08-03 13:44 UTC

Intended use

Research / education for edge LLM inference on MCU+FPGA boards. Not intended for general chat or production NLP.

How to use (Python)

# from this repo
uv run python phase-2/generate.py --prompt "Once upon a time"

Or load the SentencePiece tokenizer + Flax params from this repo and run MicroLlama in phase-2/model.py.

How to use (firmware)

Flash micro_llama.bin into the Phase-3 C inference engine (Pico SDK / UF2).
Stream tokens over USB serial from the Shrike-Lite.

Limitations

Tiny capacity: expects simple children’s-story English (TinyStories distribution).
Context limited to 128 tokens.
FP32 export by default; quantize (INT8) before aggressive on-device acceleration.

Citation

@misc{micro-llama-shrike,
  title  = {Micro-Llama for Vicharak Shrike-Lite},
  author = {weights-and-wires},
  year   = {2026},
  url    = {https://huggingface.co/weights-and-wires/vicharak-micro-llama}
}

Acknowledgements

roneneldan/TinyStories
llama2.c export conventions
Vicharak Shrike-Lite hardware platform

Downloads last month: -

weights-and-wires
/

vicharak-micro-llama