pinanolm-20m

An extremely lightweight decoder-only Transformer language model for Raspberry Pi and ARM edge devices.

pinanolm-20m is an educational and practical tiny LLM built from scratch in PyTorch. It targets ~20 million parameters, runs on CPU-only hardware, and is designed to be easy to reproduce, extend, and deploy on edge devices (Raspberry Pi 4/5, ARM SBCs).

---

Highlights

• ~19.8M parameters (decoder-only GPT-style Transformer)
• Pure PyTorch, no heavy dependencies
• RoPE (Rotary Position Embeddings) + RMSNorm + GELU + weight tying
• Trained BPE tokenizer (vocab 32,768) on public-domain text
• CPU-only inference, dynamic INT8 quantization for ARM
• TorchScript + ONNX export paths
• Reproducible training with automatic resume
• Clean, modular, tested codebase (unit tests + type hints + logging)

---

Architecture

Component	Value
Model type	Decoder-only Transformer
Hidden size	256
Layers	12
Attention heads	8 (head_dim 32)
FFN intermediate	1344
Context length	512 tokens
Activation	GELU
Positional encoding	RoPE (theta 10000)
Normalization	RMSNorm (eps 1e-5)
Weight tying	Yes (embed == lm_head)
Vocabulary	32,768 (BPE)
Parameters	19,798,272 (~20M)

The codebase is variant-ready: pinanolm-50m, pinanolm-100m, pinanolm-instruct, and pinanolm-code only require changing config.json (hidden_size / num_layers / vocab_size).

---

Repository Layout

pinanolm-20m/
  pinanolm_20m/       # model definition (model.py)
  training/           # train.py (AMP, grad accum, resume, TB)
  inference/          # generate.py (temp, top-k, top-p, rep penalty)
  export/             # export_all.py (TorchScript, ONNX, INT8)
  benchmark/          # benchmark.py (size/RAM/latency/tok-s)
  scripts/            # train_tokenizer.py, preprocess.py, upload_hf.py
  tests/              # unit tests
  examples/           # run_pipeline.py
  config.json         # model config
  generation_config.json
  tokenizer.json, tokenizer_config.json, special_tokens_map.json
  requirements.txt
  LICENSE
  README.md

---

Installation

git clone https://huggingface.co/ismailtasdelen/pinanolm-20m
cd pinanolm-20m
pip install -r requirements.txt

Requires Python 3.12+ (developed on 3.11/3.12). CPU-only; CUDA optional (AMP auto-enables).

---

Training

1. Prepare a corpus (public-domain texts). Example: data/corpus/*.txt.
2. Train the tokenizer:

   python scripts/train_tokenizer.py --vocab-size 32768 --corpus-dir data/corpus
   

3. Preprocess into token shards:

   python scripts/preprocess.py --corpus-dir data/corpus --tokenizer tokenizer.json --seq-len 512 --out data/tokenized
   

4. Train (restartable; resumes from checkpoints/checkpoint_latest.pt):

   python training/train.py --config config.json --data data/tokenized \
       --out checkpoints --epochs 2 --batch-size 16 --grad-accum 4
   

   Features: mixed precision (CUDA), gradient accumulation, cosine LR schedule with warmup, AdamW, gradient clipping, validation loss logging, TensorBoard.

Note on dataset: The default pipeline uses permissively licensed public-domain books (Project Gutenberg) for demonstration. For production pretraining, swap in FineWeb-Edu, TinyStories, or Wikipedia via scripts/preprocess.py (HF Hub access required in that environment).

---

Inference

python inference/generate.py --prompt "Explain HTTP." --max-new-tokens 128
python inference/generate.py --prompt "Once upon a time" \
    --temperature 0.8 --top-k 40 --top-p 0.9 --repetition-penalty 1.1

Parameters: temperature, top_k, top_p, max_new_tokens, repetition_penalty.

---

Raspberry Pi Optimization

• CPU inference (no GPU needed)
• Dynamic INT8 quantization via torch.quantization.quantize_dynamic (QNNPACK backend on ARM)
• TorchScript for optimized, portable execution
• ONNX for ONNX Runtime deployment

python export/export_all.py --config config.json --safetensors checkpoints/model.safetensors --out export

Outputs: export/model_torchscript.pt, export/model.onnx, export/model_int8.pt.

---

Benchmark

python benchmark/benchmark.py --config config.json \
    --safetensors checkpoints/model.safetensors --int8 export/model_int8.pt

Measures model size, peak RAM, latency (ms/token), and throughput (tokens/sec). Generates a Markdown table automatically. Run on x86 CPU, Raspberry Pi 4, and Raspberry Pi 5 to compare.

---

Example Usage (Python)

from pinanolm_20m import Pinanolm20mConfig, Pinanolm20mForCausalLM
from tokenizers import Tokenizer
import torch

cfg = Pinanolm20mConfig.from_dict(json.load(open("config.json")))
model = Pinanolm20mForCausalLM(cfg)
# load safetensors weights ...
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Hello world").ids])
out = model.generate(ids, max_new_tokens=64, temperature=0.9)
print(tok.decode(out[0].tolist()))

---

Tests

python -m unittest discover -s tests

---

License

MIT — see LICENSE.

Citation

@misc{pinanolm2024,
  title={pinanolm-20m: A Tiny Transformer Language Model for Edge Devices},
  author={Ismail Tasdelen and contributors},
  year={2024},
  howpublished={\url{https://huggingface.co/ismailtasdelen/pinanolm-20m}}
}

---

pinanolm-20m is for education and research. It is not a substitute for large language models and does not provide factual guarantees.