PiNanoLM-50M

A lightweight ~50M-parameter decoder-only Transformer for Raspberry Pi, ARM, and CPU-only edge inference.

PiNanoLM-50M is Version 2 of the PiNanoLM family — a direct evolution of PiNanoLM-20M. It keeps the same clean, from-scratch PyTorch architecture but scales capacity (hidden 256→384, GELU→SwiGLU, context 512→1024) for meaningfully better generation quality while remaining efficient on edge devices.

The whole family now shares a single, config-driven engine (pinanolm_core), so 20M, 50M, and future variants (100M, 250M, Instruct, Code, Math, Security) reuse the same model, training, inference, export, and benchmark code — no duplication.

---

Highlights

• 49,972,608 parameters (~50M), decoder-only GPT-style Transformer
• SwiGLU feed-forward (V2) with a GELU fallback for V1 compatibility
• RoPE (Rotary Position Embeddings) + RMSNorm + weight tying
• Flash attention via scaled_dot_product_attention (auto), manual fallback
• 1024-token context (2× PiNanoLM-20M)
• Shared BPE tokenizer (vocab 32,768) — identical to PiNanoLM-20M
• CPU-only inference, dynamic INT8, TorchScript, ONNX, and GGUF export
• Fully resumable training: AMP (BF16/FP16), grad accumulation, grad checkpointing, cosine LR, AdamW, grad clipping, TensorBoard + CSV + optional W&B
• Modular, typed, tested codebase (20 unit tests, PEP8, structured logging)

---

Architecture

Component	PiNanoLM-20M (V1)	PiNanoLM-50M (V2)
Parameters	19,798,272	49,972,608
Hidden size	256	384
Layers	12	12
Attention heads	8 (head_dim 32)	12 (head_dim 32)
FFN intermediate	1344	2192
FFN activation	GELU	SwiGLU
Context length	512	1024
Positional encoding	RoPE (θ=10000)	RoPE (θ=10000)
Normalization	RMSNorm (eps 1e-5)	RMSNorm (eps 1e-5)
Attention	manual	flash (SDPA) + fallback
Weight tying	Yes	Yes
Vocabulary	32,768 (BPE)	32,768 (BPE, same tokenizer)

The family scales by config alone (see pinanolm_core/variants.py):

Variant	Params (est.)
pinanolm-20m	19,798,272
pinanolm-50m	49,972,608
pinanolm-100m	102,777,344
pinanolm-250m	219,839,232

---

Repository layout

pinanolm-50m/
  pinanolm_core/      # SHARED engine: config, model, generation, data, utils, variants
  pinanolm_50m/       # 50M preset (Pinanolm50mConfig / Pinanolm50mForCausalLM)
  pinanolm_20m/       # 20M preset — V1 backwards-compatibility over the shared engine
  training/           # train.py (AMP, grad accum, grad ckpt, resume, TB+CSV+W&B)
  inference/          # generate.py (temp, top-k, top-p, typical, rep-penalty, streaming)
  export/             # export_all.py (TorchScript, ONNX, INT8, GGUF)
  benchmark/          # benchmark.py (tok/s, latency, RAM, CPU, load time; 20M vs 50M)
  scripts/            # train_tokenizer.py, preprocess.py, upload_hf.py
  tests/              # unit tests
  examples/           # run_pipeline.py
  config.json  generation_config.json
  tokenizer.json  tokenizer_config.json  special_tokens_map.json
  requirements.txt  LICENSE  README.md

---

Installation

git clone https://huggingface.co/ismailtasdelen/pinanolm-50m
cd pinanolm-50m
pip install -r requirements.txt

Requires Python 3.12+ (developed/verified on 3.11/3.12), PyTorch ≥ 2.1. CPU-only; CUDA optional (AMP auto-enables BF16/FP16).

---

Quick start (Python)

import json, torch
from pinanolm_50m import Pinanolm50mConfig, Pinanolm50mForCausalLM
from tokenizers import Tokenizer
from safetensors.torch import load_model

cfg = Pinanolm50mConfig.from_dict(json.load(open("config.json")))
model = Pinanolm50mForCausalLM(cfg)
load_model(model, "checkpoints/model.safetensors")
tok = Tokenizer.from_file("tokenizer.json")

ids = torch.tensor([tok.encode("Once upon a time").ids])
out = model.generate(ids, max_new_tokens=64, temperature=0.9,
                     top_k=40, top_p=0.9, typical_p=0.95, repetition_penalty=1.1)
print(tok.decode(out[0].tolist()))

---

Training

Fully resumable; resumes automatically from checkpoints/checkpoint_latest.pt.

# 1. (tokenizer is reused from PiNanoLM-20M; retrain only if explicitly needed)
# 2. preprocess corpus -> token shards (shared pipeline)
python scripts/preprocess.py --corpus-dir data/corpus \
    --tokenizer tokenizer.json --seq-len 1024 --out data/tokenized
# 3. train
python training/train.py --config config.json --data data/tokenized \
    --out checkpoints --epochs 2 --batch-size 8 --grad-accum 8 \
    --precision bf16 --grad-checkpointing

Features: mixed precision (BF16/FP16 on CUDA), gradient accumulation, gradient checkpointing, cosine LR + warmup, AdamW + weight decay, gradient clipping, validation loop, TensorBoard + CSV logging, optional Weights & Biases (--wandb), automatic checkpoint resume.

Dataset

The default pipeline uses permissively licensed public-domain books (Project Gutenberg) for reproducible offline training. The shared pinanolm_core.data module also supports Hugging Face streaming sources — FineWeb-Edu, TinyStories, Wikipedia, public-domain books — via scripts/preprocess.py --hf <source> (requires HF Hub access in that environment). For production pretraining, swap in FineWeb-Edu.

---

Inference

python inference/generate.py --prompt "Explain HTTP." --max-new-tokens 128
python inference/generate.py --prompt "Once upon a time" --stream \
    --temperature 0.8 --top-k 40 --top-p 0.9 --typical-p 0.95 --repetition-penalty 1.1

Sampling controls: temperature, top_k, top_p, typical_p (locally-typical sampling), repetition_penalty, max_new_tokens, plus token streaming (--stream).

---

Raspberry Pi / edge optimization

python export/export_all.py --config config.json \
    --safetensors checkpoints/model.safetensors --out export \
    --formats torchscript onnx int8 gguf --llama-cpp ~/llama.cpp

Produces:

• TorchScript — export/model_torchscript.pt
• ONNX (opset 17, dynamic axes) — export/model.onnx
• Dynamic INT8 (QNNPACK on ARM) — export/model_int8.pt
• GGUF — a llama-format HF bridge (export/hf_bridge/) + .gguf files when a llama.cpp clone is provided (PiNanoLM's RoPE+RMSNorm+SwiGLU+tied layout maps onto the llama architecture for convert_hf_to_gguf.py).

Target devices: Raspberry Pi 4, Raspberry Pi 5, Orange Pi, Rock Pi, ARM Linux.

INT8 dynamic-quant latency is only meaningful on the target ARM/RPi device (QNNPACK). On x86 the quantized .pt still exports (smaller), but latency numbers should be measured on-device.

---

Benchmark

# single model
python benchmark/benchmark.py --config config.json \
    --safetensors checkpoints/model.safetensors
# compare 20M vs 50M
python benchmark/benchmark.py --compare --variant pinanolm-20m --variant pinanolm-50m

Measures model size, load time, peak RAM, CPU utilization, latency (ms/token) and throughput (tokens/sec), and prints a Markdown comparison table (also saved to benchmark_result.json). Run on x86, RPi 4, and RPi 5 to compare across devices.

---

Tests

python -m unittest discover -s tests

20 tests: exact param count, forward shape, loss computation & decrease, weight tying, SwiGLU activation, flash-vs-manual attention agreement, gradient checkpointing, typical sampling, tokenizer roundtrip, scheduler warmup/decay, and variant-registry scaling.

---

Backwards compatibility

pinanolm_20m re-exposes the original Pinanolm20mConfig / Pinanolm20mForCausalLM API on top of the shared engine (GELU FFN, ctx 512), so existing V1 code and the shared tokenizer keep working unchanged.

---

Future variants

The codebase is designed so future models require only a preset entry in pinanolm_core/variants.py: PiNanoLM-100M, PiNanoLM-250M, PiNanoLM-Instruct, PiNanoLM-Code, PiNanoLM-Math, PiNanoLM-Security. The shared engine handles architecture, training, inference, export, and benchmarking for all of them.

---

License

MIT — see LICENSE.

Citation

@misc{pinanolm50m2026,
  title  = {PiNanoLM-50M: A Lightweight Transformer Language Model for Edge Devices},
  author = {Ismail Tasdelen and contributors},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/ismailtasdelen/pinanolm-50m}}
}

---

PiNanoLM-50M is for education and research. It is a tiny model and does not provide factual guarantees.