ice40GPT: a 1069K-parameter chat model that runs on an iCE40UP5K

Decoder-only transformer trained on TinyChat. Weights, KV cache and tables total 134KB, which fits in the iCE40UP5K's on-chip memory (128KB SPRAM + 15KB EBR).

parameters	1,069,280 stored; one block looped 6 times, own norms per pass
width / heads / KV heads / FFN	96 / 4 / 1 / 256
vocab / context	2048 BPE tokens / 32 tokens
embedding	64-wide token table projected up to 96, tied with the output head
weights	int4 blocks, int4 embeddings, per-row scales, trained with QAT
validation	2.631 nats/word, ppl 10.3, top-1 47.6%

Prompt format: [INST] user text [/INST] reply [INST] .... Text is lowercased and split into words and punctuation marks, then BPE-encoded; the last piece of each word ends in </w>. Generation stops at <eos>, [INST] or [/INST].

The stored weights are float master copies; the model applies the int4/int4 rounding itself in its forward pass (fake_quant in modeling_ice40gpt.py), so its logits already reflect the quantized weights. generate slides a window of 32 tokens, as the chip does, and runs without a KV cache.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained(".", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)

prompt = tok("[INST] hello, how are you today? [/INST]", return_tensors="pt").input_ids
out = model.generate(prompt, max_new_tokens=40)  # greedy; stops at <eos>, [INST] or [/INST]
print(tok.decode(out[0, prompt.shape[1] :], skip_special_tokens=True))

Training data is TinyChat (CC BY-NC 4.0, GPT-4o mini generated), so the weights are non-commercial. Replies stay on everyday and emotional small talk; factual questions are not answered.