point-1

A language model with one parameter. The parameter is called w.

██████╗  ██████╗ ██╗███╗   ██╗████████╗    ██████╗
██╔══██╗██╔═══██╗██║████╗  ██║╚══██╔══╝   ██╔═══██╗
██████╔╝██║   ██║██║██╔██╗ ██║   ██║      ██║   ██║
██╔═══╝ ██║   ██║██║██║╚██╗██║   ██║      ██║   ██║
██║     ╚██████╔╝██║██║ ╚████║   ██║      ╚██████╔╝
╚═╝      ╚═════╝ ╚═╝╚═╝  ╚═══╝   ╚═╝       ╚═════╝

logits[b] = w * h[b]

h[b] = (b - 79) / 47, clipped to [-1, 1], over printable ascii + newline (96 tokens)

No tokenizer, bytes are tokens. No attention, there's nothing for it to attend between. The forward pass is one multiplication.

Training

TinyShakespeare, 1,115,394 bytes. Only the byte histogram gets kept, the corpus is streamed and never held in memory. The loss is convex in exactly one variable, so gradient descent would be the wrong tool: Newton's method converges to the actual optimum in four steps.

newton 0  w = +0.000000  loss = 4.5643
newton 1  w = +0.595712  loss = 4.5020
newton 2  w = +0.610758  loss = 4.5020
newton 3  w = +0.610785  loss = 4.5020   <- global optimum, done

	nats/byte
uniform baseline	4.5643
unigram floor	3.3128
point-1	4.5020

So one parameter captures about 5% of what even a unigram model knows here. A unigram model needs 96. Nobody has released one yet, apparently.

What the parameter does

Tilts probability toward high bytes. That's the entire optimal policy, there is nothing else to learn. Consequence: 'z' and '~' sit 0.085 apart in feature space, so to point-1 they're roughly the same letter. Greedy decoding is an infinite stream of tildes. Temperature 0.8 looks like a modem.

$ python3 generate.py "to be or not to be" --temp 0.01 --n 60
~|~}~~~}~|}~|}~~}~~{~~~}~~}}~~~~~~~~~~~~~|}~}}}~~~~}}~~~}~}~~

The prompt is parsed, then ignored. transformers also works if you need torch for some reason:

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("tsfrm/point-1", trust_remote_code=True)
m.generate_bytes(64)

Files

model.safetensors is 97 bytes, 4 of which are the float32. The header takes the rest.

16,501,264,351,232x smaller than vacuum-16t.