vynly/mica-flame-w-0.3.1

MICA Flame-W 0.3.1

A newer release is available: Flame-W0.3.5 includes a verified CPU notebook and improved held-out text prediction. This historical checkpoint remains available for reproducibility. Sentence quality and broad reasoning remain experimental.

MICA family: View the collection. Ember v0.3a provides byte-level word suggestions and completion. This release uses the bundled MICA integer engine; standard Transformers loaders do not load its .mica checkpoint.

A one-of-a-kind language model. Not a transformer, not a neural network.

MICA writes language with learned integer rewrite rules on a ring of cells, a cellular automaton. It is an original architecture, designed and trained from scratch. Flame-W predicts whole words over a 16,384-word vocabulary.

Why MICA is different

• No transformer inside. There is no attention and no stack of neural layers. Words are written onto a ring of integer cells, learned local rules rewrite the ring, and probes read the next word off it.
• Integers only. The rules, the probes, the next-word scores and the memory are all integer arithmetic.
• Deterministic. The same input always gives the same output.
• Small and CPU-only. About 35 MB in total. It runs on any CPU with numpy alone.

File	What it is
model.mica	The automaton: int8 rules, codes and readout. 34,111,616 bytes, SHA-256 1f4d5503…9e1a1f6d
memory.npz	Memory readout: integer recall plus a 16-channel association over the last 64 words. 673,650 bytes, SHA-256 5fda5a6c…8b928be7
answer_mode.npz	Answer mode, for multiple-choice questions: +3.25 nats for content words already in the context
vocab.json, package.json	The vocabulary and the engine geometry
mica_r1/, *.py	The exact integer engine, tokenizer, decoders and scripts. numpy only

About 35 million stored integers in total (int8, plus small int16/int64 tables). It runs on a CPU with numpy, and the same input always gives the same output.

Tiny Theory of Mind

AxiomicLabs/Tiny_Theory_of_Mind: 2,000 items, 4 endings each, chance 25%.

Protocol: the context is fed in full. Each ending is scored teacher-forced from a copy of that state, and the choice is the ending with the highest length-normalized log-likelihood.

Configuration	mean ln p per ending token (words)	per character (acc_norm)
automaton alone	28.15%	26.15%
+ memory (a plain language model)	31.15%	30.15%
+ memory + answer mode (0.3.1)	31.25%	31.45%

On the held-out half (the odd rows of each topic, 1,000 items, not used to tune anything since the split was fixed), 0.3.1 scores 31.80% per word token and 32.50% per character.

Reproduce:

pip install numpy huggingface_hub
python run_tom.py                      # 0.3.1; prints acc_norm and acc_word
python run_tom.py --config memory      # without answer mode
python run_tom.py --config automaton   # the automaton alone

Per-item predictions are in results/tom_0.3.1.json.

Notes for a fair comparison:

• Answer mode is a calibration for answering questions about a text. It adds a fixed bonus to content words already in the context (vocabulary words of frequency rank 150 or more). Its strength was chosen on synthetic practice items written for this purpose, never on this benchmark. Without it, the plain language model scores 31.15% / 30.15%.
• Selection: the memory was fitted and selected on held-out text, and the answer-mode strength on synthetic items. During the project the full benchmark was still scored several times while comparing approaches (for example, whether to use answer mode at all), so these are not single blind looks. Later tweaks were judged on the even rows only.
• Tokens: the leaderboard models normalize by their own subword tokens. Flame-W's tokens are words, so the per-character number is given too.
• No ToM training: neither the automaton nor the memory saw any ToM items.

Changes from v0.3

• Memory refit: the same readout, trained on 7 times more text. Bits per word on held-out validation text: 5.268 → 5.208.
• Answer mode was added for multiple-choice questions.
• Tiny Theory of Mind did not change significantly. v0.3 with answer mode scored 32.20% per word token and 31.40% per character (paired p = 0.17). On the held-out half, the two versions are within 0.2 points.

Generate text

python generate.py "I was thinking about"
python generate.py --mode suggest "Can you send me"

These use the same decoders as the playground at https://mica-ai-ten.vercel.app. Answer mode is not used for free text.

How it works

• Automaton. Each word is written onto a ring of 768 cells × 112 integer channels. 16 phases of learned local rules run, and 240 probes read the cells. The next-word scores are bias + co · f, all integers. The rules see about the last 4 words.

• Memory readout. Every word in the last 64 tokens adds to the scores:

  • recall of the same word, by word class and distance;
  • a 16-channel association, so related words also become more likely.

  int8 tables with int64 sums. 0.3.1 refits it, warm-started from v0.3, on about 4.5 million word positions (7 times more than v0.3).

• Answer mode (optional) adds +3.25 nats to content words already seen in the window.

Training data

English text collected by the project: chat dialogue, everyday sentences, image captions and short stories. Sources and builders are in the GitHub repository.

Limits

This is a research model, not an assistant. The memory knows which words came up in the last 64, but it does not track who knows what. On Tiny Theory of Mind it is weakest on bluffing, communication failure, diverse beliefs and white lies (16–18%, below chance).

License

MICA AI Non-Commercial License: personal and research use only. Commercial use is forbidden. Code, docs and history: https://github.com/Vovala14/Mica-Ai.