esp32-ai-barista

An 8.9M-parameter question-answering model that runs entirely offline on an ESP32-S3 microcontroller, answering espresso questions a piece at a time at roughly 50 ms per position.

This is the Barista model from the esp32-ai project, which pairs it with a portable C inference runtime that runs on the device.

What it does

One question in, one answer out. No chat history, no retrieval. The model writes the answer itself; it does not select from a catalogue of canned replies.

Real output from the board:

Q: my espresso tastes really bitter, what should i change?
A: if the cup reads bitter, coarsen a step. if it reads sour, go finer instead.
   which of those does your drink read as?

Q: how do i clean my machine
A: if the machine has never had maintenance, the manual is still where you
   start. use the recommended cleaner and descaler, and improvise nothing.
   whats your machine model?

Q: what grind size should i use for a light roast
A: when you need a procedure for a double drink or another coffee recipe, i
   cannot provide it. i can help improve the espresso shot. what problem does
   the shot have?

The third answer is the model declining a request outside what it was built to handle, which is a trained behaviour rather than a filter.

The asymmetric vocabulary

This is the design point that separates Barista from a conventional small model. It reads 8,057 input tokens so it can accept varied typing, and writes only its 854-class output alphabet.

Two consequences follow:

• The output head is 854 rows instead of 8,057, so it is 109K parameters rather than a million. A full-vocabulary head would be the largest single matvec the model runs per position; this makes it one of the smallest.
• Words outside the 854 are not merely unlikely, they are unsayable. There are no digit tokens at all, so the model cannot invent grind 2 steps finer. This restriction is a property of the vocabulary, not of a post-filter.

Because the two alphabets differ, a sampled index is an output class, not a token id. Feeding it back into the model requires a mapping, which is what layout.json holds:

854 output classes
 -> 135 reuse an existing BPE id
 -> 719 map to rows appended above the BPE vocabulary
input vocabulary total: 7,338 BPE + 719 appended = 8,057

Without vocab.json and layout.json the weights are not usable: you could score logits but not turn them into words, nor continue past the first one.

Why it fits on a microcontroller

8.9M parameters, of which only 1.58M are dense transformer core. The rest is a Per-Layer Embedding table read one row per token straight from memory-mapped flash.

component	params	share	lives in
PLE table	6,187,776	69%	flash, memory-mapped
token embedding	1,031,296	12%	flash, memory-mapped
dense transformer core	1,575,424	18%	flash, staged to PSRAM
output head, untied	109,312	1%	flash, staged to PSRAM
total	8,903,808		4.6 MB at int4

The constraint is fast memory, not total memory. The board has 8 MB of PSRAM but only 512 KB of internal SRAM. PLE keeps the large tables in flash, where a one-row-per-token read costs bandwidth rather than the scarce pool.

Architecture

architecture   PLE
format         version 1, TIED_HEAD clear
input vocab    8,057    embedding and PLE table rows
output vocab   854      logits the model produces
d_model        128
layers         6
heads          4
ffn_hidden     384
ple_dim        128
seq_len        128
rope_theta     10000.0
weights        int4, group size 128

TIED_HEAD is clear, so the output head is its own tensor stored after out_norm, not a view of the token embedding. The header states both vocabulary sizes, so the runtime is never told separately how many logits to score.

Runtime placement

tier	holds
flash, memory-mapped	PLE table and token embedding
PSRAM	per-position core and the untied head, staged to int8 at boot; KV cache
SRAM	float scratch buffers and RMSNorm vectors, 20,940 B plus 20 norm vectors

Activations are quantized to int8 for each staged matvec, and per-layer matvecs are split across both LX7 cores. At boot the firmware reports 44 staged tensors and zero SRAM fallbacks.

Measured speed

The model emits one output piece at a time, and punctuation is a piece, so throughput is quoted in both units. Over the eight-prompt benchmark set below, 253 pieces render as 213 readable words.

The firmware can stream answers to an OLED as well as to serial, and the panel is redrawn once per piece, so both modes are measured.

	pieces/s	readable words/s	ms per piece
serial only	16.60	13.97	60.25
with the OLED	11.26	9.48	88.83

The panel costs 28.58 ms per output piece. The display runs outside the model forward path: a profiled serial-only run measured 49.6 ms per forward, and enabling the OLED adds a framebuffer transfer per emitted piece. The generated text is identical in both modes.

Measured on the board on 2026-08-02 with scripts/benchmark_device.py: eight fixed prompts, two passes per mode, spread under 0.1%. Both modes ran the same weights (fp=e602146b) and the same build switches apart from the display, and the benchmark refuses to report a comparison otherwise.

A forward is one position: the prompt tokens plus one per emitted piece, so wall-clock throughput falls as the question gets longer.

Files

file	what it is
model.bin	int4 weights and header, flashed to the model partition
tokenizer.json	canonical 7,338-entry ByteLevel BPE
vocab.json	the 854 output classes, in class-id order
layout.json	class id to input token id map, and the vocabulary layout
metadata.json	architecture, parameters, runtime placement, SHA-256 and size of the four assets above
LICENSE	MIT

Verify a download before trusting it:

shasum -a 256 model.bin
# 1359a1cb74de4143d630c2c192990de814cd47255bcdfa9cc135f07ef0a39fc4

The firmware also prints an FNV-1a fingerprint of the mapped image at boot, e602146b, which must match device_fingerprint_fnv1a in metadata.json.

The firmware's three generated C headers, the word tables and the packed BPE encoder asset, are built from vocab.json, layout.json and tokenizer.json by the source repository, so they are not distributed here.

Verification

Reference logits are not shipped in this bundle. Verification lives with the runtime, in the source repository, and covers four distinct things:

• runtime/host_verify/verify.c checks the exact int4, float-activation path against PyTorch, to 1e-5.
• runtime/host_verify/staging_verify.c checks int8 weight staging, scale alignment, ranged matvec equivalence, platform hook dispatch, header validation and the untied-head format branch.
• runtime/host_verify/bpe_tokenizer_verify.c checks the BTK1 loader and the ASCII encoder against a hand-built asset.
• firmware/esp32_barista/tools/verify_tokenizer.py checks that the device encoder produces the same ids as the reference tokenizer, and that every one of the 135 reused mappings resolves to the id it claims.

The device path enables int8 activations and is therefore not bit-identical to the host golden. It is validated separately: the same model.bin driven through the same tables on host reproduces the device's answers word for word.

Usage

These weights are not usable on their own. The firmware also needs C headers generated from vocab.json, layout.json and tokenizer.json, and it has to be compiled and flashed alongside the model. The esp32-ai repository does both steps:

scripts/fetch_model.sh barista   # downloads and verifies these files
scripts/deploy.sh barista        # generates headers, runs host gates, compiles, flashes

fetch_model.sh checks the assets above against a SHA-256 and byte size pinned in the script, and cross-checks metadata.json against those same pins. It installs nothing unless every check passes. deploy.sh never reaches the network. Use deploy.sh rather than writing model.bin by hand: it regenerates the tables the firmware compiles against, and writes both the model and the firmware.

Questions arrive over USB serial, ASCII only, and answers stream back one output class at a time.

Training data

A private question-and-answer corpus written for this project. It is not redistributed here, and neither are the training checkpoints or the evaluation sets.

What is distributed is the deployable binary and the inference assets needed to run it. That boundary is deliberate: the weights are usable and verifiable without the corpus, and the corpus is not part of the release.

The training checkpoint is not distributed. Its SHA-256 is recorded in metadata.json so a given model.bin can be tied to the weights it came from.

Limitations

• Espresso troubleshooting only. Outside that, it declines or answers poorly. 1.58M dense parameters do not store facts.
• 854-word output vocabulary. It cannot say anything outside that set, by construction. It cannot emit digit characters, or number words beyond the six it includes: double, few, half, three, twice, two.
• ASCII input only. Non-ASCII questions are refused rather than encoded differently from the reference tokenizer.
• 128-token context, greedy decoding. No sampling, no history between questions. The same question always produces the same answer.
• It can still produce a degenerate tail on some inputs.
• No published accuracy numbers. The evaluation sets influenced model selection and are not held out, so quoting a score from them would overstate what is known.

License

what	license
model weights (model.bin)	MIT
tokenizer (tokenizer.json)	MIT
vocabulary and layout (vocab.json, layout.json)	MIT
training corpus, checkpoints, evaluations	not distributed

The weights were trained from scratch and contain no third-party weights. MIT covers what is in this repository; it grants nothing over the training corpus, which is not published here.

Credits

The PLE design is reproduced from Google's published Gemma 3n Per-Layer Embeddings work. No model, checkpoint or method here derives from it beyond the published description.