vladimir94/dory-1.0-4b-ane-coreml

Dory 1.0 4B — Core ML for the Apple Neural Engine (24,576-token context)

This is a conversion + 4-bit palettization (quantization, not a fine-tune and not a merge) of nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16, split into 102 Core ML programs so that a looped 4B model with gated attention can run on the Neural Engine of a Mac at a 24,576-token context.

Weights are unchanged conceptually: the same model, converted to Core ML and palettized to 4 bits (with --lut-blocks none you get a 15 GB full-precision variant instead). Gated attention, the |α| nGPT residual and NEOX RoPE are all represented; nothing is distilled, pruned or retrained.

Measured on an M-series laptop

	4-bit (this repo)	fp16
decode, loops=1	11.3 tok/s	5.9 tok/s
prefill (batch)	130 tok/s	130 tok/s
size	5.6 GB	15 GB
KLD vs the fp16 reference model	0.077 – 0.632	0.019 – 0.056
top-1 agreement	62.5 – 100 %	87.5 – 100 %

The 4-bit set is fluent and correct on ordinary chat (it answers questions, writes stories, keeps a conversation), but in one measured window it disagrees with the reference on 3 of 8 next tokens. If fidelity matters more than speed or disk, use the full-precision conversion.

What is in here

• meta.yaml — the 102-program plan: dispatch order, per-program files, state declarations, and the gate chain that threads through the stack
• *.mlpackage — the programs themselves (infer and prefill variants)
• rope_tables.npz — RoPE rows as inputs
• tokenizer files and chat_template.jinja — the model's own template, including its thinking mode

Running it

git clone https://github.com/vladimir-voinea/dory-ane && cd dory-ane
python3.12 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
huggingface-cli download vladimir94/dory-1.0-4b-ane-coreml --local-dir dory-1.0-4b-ane-coreml
python scripts/dory_serve.py --model-dir ./dory-1.0-4b-ane-coreml   # OpenAI API + chat UI

The runtime reads meta.yaml and needs nothing else: it loads one phase of packages at a time, keeps one MLState per stated program, and rejoins the split LM head.

Limits worth knowing before you download

• Prefill and decode cannot be resident together (204 packages fails on this hardware), so a prompt is fed token by token at ~10 tok/s unless you accept a 100–270 s switch to the batch-prefill packages.
• An int8 KV cache is impossible: Core ML states are fp16 only, and integer inputs are rejected, so 999 MiB of fp16 state is the floor for 24K.
• One conversation cache is 999 MiB, so a server holds a couple of sessions, not hundreds.

Built on Anemll. The code is MIT; the weights follow the source model's NVIDIA Open Model License.