poxek/OrcaCyber-Nano-1.0-oMLX

OrcaCyber Nano 1.0 on oMLX (recipe)

How to serve orcarouter/OrcaCyber-Nano-1.0 in oMLX on Apple Silicon with full speed: GGUF weights as published, the Splash external engine, DFlash2 speculative decoding, Neural Engine prefill, an INT8 KV cache, and a persistent SSD prefix cache.

This repository contains no weights. The original repo is gated: request access there, then follow the steps below. Nothing here bypasses that gate.

TL;DR: what changed

Layer	Change
Weights on disk	None. The GGUF files are used byte-for-byte as published (SHA-256 below).
Format conversion	None. No GGUF → safetensors/MLX conversion, no requantization.
Draft model	Added incoai/Qwen3.8-27B-DFlash2 (BF16 safetensors) as the speculative draft.
oMLX config	GGUF routed to the Splash external engine; alias orcacyber-nano.
Runtime (in memory, every load)	Splash repacks weights into its kernel layouts, patches the chat template, splits the FFN onto the Neural Engine, and quantizes the draft. Details below.

Why it's done this way

1. oMLX's in-process engines only load MLX checkpoints. oMLX lists a GGUF as a model but serves it only through an external engine (docs/Splash.md).
2. No native MLX build is possible from published files. oMLX's quantizer (oQ) needs a BF16/FP16 source. The card names orcarouter/OrcaCyber-Nano-1.0-AI-Defense as the base model, but that repo is not public (Model not found). Rebuilding BF16 by inverting llama.cpp's conversion and then requantizing would stack a second quantization on top of IQ4_XS, so it was not done.
3. Splash accepts this GGUF unmodified. Splash identifies a model by its geometry, not its repo name. This file is a dense qwen35 model with 64 layers, hidden size 5120, 24 query / 4 KV heads, and 48 GDN + 16 full-attention layers, which is the Qwen3.8-27B family. Every tensor type in the file is on Splash's list of accepted GGUF types.

Source files (unchanged)

From orcarouter/OrcaCyber-Nano-1.0:

File	Size	SHA-256
OrcaCyber-Nano-1.0-AI-Defense-Mix-15.7G.gguf	15,695,148,832 B	4214beb4c3ee2323d97d19a5c6a3327f25d969c7db26956e2fc917cf297dc289
mmproj-OrcaCyber-Nano-1.0-AI-Defense-F16.gguf	927,606,848 B	f854eb8f1e4fdd133a7375b9aa78792548e561f6d8b95da7660e91db56f85382

Not used:

• Mix-13G.gguf: smaller variant, not needed with 48 GB.
• DFlash2-Q8_0.gguf: Splash needs a DFlash2 draft as config.json + BF16 safetensors, so it can't load this GGUF draft.

Draft: incoai/Qwen3.8-27B-DFlash2 @ 015e795645c74b1a0eeef3b570031fb62e769bc5. It is a 5-layer DFlash2 draft that reads target layers [5, 19, 33, 47, 61], with block size 8.

Compatibility audit of the GGUF

Header: general.architecture = qwen35, block_count = 65 (64 + 1 MTP/nextn layer), context_length = 262144, tokenizer.ggml.model = gpt2, tokenizer.ggml.pre = qwen35, file_type = 30 (IQ4_XS-based mix), imatrix-quantized.

Tensor types, grouped by role (count × type), checked against Splash's GGUF target rules:

Role	Types in file	Splash requirement
FFN gate/up/down	IQ4_XS, IQ3_S (18× ffn_up), Q4_K, Q5_K	linears: any listed quant ✔
Attention q/k/v/qkv/gate/output	IQ4_XS, Q4_K, Q5_K, Q6_K	✔
GDN ssm_out	Q5_K	✔
GDN ssm_alpha / ssm_beta	Q6_K / Q6_K (same type pair)	loaded as one segment of the GDN input projection ✔
GDN ssm_conv1d, ssm_a, ssm_dt.bias	F32	must be F32 ✔
Norms (incl. ssm_norm, q/k norms)	F32	F32/BF16/F16 ✔
token_embd / output	Q4_K / Q6_K	✔
nextn.* (MTP head, layer 64)	Q4_K, F32	not read: Splash subtracts declared MTP layers
mmproj	clip, F16, qwen3vl_merger	F16 projector accepted, converted to BF16 at load ✔

Transformations Splash applies at load time

None of these touch the files on disk. Splash repeats them on every start; this list comes from its docs and the startup log of this setup.

1. Family resolution: Installing … as Qwen3.8-27B (gguf); draft …/incoai/Qwen3.8-27B-DFlash2; vision enabled.
2. Tokenizer and config derived from the GGUF header (gpt2/qwen35 profile). oMLX also writes a derived config.json (model_type: qwen3_5, vision_config.projector_type: qwen3vl_merger) under ~/.omlx/gguf/.
3. Layer count: the declared nextn/MTP layer is subtracted, so the target runs 64 layers. The built-in MTP head is unused; DFlash2 does the speculation.
4. Weight repacking into Splash's MDGG0001 kernel images. Quantized blocks keep their GGUF bits per weight; norms load as F32; the Q6_K ssm_alpha/ssm_beta pair loads as one segment of the GDN input projection.
5. Vision tower: F16 mmproj tensors are converted to BF16, which is exact for a tower exported from BF16.
6. Draft quantization: DFlash2 projections are quantized in memory to affine 4-bit, group 64 (af4g64).
7. Chat template patch: patched to render later system messages in place (the original template rejects them). Splash keeps the patch only if ordinary conversations render byte-identically.
8. Neural Engine FFN split for prefill (--disable-ane turns it off).
9. KV cache stored as INT8, plus a persistent SSD prefix cache.

Setup

Tested with oMLX 0.7.2, Splash 1.4.0, macOS 27.0.1, and a MacBook Pro M5 Pro with a 20-core GPU and 48 GB. Splash needs an M3 or later and macOS 26.4+.

# 1. Splash engine
brew tap incoai/tap && brew install incoai/tap/splash

# 2. Weights (after access is granted on the original repo)
MODEL_DIR=~/.omlx/models/orcarouter/OrcaCyber-Nano-1.0   # any oMLX model dir, <owner>/<repo>/ layout
hf download orcarouter/OrcaCyber-Nano-1.0 \
  --include "OrcaCyber-Nano-1.0-AI-Defense-Mix-15.7G.gguf" \
  --include "mmproj-OrcaCyber-Nano-1.0-AI-Defense-F16.gguf" \
  --local-dir "$MODEL_DIR"
shasum -a 256 "$MODEL_DIR"/*.gguf   # compare with the table above

If you already have the GGUFs elsewhere on the same volume, hard-link them into the model dir (ln src dst). That uses no extra space.

3. Start oMLX and open Settings → Integrations → External engines. oMLX should detect splash.
4. Open the GGUF model's Model Settings → Advanced and press the button that downloads the DFlash2 draft (incoai/Qwen3.8-27B-DFlash2). Then turn on Use external engine → Splash.

The same steps work headless through the admin API. Use port 8888 for the app's default config, or 8000 for omlx serve:

OMLX=http://localhost:8888; AUTH="Authorization: Bearer <your oMLX API key>"
M=OrcaCyber-Nano-1.0-AI-Defense-Mix-15.7G
curl -X POST -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"repo_id":"incoai/Qwen3.8-27B-DFlash2"}' $OMLX/admin/api/hf/download
# after the download completes, check: external_engines[0].compatible == true
curl -s -H "$AUTH" $OMLX/admin/api/models | jq '.. | objects | select(.id?=="'$M'") | .external_engines'
curl -X PUT -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"external_engine":"splash","model_alias":"orcacyber-nano"}' $OMLX/admin/api/models/$M/settings
curl -X POST -H "$AUTH" $OMLX/admin/api/models/$M/load

The first load hashes the GGUF once; it took about 3 minutes here. Later loads only check file sizes and times.

Settings used

These are oMLX's defaults for Splash, under Integrations → External engines:

Setting	Value	Splash flag
KV cache precision	int8	--kv-format int8
Neural Engine prefill	on	(no --disable-ane)
SSD cache	oMLX SSD cache size; persistent	--max-cache-disk … --persistent-cache
Memory limit	empty = sized for the context window	--max-memory
Max context	model default, 262144 → 256K	--max-context

Splash caps the SSD cache so that 10 GiB of the volume stays free. Here it planned 54.4 GiB.

Usage

curl http://localhost:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "orcacyber-nano",
  "reasoning_effort": "none",
  "messages": [{"role": "user", "content": "Who are you? One sentence."}]
}'

Recommended sampling from the original card: temperature 1.0, top_p 0.95, top_k 20. These are also stored in the GGUF, and oMLX picks them up. Images go in as image_url with a base64 data URI.

Measured results (M5 Pro, 48 GB)

Engine	Decode tok/s
oMLX + Splash + DFlash2	52–80 (4.5–7.0 accepted tokens per draft cycle)
Ollama 0.40.10 (llama.cpp runner)	14.1
Docker Model Runner (llama.cpp)	13.8

The oMLX numbers come from three short requests with thinking off: 38-token prompts with code outputs of 600 and 476 tokens, plus one 551-token image prompt. Speculative decoding speed depends on content. The other two engines ran the same GGUF.

• Time to first token was 0.2 s on short prompts.
• Resident size reported by oMLX: 19.14 GB, covering the target, the draft and the KV/state plan.
• Vision was verified: an RGBA PNG screenshot came back correctly transcribed.

Limitations

• Splash serves chat only: /v1/completions returns HTTP 400.
• oMLX's own MTP, DFlash, TurboQuant KV and ANE-prefill toggles configure its MLX engines; they don't apply here.
• The draft was trained for Qwen3.8-27B, not for this fine-tune. Acceptance was still 4.5–7 tokens per cycle in the runs above.
• The model is tagged abliterated/uncensored in its GGUF metadata (general.tags).

License

The weights are not redistributed here. Use of the model is governed by the original repository's terms (license: other, license_name: qwen). The DFlash2 draft has its own license; see incoai/Qwen3.8-27B-DFlash2.