leok7v/Qwen3.8-27B-coreml-q4

Model Card for Qwen3.8-27B (Core ML, Apple Neural Engine, 4-bit)

A compiled Core ML build of Qwen3.8-27B, packaged to run entirely on device on the Apple Neural Engine (ANE). It targets macOS (Apple silicon), needs no network at inference time, and streams tokens locally with the trunk weights palettized to 4 bits.

This repository holds the compiled model set only, not training or original weights. It is a self-contained artifact that a companion on-device runtime loads to drive generation. Nothing here calls out to a server.

This is an experimental build, not a shipped model. It exists as an online base for further work, because moving 14 GB between machines by hand is expensive. No released application fetches it, and nothing here is promised to stay put: the set may be re-emitted, re-quantized or replaced without notice.

Which of the two builds to take

This set is the sibling of Qwen3.8-27B-coreml-q6. They are the same conversion at two LUT widths, and the choice is about size: this one is about 14 GB on disk against the 6-bit set's 20 GB. Take the 6-bit build where the disk and memory are available, and this one where they are not.

Model Details

Model Description

Qwen3.8-27B is a dense (not mixture-of-experts) hybrid: of its 64 layers, 48 are Gated DeltaNet linear-attention blocks and 16 are full softmax-attention blocks, interleaved three-linear-one-attention. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall. It is multimodal: a 27-layer vision tower projects images into the language model's embedding space. Input and output embeddings are not tied, so a separate output-projection matrix ships.

The trunk, the head, the speculative-verify function and the drafter are all 4-bit. The vision tower is fp16.

Everything in this repository derives from the origin checkpoint's own tensors, including the vision tower and the drafter.

• Developed by: Qwen, Alibaba (base model); Core ML conversion by leok7v
• Model type: Hybrid Gated DeltaNet + attention causal language model, multimodal (image and text in, text out)
• Language(s): English and the languages of the base model
• License: Apache 2.0 (inherited from the base model)
• Converted from model: Qwen/Qwen3.8-27B

Model Sources

• Repository (this build): https://huggingface.co/leok7v/Qwen3.8-27B-coreml-q4
• Sibling build (6-bit): https://huggingface.co/leok7v/Qwen3.8-27B-coreml-q6
• Base model: Qwen/Qwen3.8-27B

Uses

Direct Use

On-device chat, summarization, drafting, question answering over supplied text, and image understanding, on an Apple silicon Mac with no network. The set also carries a multi-token-prediction drafter for self-speculative decoding, which a runtime may use to raise decode throughput without changing what is generated.

Out-of-Scope Use

Anything the base model's licence and card exclude. This build adds no alignment, no safety tuning and no filtering of its own; it is the upstream model in a different numeric format. It is not a general Core ML package that a single prediction call drives, and it is not intended for server deployment.

Bias, Risks, and Limitations

The base model's biases and failure modes carry over unchanged.

Quantization to 4 bits is lossy, and more so than to 6. Outputs will not match the upstream fp16 model token for token, and the gap is largest where the model was already uncertain. Anyone depending on this build should evaluate it on their own task rather than assume the 6-bit set's behaviour.

Recommendations

Treat generations as drafts to verify. The sampling parameters in generation_config.json are the base model card's own recommendations.

How to Get Started with the Model

The repository contains a set of compiled .mlmodelc programs plus the tokenizer. It is designed to be loaded by an on-device runtime that implements paged key/value attention on the host between the Neural Engine trunk programs; it is not a single .mlpackage you call with one prediction. The included chat_template.jinja documents the chat format (roles, the thinking block, tool-call rendering).

Two behaviours of that template are worth knowing before wiring a runtime to it. preserve_thinking defaults to true, so past assistant turns re-render with their reasoning block rather than stripped; a runtime that does not retain reasoning should pass preserve_thinking=false. And reasoning_effort defaults to xhigh, which injects an instruction to reason thoroughly; low and medium are supported and cost fewer tokens per turn.

Technical Specifications

Model Architecture and Objective

Property	Value
Objective	Autoregressive next-token prediction
Hidden size	5120
Layers	64 (48 Gated DeltaNet + 16 attention)
Attention heads	24 query / 4 key-value (grouped-query)
Head dimension	256 (partial rotary, 64)
FFN size	17408
Vocabulary	248320
Context length	up to 262144 tokens
Tied embeddings	no (separate output projection)
Vision tower	27 layers, width 1152, projected to 5120
Weight format	4-bit palettized (per-grouped-channel k-means, group 16)
Activations	fp16

Files

• mf0of17.mlmodelc ... mf16of17.mlmodelc - the transformer trunk, cut into 17 programs around the attention layers, 600 to 740 MB each. Each is a multifunction bundle whose "prefill" (batched), "decode" (single-token) and "verify" (speculative batch) graphs share one quantized weight blob, so the weights are resident once.
• head.mlmodelc - the vocabulary matrices as one bundle of four functions: the token-embedding matrix (shared by the single-token and batched embed lookups, one quantized copy), and the output projection in single-row and batched forms, since this model does not tie its embeddings.
• vision.mlmodelc - the vision tower and its merger, fp16, unchanged from the 6-bit set.
• mtp_front.mlmodelc, mtp_back.mlmodelc - the multi-token-prediction drafter for self-speculative decoding.
• tile.mlmodelc, tile_prefill.mlmodelc, tile_carry.mlmodelc - weightless flash-attention kernels for the paged key/value cache.
• tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, chat_template.jinja - the byte-level BPE tokenizer and chat template.
• config.json, generation_config.json, preprocessor_config.json, video_preprocessor_config.json - geometry, the sampling parameters recommended by the base model card, and the origin's own image and video preprocessing settings.

Compute Infrastructure

• Hardware: Apple silicon Mac with a Neural Engine. The set is about 14 GB on disk and the whole trunk is read per token, so decode is bandwidth bound.
• Software: macOS 15+. The first launch compiles the programs for the Neural Engine, which takes a while for a set this size and longer on a fanless machine; the OS caches that build, so later launches are fast.

Citation

Please cite the upstream Qwen3.8-27B model. See the base model card at Qwen/Qwen3.8-27B for the canonical reference and license.

Model Card Contact

leok7v on Hugging Face.