leok7v/Qwen3.8-27B-coreml-q6

Model Card for Qwen3.8-27B (Core ML, Apple Neural Engine, 6-bit)

A compiled Core ML build of Qwen3.8-27B, packaged to run entirely on device on the Apple Neural Engine (ANE). It targets macOS (Apple silicon), needs no network at inference time, and streams tokens locally with the trunk weights palettized to 6 bits.

This repository holds the compiled model set only, not training or original weights. It is a self-contained artifact that a companion on-device runtime loads to drive generation. Nothing here calls out to a server.

This is an experimental build, not a shipped model. It exists as an online base for further work, because moving 20 GB between machines by hand is expensive. No released application fetches it, and nothing here is promised to stay put: the set may be re-emitted, re-quantized or replaced without notice.

The -q6 in the repository name is the weight format. A 4-bit build, if one is made, gets its own repository rather than a branch, so a downloader pins a quantization by URL.

Model Details

Model Description

Qwen3.8-27B is a dense (not mixture-of-experts) hybrid: of its 64 layers, 48 are Gated DeltaNet linear-attention blocks and 16 are full softmax-attention blocks, interleaved three-linear-one-attention. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall. It is multimodal: a 27-layer vision tower projects images into the language model's embedding space. Input and output embeddings are not tied, so a separate output-projection matrix ships.

This build converts the upstream model to Core ML and palettizes the trunk and head to 6 bits for on-device use. The vision tower stays fp16, matching the convention of the smaller builds in this series.

• Developed by: Qwen, Alibaba (base model); Core ML conversion by leok7v
• Model type: Hybrid Gated DeltaNet + attention causal language model, multimodal (image and text in, text out)
• Language(s): English and the languages of the base model
• License: Apache 2.0 (inherited from the base model)
• Converted from model: Qwen/Qwen3.8-27B

Model Sources

• Repository (this build): https://huggingface.co/leok7v/Qwen3.8-27B-coreml-q6
• Base model: Qwen/Qwen3.8-27B

Uses

Direct Use

On-device chat, summarization, drafting, question answering over supplied text, and image understanding, on an Apple silicon Mac with no network. The set also carries a multi-token-prediction drafter for self-speculative decoding, which a runtime may use to raise decode throughput without changing what is generated.

Out-of-Scope Use

Anything the base model's licence and card exclude. This build adds no alignment, no safety tuning and no filtering of its own; it is the upstream model in a different numeric format. It is not a general Core ML package that a single prediction call drives, and it is not intended for server deployment.

Bias, Risks, and Limitations

The base model's biases and failure modes carry over unchanged. Quantization to 6 bits is lossy: outputs will not match the upstream fp16 model token for token, and the gap is largest where the model was already uncertain.

Recommendations

Treat generations as drafts to verify, not as fact. For anything where accuracy matters, check against a source. The sampling parameters in generation_config.json are the base model card's own recommendations and are the sensible starting point.

How to Get Started with the Model

The repository contains a set of compiled .mlmodelc programs plus the tokenizer. It is designed to be loaded by an on-device runtime that implements paged key/value attention on the host between the Neural Engine trunk programs; it is not a single .mlpackage you call with one prediction. The included chat_template.jinja documents the chat format (roles, the thinking block, tool-call rendering).

Two behaviours of that template are worth knowing before wiring a runtime to it. preserve_thinking defaults to true, so past assistant turns re-render with their reasoning block rather than stripped; a runtime that does not retain reasoning should pass preserve_thinking=false. And reasoning_effort defaults to xhigh, which injects an instruction to reason thoroughly; low and medium are supported and cost fewer tokens per turn.

Technical Specifications

Model Architecture and Objective

Property	Value
Objective	Autoregressive next-token prediction
Hidden size	5120
Layers	64 (48 Gated DeltaNet + 16 attention)
Attention heads	24 query / 4 key-value (grouped-query)
Head dimension	256 (partial rotary, 64)
FFN size	17408
Vocabulary	248320
Context length	up to 262144 tokens
Tied embeddings	no (separate output projection)
Vision tower	27 layers, width 1152, projected to 5120
Weight format	6-bit palettized (per-grouped-channel k-means, group 16)
Activations	fp16

Files

• mf0of17.mlmodelc ... mf16of17.mlmodelc - the transformer trunk, cut into 17 programs around the attention layers. Each is a multifunction bundle whose "prefill" (batched), "decode" (single-token) and "verify" (speculative batch) graphs share one quantized weight blob, so the weights are resident once.
• head.mlmodelc - the vocabulary matrices as one bundle of four functions: the token-embedding matrix (shared by the single-token and batched embed lookups, one quantized copy), and the output projection in single-row and batched forms, since this model does not tie its embeddings.
• vision.mlmodelc - the vision tower and its merger, fp16.
• mtp_front.mlmodelc, mtp_back.mlmodelc - the multi-token-prediction drafter for self-speculative decoding.
• tile.mlmodelc, tile_prefill.mlmodelc, tile_carry.mlmodelc - weightless flash-attention kernels for the paged key/value cache.
• tokenizer.json, tokenizer_config.json, chat_template.jinja - the byte-level BPE tokenizer and chat template.
• config.json, generation_config.json - geometry, and the sampling parameters recommended by the base model card.

Compute Infrastructure

• Hardware: Apple silicon Mac with a Neural Engine. The set is about 20 GB on disk and the whole trunk is read per token, so decode is bandwidth bound.
• Software: macOS 15+. The first launch compiles the programs for the Neural Engine, which takes on the order of fifteen minutes for a set this size; the OS caches that build, so later launches are fast.

Citation

Please cite the upstream Qwen3.8-27B model. See the base model card at Qwen/Qwen3.8-27B for the canonical reference and license.

Model Card Contact

leok7v on Hugging Face.