erjigit17/qwen3-embedding-0.6b-ane-coreml-8bit

Qwen3-Embedding-0.6B — 8-bit Core ML / ANE (quantized)

An 8-bit linear-quantized variant of neuradex/Qwen3-Embedding-0.6B-CoreML-ANE, which did the actual work of converting Qwen/Qwen3-Embedding-0.6B to a Core ML package that runs on the Apple Neural Engine. This repository does not reimplement that conversion — it takes neuradex's published .mlpackage and quantizes its weights, published here because the quantization behavior at multiple bit-widths was measured and only 8-bit turned out worth shipping.

What this adds

Precision	Latency, b1_s128 (M4)	Cosine similarity vs. unquantized
fp16 (neuradex's published package)	~24.6 ms	1.000 (reference)
8-bit (this repo)	~18.6 ms	0.999
6-bit (tested, not published)	~19.0 ms	0.993
4-bit (tested, not published)	~19.0 ms	0.50–0.60 — unusable

8-bit is the whole win here: faster than fp16 with no measurable quality loss. 6-bit is no faster and measurably worse; 4-bit palettization destroys this model's embeddings outright — below 8-bit the latency cost isn't in the weight precision anymore, so further compression buys nothing but quality loss. If you only take one number from this: don't go below 8-bit for this model.

Files

• qwen3-b1_s128-8bit.mlpackage — fixed 128-token input shape.
• qwen3-b1_s512-8bit.mlpackage — fixed 512-token input shape.
• tokenizer.json — from neuradex/Qwen3-Embedding-0.6B-CoreML-ANE's tokenizer/ directory.
• quantize.py — the exact quantization call used to produce both packages from neuradex's source .mlpackage files. Two lines of coremltools.optimize.coreml, nothing more — reproducing this yourself is faster than downloading it from here.

Usage

Same input/output contract as the base package: Qwen/Qwen3-Embedding-0.6B is a decoder-only model using last-token pooling, with an instruction prefix on queries only (not on documents) — see the base model's and neuradex's model cards for the exact prefix and tokenization details, since this repo doesn't change any of that, only the weight precision.

import coremltools as ct

model = ct.models.MLModel(
    "qwen3-b1_s128-8bit.mlpackage",
    compute_units=ct.ComputeUnit.CPU_AND_NE,
)

Why 8-bit and not the published fp16

For a small (300M–600M-class) embedding model, weights dominate memory bandwidth on the Neural Engine more than compute does — quantizing them to 8-bit cuts that bandwidth roughly in half with a linear-symmetric scheme simple enough to introduce no measurable error (cosine 0.999 against the unquantized model). That's a real, verified latency win, not a theoretical one.

License

Apache-2.0, inherited from both Qwen/Qwen3-Embedding-0.6B and neuradex/Qwen3-Embedding-0.6B-CoreML-ANE. No additional restrictions from the quantization step.