UEmbed-4B-MLX-8bit

8-bit MLX quantization of the dense text-embedding tower of Alibaba-NLP/UEmbed-4B, packaged for Apple silicon. UEmbed-4B is built on the Qwen3.5 text architecture (hybrid linear attention); this pack runs it natively with mlx-lm and produces L2-normalized dense embeddings that track the upstream PyTorch reference (see Verification below).

This repo was briefly held private after an initial parity check failed. The failure turned out to be a chat-template mismatch in our verification harness (UEmbed-4B's shipped embedder uses the open <think> generation tail, unlike UEmbed-2B's closed block), not a model problem. With the correct template the pack passes parity on stock mlx-lm — see Verification. Along the way we also found a small epsilon deviation in mlx-lm's gated-delta q/k normalization (≤5e-5 cosine impact for this model); it has been reported upstream.

At a glance

Source	Alibaba-NLP/UEmbed-4B @ 2fab6202a2fb43481772eeb7a95f4e3d12a8ff3d
Format	MLX
Quantization	8-bit, group size 64 (9.0 bits/weight incl. overhead)
Disk size	4.4 GB

Variants

Variant	Quantization	Size	Status
UEmbed-4B-MLX-8bit (this repo)	8-bit, gs 64	4.4 GB	public
UEmbed-4B-MLX-4bit	4-bit, gs 32	3.0 GB	public

See also the smaller UEmbed-2B-MLX packs (note: 2B uses a different chat-template tail — enable_thinking=False).

Verification (2026-08-11, M-series Mac, stock mlx-lm git main)

• Quantization gate (this pack): worst cosine-vs-bf16 0.9998, worst cosine-vs-torch 0.9997; retrieval ordering PASS (margins >0.7)
• bf16 port parity: MLX bf16 vs upstream torch (transformers 5.4.0, shipped Qwen35Embedder, fp32): cosine >= 0.999967 on all 6 fixed parity inputs; same-input fp32-vs-fp32 pooled cosine 0.999923.

Usage

Requires: mlx-lm with qwen3_5 support (git main as of 2026-08-11; newer than the 0.31.3 release).

from mlx_lm import load
import mlx.core as mx
model, tok = load("majentik/UEmbed-4B-MLX-8bit")
def embed(text, instruction="Represent the user's input."):
    ids = tok.apply_chat_template(
        [{"role": "system", "content": instruction}, {"role": "user", "content": text}],
        add_generation_prompt=True)
    h = model.model(mx.array([ids]))[0, -1, :].astype(mx.float32)
    return h / mx.sqrt((h * h).sum())

Embedding contract: messages=[system(instruction), user(text)]; tokenizer.apply_chat_template(add_generation_prompt=True) — do NOT pass enable_thinking=False: UEmbed-4B's upstream embedder uses the template-default open <think> tail (this differs from UEmbed-2B); dense embedding = final-norm hidden state at the LAST prompt token, L2-normalized. (Upstream appends 16 EOS and pools at last-16 — identical under causal attention.) Default instruction: "Represent the user's input."

Scope — DENSE TEXT embeddings v1: the pack contains the Qwen3.5 text tower only (the upstream repo's visual tower and SPLADE sparse heads are NOT included; use upstream for image/video/sparse). Checkpoint keys normalized from the upstream tripled prefix (model.language_model.language_model.language_model.*) to the standard text layout.

Reproduce

Prerequisite: the upstream checkpoint stores the text tower under a tripled prefix (model.language_model.language_model.language_model.*) across sharded safetensors; normalize the keys to the standard Qwen3.5 text layout first, then convert the normalized checkpoint:

mlx_lm.convert --hf-path <normalized-UEmbed-4B> \
  --mlx-path UEmbed-4B-MLX-8bit \
  -q --q-bits 8 --q-group-size 64

License

CC-BY-4.0, inherited from the upstream Alibaba-NLP/UEmbed-4B. Attribution: this pack is a derivative of UEmbed-4B by Alibaba-NLP, redistributed with modifications (text-tower extraction, key normalization, 8-bit MLX quantization) under the terms of the CC-BY-4.0 license.