juliensimon/Octen-Embedding-8B-mlx-4bit

Octen-Embedding-8B for MLX (4-bit)

Octen/Octen-Embedding-8B converted to MLX so it runs natively on Apple Silicon: 4-bit affine quantization, group size 64, 4.3 GB on disk. The model embeds text in 100+ languages into 4096 dimensions, for search, retrieval, clustering and similarity. It is an 8B-parameter Qwen3 model fine-tuned for retrieval, with a 32K-token context.

Quality note. Embeddings move to a mean cosine of 0.97 from bf16, and top-1 retrieval in the test below drops by 3 tickets (1.7 points). Check it on your own data, and prefer the 8-bit or bf16 variant when memory allows.

This is a community conversion, not an official Octen release. The weights are Octen's, fine-tuned from Qwen/Qwen3-Embedding-8B. Both are Apache 2.0, and so is this conversion. Credit for the model goes to the Octen and Qwen teams.

The three variants

Repository	Weights	Size
juliensimon/Octen-Embedding-8B-mlx-bf16	bfloat16 weights, no quantization	15.1 GB
juliensimon/Octen-Embedding-8B-mlx-8bit	8-bit affine quantization, group size 64	8.0 GB
juliensimon/Octen-Embedding-8B-mlx-4bit (this repository)	4-bit affine quantization, group size 64	4.3 GB

The three sizes

Size	Variants	Dimensions	Time to embed 318 texts
0.6B	bf16, 8-bit, 4-bit	1024	5 to 6 seconds
4B	bf16, 8-bit, 4-bit	2560	35 to 55 seconds
8B	bf16, 8-bit, 4-bit	4096	70 to 80 seconds

The last column is the time to embed the 178 tickets and 140 articles of the test below, 8 texts per batch, up to 1,024 tokens each, on an M3 Max with 48 GB, measured on 21 October 2026 and given as a range over the three variants. Quantization saved disk and memory and did not make embedding faster in these runs. Larger models retrieve better on the test below: top 1 is 67.1% at 0.6B, 72.4% at 4B and 78.2% at 8B (MLX bf16).

Quick start

pip install mlx-embeddings

import mlx.core as mx
from mlx_embeddings import load

model, tokenizer = load("juliensimon/Octen-Embedding-8B-mlx-4bit")

texts = [
    "How many days of webhook logs can I see on the Business plan?",
    "Business accounts can view up to 90 days of webhook logs.",
    "Wie lange werden Webhook-Protokolle gespeichert?",
    "Pasta carbonara needs eggs, pecorino and guanciale.",
]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=1024, return_tensors="mlx")
embeddings = model(inputs["input_ids"], attention_mask=inputs["attention_mask"]).text_embeds  # (4, 4096)
embeddings = embeddings.astype(mx.float32)
embeddings = embeddings / mx.linalg.norm(embeddings, axis=1, keepdims=True)  # the model computes in bf16: renormalize in float32
mx.eval(embeddings)
print((embeddings @ embeddings.T).round(3))  # cosine similarities: related texts score high, the carbonara line scores low

The embedding is the last token's hidden state, normalized, as in the original model. The original model card asks for no instruction prefix, so queries and documents are embedded the same way.

Search across languages

import mlx.core as mx
from mlx_embeddings import load

model, tokenizer = load("juliensimon/Octen-Embedding-8B-mlx-4bit")


def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=1024, return_tensors="mlx")
    e = model(inputs["input_ids"], attention_mask=inputs["attention_mask"]).text_embeds.astype(mx.float32)
    e = e / mx.linalg.norm(e, axis=1, keepdims=True)
    mx.eval(e)
    return e


articles = [
    "Webhook logs: Starter accounts can view up to 7 days of logs, Team up to 30 days, Business up to 90 days.",
    "To create an invoice, select Invoices, click New invoice and choose the customer.",
    "Harborline retries a failed webhook delivery three times, after 10 seconds, 1 minute and 5 minutes.",
]
queries = [
    "Combien de jours de journaux de webhook puis-je consulter avec le plan Business ?",
    "Hoe maak ik een nieuwe factuur aan?",
]
scores = embed(queries) @ embed(articles).T  # (2, 3): one row per query
for q, row in zip(queries, scores.tolist(), strict=True):
    best = max(range(len(articles)), key=row.__getitem__)
    print(f"{q}\n  -> {articles[best][:60]}... ({row[best]:.2f})")

A French question finds the English webhook article, and a Dutch question finds the invoice article.

How faithful is the conversion

Run on 21 October 2026. The test: 170 support tickets in English, French, Dutch and German, each written from one of 140 English help articles (the development set of a workshop on fine-tuning small language models). The question is whether the ticket's own article is among the top k articles retrieved. The reference is the MLX bf16 conversion, because the original PyTorch model is too slow on CPU at this size to embed the set. The 0.6B conversion matched PyTorch within one ticket at every k (see the 0.6B cards), so this card checks the quantized variants against bf16.

Model	Top 1	Top 3	Top 5	Top 10	Cosine to bf16, mean	Cosine to bf16, worst	Same top 1 as bf16
MLX bf16 (reference)	78.2%	92.9%	97.1%	99.4%	1	1	100%
MLX 8-bit	78.2%	92.9%	96.5%	99.4%	0.9995	0.9928	98.9%
MLX 4-bit (this repository)	76.5%	92.9%	95.3%	98.8%	0.9733	0.9373	97.2%

8-bit matches bf16 within one ticket at every k. 4-bit loses 3 tickets at top 1 and 3 at top 5 while 97.2% of top-1 picks stay the same. One benchmark on one language mix is not a general claim: check retrieval quality on your own data before you rely on it. If you store embeddings, embed documents and queries with the same variant.

Reproduce the conversion

mlx_embeddings.convert (mlx-embeddings 0.1.0, MLX 0.32.4) crashes on this model while re-saving tokenizer.json, a read-only file it copied from the Hub cache. This short wrapper copies files without their read-only mode:

import importlib, os, shutil

c = importlib.import_module("mlx_embeddings.convert")
_copyfile = shutil.copyfile


def plain_copy(src, dst, **_):
    dst = os.path.join(dst, os.path.basename(src)) if os.path.isdir(dst) else dst
    return _copyfile(src, dst)


c.shutil.copy = plain_copy
c.convert("Octen/Octen-Embedding-8B", mlx_path="out", dtype="bfloat16", quantize=True, q_bits=4, q_group_size=64)

The sentence-transformers files of the original (modules.json and the like) are left out, because this repository holds MLX weights.

Limitations

• The same limits as the original model apply. See its card.
• Quantization changes embeddings a little, and more at 4 bits. Do not mix variants inside one index.
• The test above is mine and small. It measures fidelity to the original and retrieval on one kind of text.

Citation

Please cite the Qwen3 Embedding work this model builds on:

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}