stellars/mmBERT-base-openvino-int8

jhu-clsp/mmBERT-base - OpenVINO int8

OpenVINO int8 IR of jhu-clsp/mmBERT-base, a multilingual ModernBERT encoder, quantized with NNCF SmoothQuant for CPU inference. int8 preserves the fp32 sentence-similarity structure at pearson 0.9789 - near-lossless, no onnxruntime dependency.

Purpose

Run multilingual sentence embeddings on a commodity CPU box with no GPU. Produces mean-pooled, L2-normalizable token embeddings for semantic-similarity work - document-distance scoring, optimal-transport statement matching, and retrieval pre-filtering - where a quantized encoder keeps the similarity geometry intact while halving memory.

Usage scenarios

• CPU-only / GPU-free serving - sentence embeddings where no GPU is available; ~310 MB int8 vs ~1.2 GB fp32
• Statement-level similarity - embed many short statements for optimal-transport or cosine comparison between documents
• Multilingual retrieval - cross-lingual claim/passage similarity (mmBERT covers a very wide language set)
• Not for - latency-critical GPU serving (use a GPU FP8/INT8 path instead), or tasks needing the fp32 last-decimal of similarity without first validating int8 parity

Model

• Base - jhu-clsp/mmBERT-base (multilingual ModernBERT, ~307M params, 768-dim)
• Format - OpenVINO IR (openvino_model.xml + .bin), int8
• Quantization - NNCF SmoothQuant: per-channel symmetric int8 weights, per-tensor static int8 activations, SmoothQuant alpha 0.6
• int8 parity vs fp32 - pearson 0.9789, spearman 0.9720, mean embedding cosine 0.9898 on a public generic sentence set
• Size - ~310 MB int8 (fp32 ~1.2 GB)
• Runtime - OpenVINO CPU (x86-64 AVX2 / AVX-512-VNNI; ARM via the OpenVINO ARM plugin)

Quantization details

• Pipeline - openvino.convert_model traces the ModernBERT graph (optimum-intel does not yet support the architecture for export), then nncf.quantize(model_type=TRANSFORMER) with SmoothQuant applies the int8 PTQ
• Calibration - 64 short English statements (public text)
• Why SmoothQuant - ModernBERT carries large per-channel activation outliers; SmoothQuant migrates them from activations into weights so per-tensor int8 activations stay accurate

Files

• openvino_model.xml / openvino_model.bin - the int8 IR
• config.json - base model config
• openvino_config.json - quantization metadata
• tokenizer.json / tokenizer_config.json - tokenizer

Usage

import numpy as np
import openvino as ov
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

repo = "stellars/mmBERT-base-openvino-int8"
xml = hf_hub_download(repo, "openvino_model.xml")
hf_hub_download(repo, "openvino_model.bin")
tok = AutoTokenizer.from_pretrained(repo)

core = ov.Core()
model = core.compile_model(core.read_model(xml), "CPU", {"PERFORMANCE_HINT": "LATENCY"})
out_port = model.outputs[0]

def embed(sentences):
    vecs = []
    for s in sentences:
        e = tok(s, return_tensors="np", padding="max_length", truncation=True, max_length=128)
        last = model({"input_ids": e["input_ids"], "attention_mask": e["attention_mask"]})[out_port]
        mask = e["attention_mask"][..., None].astype("float32")
        v = (last * mask).sum(1) / np.clip(mask.sum(1), 1e-9, None)
        vecs.append((v / np.linalg.norm(v, axis=-1, keepdims=True))[0])
    return np.stack(vecs)

emb = embed(["The cat sat on the mat.", "A feline rested on the rug."])
print("cosine:", float(emb[0] @ emb[1]))

License and attribution

Inherits the mit license of the base model jhu-clsp/mmBERT-base. Quantized to OpenVINO int8 with NNCF SmoothQuant.