znaat/modernbert-coreml

ModernBERT-base in Core ML

ModernBERT-base converted to Core ML with its masked-LM head, at a fixed sequence length of 64.

Verified against the PyTorch model rather than assumed: correlation of the logits at a masked position is 0.9964 to 1.0000 over four sentences. verify.py reproduces it.

The capital of Ireland is [MASK].
   torch    Dublin, Cork, Belfast, London
   coreml   Dublin, Belfast, Cork, London

About 7 ms per call on an M-series Mac after warm-up, against ~55 ms for the same model in PyTorch on the same machine. The first call pays roughly 7 s of Core ML compilation, so warm it once at launch.

Why this exists

There is an earlier Core ML conversion of ModernBERT on the Hub. It is wrong, in both its full-precision and 4-bit forms, and nothing about it says so — it loads, it runs, and it returns confident nonsense.

The capital of Ireland is [MASK].
   the earlier conversion    £, isation, organisation, ised

Measured against PyTorch, its logits correlate at 1.0000 at a sequence length of one and 0.17 to 0.39 at any real length. Its converter traced the model with a single token and a flexible RangeDim. At one token ModernBERT's sliding-window attention mask is optimised away; past it, the graph is wrong.

That failure is silent, which is the reason for this repository. If you convert this model yourself, check the correlation at a real length — not the top predictions of a short example, which can look plausible while the model is broken.

Using it

import coremltools as ct, numpy as np
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = ct.models.MLModel("ModernBERT-base-64.mlpackage")

text = "The capital of Ireland is " + tok.mask_token + "."
encoded = tok(text, return_tensors="np")
ids = np.full((1, 64), tok.pad_token_id, np.int32)
ids[0, : encoded.input_ids.shape[1]] = encoded.input_ids

logits = model.predict({"input_ids": ids})["logits"][0]
at = int(np.where(encoded.input_ids[0] == tok.mask_token_id)[0][0])
print([tok.decode([int(i)]) for i in np.argsort(-logits[at])[:5]])

tokenizer.json is here too, copied unchanged from answerdotai/ModernBERT-base. It is byte-level BPE, 50280 entries and 50009 merges, with an NFC normaliser. You do not need it from Python — AutoTokenizer fetches its own. It is here so a Swift, Rust or C++ caller can get the model and the tokenizer that matches it from one place, without a Python step.

Downloading from the Hub leaves the weights as a symlink, and the Core ML compiler does not follow one. Copy the package with cp -RL before loading it.

Two things to know before you rely on it

Fidelity falls off with length, and 64 is what is verified here. The same recipe converted at 128 correlates at only 0.90 to 0.99 across the same four sentences, against 0.9964 to 1.0000 at 64. The conversion emits an overflow encountered in cast warning while building the attention mask, and the longer the window the more it seems to cost. I published the 128 build before checking it and had to take it down — which is the same mistake this repository exists to warn about, so it is recorded here rather than quietly fixed.

Verify at the length you convert. Do not assume a recipe that is faithful at one length is faithful at another. verify.py takes --length.

The length is fixed, and padding is attended. The attention mask is ones_like(input_ids) inside the graph, so padding tokens are read as text. On one sentence-boundary task this cost nothing measurable, but that is a property of that task and not a guarantee. Convert at a length close to your inputs, or convert several — convert.py --length N.

Taking attention_mask as a real input is the obvious fix and does not work: ModernBERT then builds the mask with new_ones, which coremltools has no conversion for.

Quantising costs more than it saves, at least here. Same conversion, same sentence-boundary task, 150 real sentences:

	size	latency	quality
float	286 MB	10 ms	reference
int8	152 MB	23 ms	indistinguishable
int4	81 MB	21 ms	shifted, and worse where it matters

Those three were measured on the 128 build, before its fidelity problem was found. The ranking is about quantisation and should carry over, but the numbers were taken on a build that is not faithful — treat them as a direction, not a measurement.

int8 halves the size and doubles the latency, because the weights are dequantised on every pass. int4 changes the distribution enough to matter.

Reproducing

Pin the toolchain. transformers 5.x emits new_ones in ModernBERT's attention path and coremltools cannot convert it — that failure also hits BERT and DistilBERT, so it is the toolchain and not this model.

pip install torch==2.7.0 transformers==4.48.3 coremltools==9.0
python convert.py --length 64
python verify.py ModernBERT-base-64.mlpackage --length 64

Licence

Apache-2.0, from ModernBERT-base. Only the conversion is new here.