bge-reranker-base-ane-lin8

A Core ML build of BAAI/bge-reranker-base for the Apple Neural Engine, for use with Krill (v0.27.1 or later). It has int8 per-channel weights and four fixed-length buckets (64, 128, 256, 512 tokens). Every compute op is placed on the Neural Engine.

Parity with the GPU build served through Krill: Spearman 0.99962 on 730 scored pairs.

Use

krill pull bge-reranker-base-ane-lin8
krill pull bge-reranker-base
echo 'embed_device = "ane"' >> ~/.krill/config.toml    # or KRILL_EMBED_DEVICE=ane
krill serve
curl -s localhost:57455/v1/rerank -d '{"model":"bge-reranker-base","query":"what is a neural engine","documents":["Apple silicon accelerator for ML","a kind of bread"]}'
# x-krill-embed-device: ane

Scores text up to 512 tokens on the Neural Engine; longer input falls back to the MLX build. Pull the source model alongside this package (the command above does).

Requires Apple Silicon and macOS. Details: docs/ANE_EMBEDDER.md.