tykan/KURE-Reranker-nano-GGUF

KURE-Reranker-nano GGUF

GGUF of nlpai-lab/KURE-Reranker-nano (revision 8514b9e) for llama-server --reranking. License and model credit belong to the original authors (Apache-2.0).

File	Precision	Size	RSS	sha256
KURE-Reranker-nano.F32.gguf	F32, same as the original	570 MB	0.68 GB	e7a374a0af01461a9f3d834beea66dc02999174b54261e99f271731c3d21174d
KURE-Reranker-nano.F16.gguf	F16	286 MB	0.39 GB	ab051c4c162047b8eb263809aa5fc0f320a30f1d1ae601074c8e2209d81c9b63

F32 vs F16 on PublicHealthQA (77 queries × 50 documents): max logit difference 0.006, top-10 order changed in 1 query.

Run

llama-server -m KURE-Reranker-nano.F32.gguf --reranking --pooling rank -c 8192 -b 8192 -ub 8192
curl -s localhost:8080/v1/rerank -d '{"query":"대한민국의 수도는?","documents":["대한민국의 수도는 서울입니다.","바나나는 과일입니다."]}'

relevance_score is the raw logit. Apply a sigmoid for 0–1 scores.

Known differences from the original tokenizer

llama.cpp's WordPiece tokenizer differs from the original in two cases. Hangul, Latin, digits, punctuation and emoji tokenize identically.

• Hanja (CJK ideographs). llama.cpp always splits each ideograph into its own word (大韓民國 → 大 韓 民 國), while the original keeps word pieces (大 ##韓 ##民 ##國). Hanja-heavy text gets different scores and can change ranking.
• Non-NFC input. llama.cpp does not apply NFC for WordPiece. Decomposed Hangul (NFD, e.g. from macOS file names) breaks into jamo and scores lose contrast. Normalize to NFC before sending.

Conversion

The tokenizer is WordPiece, while llama.cpp's ModernBERT converter assumes GPT-2 BPE. convert_wordpiece_modernbert.py patches only the vocab step: it writes the WordPiece vocab with add_bos=true, add_eos=true, add_sep=false, and disables lowercasing and accent stripping. Input becomes <s> query </s> document </s>, the same layout as the original. Weights, the ModernBERT encoder, the mean-pooled classifier head and sliding-window attention go through the stock converter unchanged.

python convert_wordpiece_modernbert.py /path/to/llama.cpp KURE-Reranker-nano --outtype f32 --outfile KURE-Reranker-nano.F32.gguf

Converted with llama.cpp cd959ba (b12110) and tested on llama.cpp 0.9.0, Apple M5 Pro.

Verification

• Parity with the original (sentence-transformers): 400 pairs, Pearson 1.0000 and identical ranking for both F32 and F16. Tokenizer mismatches 0/210 on benchmark text.
• nDCG@10: reranker-simple-benchmark protocol (gold + BM25 top-50).

Task	F16 GGUF	Original (model card)
PublicHealthQA	0.8078	0.8078
AutoRAGRetrieval	0.9741	0.9741
Ko-StrategyQA	0.8598	0.8582
BelebeleRetrieval	0.9766	0.9768

• Latency (Apple M5 Pro, Metal): 60 short documents 0.12 s (F32) / 0.11 s (F16), 300 documents 0.59 s / 0.55 s.