PII-Redact-Name-GGUF

GGUF quantization(s) of OpenPipe/PII-Redact-Name for CPU/GPU inference via llama.cpp.

This is the same model family as the MLX 4-bit quant, so both appear together as quantizations of the base model.

Quantization details

• Model type: llama

• Format: GGUF

• Quants available: Q4_K_M, Q8_0

• Q4_K_M — 807 MB — ~26.5 t/s on 16 CPU cores — best size/speed tradeoff

• Q8_0 — 1.32 GB — ~16.4 t/s on 16 CPU cores — near-lossless

• F16 — 2.48 GB — same precision as the base, therefore not published

Behavioural check: across a tagged sample the F16, Q8_0 and Q4_K_M outputs were byte-for-byte identical, so 4-bit loses nothing observable on short documents.

Why these exist

The full-precision model is served through transformers, which processes one request at a time. GGUF served by llama-server uses continuous batching (--parallel), so many documents decode concurrently and share each weight read. A 1B model on a GPU is memory-bandwidth bound, so ~4x smaller 4-bit weights convert directly into throughput.

Usage

llama-server -m Pii-Redact-Name-Q4_K_M.gguf --port 8080 --parallel 8 -c 65536

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"My phone is 555-867-5309"}]}'

This model is a redactor: it echoes the document back with PII wrapped in XML-ish tags, e.g. <PII:phone_number>555-867-5309</PII:phone_number>. The tag vocabulary includes person_name, phone_number, email, street_address, banking_number, medical_condition, organization_name, password, secure_credential and other_id.

Set the token budget to cover the input. The model early-stops rather than filling a budget, so a generous cap costs nothing at runtime -- but a cap that is too SMALL truncates the echoed document mid-tag and silently corrupts the output. A 1,820-token input emits ~2,712 tokens, so a 1024 cap loses most of the document without erroring.

One server process serves exactly one GGUF. To serve this model together with the General variant behind a single OpenAI-compatible endpoint, run two servers on separate ports and route by the model field.

Related

• MLX 4-bit quant: PII-Redact-Name-oQ4
• General variant: PII-Redact-General-GGUF