hoja-solutions/humanizer-NVFP4A16

humanizer 12B, NVFP4A16

This is jialinyyzz/humanizer with the weights of every decoder projection stored in NVFP4: FP4 (E2M1) values, one FP8 scale per 16 weights and one FP32 scale per tensor. Activations stay in bf16. The checkpoint is 8.25 GB instead of 23.9 GB, and vLLM loads it as a compressed-tensors model.

humanizer rewrites AI-written drafts in English and Chinese so they read as if a person wrote them, and it is trained to keep every number, name, date and quotation. The original model card covers the model, its prompt and its training. This card covers the quantization.

Quality

The comparison is against the bf16 weights, both run in vLLM 0.30.0, on the authors' held-out evaluation set: 209 English and 102 Chinese drafts, each followed by the authors' first published bf16 rewrite of it. Every token of the draft and the rewrite is scored, 187,551 positions in English and 99,166 in Chinese.

Build	Size	Top-1 vs. bf16, EN / ZH (avg)	KL to bf16, EN / ZH
bf16, second run in vLLM	23.9 GB	99.20% / 99.10% (99.15%)	0.0000 / 0.0001
NVFP4A16 (this build)	8.25 GB	95.03% / 94.51% (94.77%)	0.0120 / 0.0121

Top-1 is how often the quantized model's most likely next token is the same as bf16's. KL is averaged over positions and computed on bf16's 20 most likely tokens plus one bucket for the rest of the vocabulary, so it reads slightly lower than a full-vocabulary KL would. The first row is the floor that vLLM itself sets: running bf16 again in a different batch order changes the top token at 0.85% of positions.

For scale, the authors measured 95.2% top-1 and KL 0.0136 / 0.0146 for their quantization-aware Q4_K_M GGUF (7.6 GB), 93.9% for a standard Q4_K_M and 97.7% for Q6_K, using llama.cpp on 2,048-token chunks of the same evaluation set. The runtime and the exact tokens differ, so treat that comparison as approximate.

The rewrites tell the same story. Each build rewrote all 311 drafts once with the authors' sampling settings. A number counts as missing when a digit sequence in the draft does not appear in the rewrite (Chinese dates such as 9月17日 are normalized to 9.17 first). This check cannot see names or reworded claims.

Rewrites, one per draft	EN: rewrites missing a number	ZH: rewrites missing a number	ZH: numbers missing
bf16, authors' published samples	65 / 209	31 / 102	85 / 1,928
bf16, sampled again in vLLM	71 / 209	28 / 102	73 / 1,928
NVFP4A16 (this build)	64 / 209	32 / 102	68 / 1,928

Two bf16 samples differ from each other about as much as this build differs from either of them. Many of the "missing" numbers are dates and times that the rewrite writes another way, so these counts are not fact-error rates.

Speed

RTX 5090, vLLM 0.30.0, rewriting the 311 evaluation drafts (about 144,000 generated tokens):

Build	One request at a time	311 requests, up to 32 at once
bf16	59 tok/s	501 tok/s
NVFP4A16 (this build)	136 tok/s	1,585 tok/s

The single-request figure comes from four runs of 512 generated tokens after a warm-up. bf16's batched number is most likely held back by how little room 24 GB of weights leaves for the KV cache.

On this GPU vLLM runs the quantized layers on its Marlin kernel, which reads FP4 weights and computes with bf16 activations. At load time vLLM logs "Your GPU does not have native support for FP4 computation". That warning is expected.

Usage

Tested with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0).

vllm serve hoja-solutions/humanizer-NVFP4A16 --served-model-name humanizer --max-model-len 8192 \
  --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'

The model is a text-completion model. Send the instruction, the draft and the separator from prompt_format.json to /v1/completions, byte for byte:

import json, urllib.request

PF = json.load(open("prompt_format.json", encoding="utf-8"))

def humanize(draft: str, url: str = "http://127.0.0.1:8000") -> str:
    body = {"model": "humanizer", "prompt": PF["instr"] + "\n\n" + draft.strip() + PF["sep"],
            "temperature": 1.0, "top_p": 0.95, "top_k": 0, "min_p": 0.0,
            "repetition_penalty": 1.0, "max_tokens": 2048}   # no "stop": the model ends at EOS
    req = urllib.request.Request(url + "/v1/completions", json.dumps(body).encode("utf-8"),
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=900) as r:
        return json.load(r)["choices"][0]["text"].strip()

/v1/chat/completions also works, with one draft per user message. chat_template.jinja is the template from the authors' GGUF files with one change: it emits the BOS token itself, because vLLM and transformers don't add special tokens to chat-templated text. The template ignores system prompts and earlier turns. With the same seed, the chat endpoint returned exactly the same rewrite as /v1/completions, and the template produces the same token ids as the raw prompt.

Use temperature 1.0 and top-p 0.95, with top-k, min-p and the repetition penalty switched off, and never set stop strings such as "###". The generation_config.json here makes those the server defaults. The original file sets top_k: 64, which the authors tell users to switch off. This copy sets top_k: 0, the same default their GGUF files store.

How it was made

• llm-compressor 0.14.0 with compressed-tensors 0.19.0, transformers 5.17.0 and torch 2.14.0. recipe.yaml has the exact recipe.
• GPTQ (static activation ordering, block size 128, dampening 0.01) with the NVFP4A16 scheme on all 328 q/k/v/o/gate/up/down projections of the 48 decoder layers. The eight full-attention layers share keys and values and have no v_proj. The token embeddings (tied to the output layer) and the small vision and audio projections stay in bf16.
• Calibration used 384 sequences in the model's own prompt format, 365,486 tokens in all, at most 2,048 tokens each, taken from train rows of the redistributable split of jialinyyzz/humanizer-data. They are 184 DPO drafts with their chosen rewrite (56 in Chinese), 40 SFT drafts with their human original, and 160 RL drafts (100 in Chinese) with a rewrite sampled from the bf16 model. None of the evaluation drafts are among them. The dataset's DATA.md lists the licenses of its sources and the terms of the commercial models that wrote its drafts.
• The bf16 weights stayed on disk during calibration, and one decoder layer at a time went to the GPU. On a machine with 30 GB of RAM and an RTX 5090 the run took 8 minutes and peaked at 19.8 GB of RSS.

GPTQ with an importance-weighted (imatrix) scale search was also tried. It came out slightly worse: 94.30% top-1, KL 0.0146 / 0.0142.

Limits

• Only tested on an RTX 5090 (sm_120) with vLLM 0.30.0. Inference in transformers, Hopper GPUs and datacenter Blackwell were not tried.
• humanizer is used for text only. The vision and audio projections are present in bf16, but image and audio input were not tested.
• The AI-detector results on the original card were not measured again for this build.
• The authors' advice still applies: read the numbers, dates, names and the direction of each claim before you send a rewrite.

Credits

The model, its training data and the evaluation set are by the humanizer authors (jialinyyzz/humanizer, sgaofen/humanizer-local-model), released under Apache-2.0. It is a finetune of google/gemma-4-12B.