manateelazycat/ReaderLM-v2-NVFP4

ReaderLM-v2 calibrated NVFP4 for Jetson Thor

This is a ModelOpt 0.43.0 offline PTQ derivative of jinaai/ReaderLM-v2 revision 1d07078459ee1e880a22d67387b5e683d50a6e4b, prepared by manateelazycat. It is not an official Jina AI quantization.

196 linear weight tensors are packed NVFP4 with 196 FP8 block scales. Embedding and the tied output head remain BF16. KV cache is not quantized. The independent synthetic calibration set has 32 mixed Chinese/English HTML documents; calibration and benchmark fixtures are separate. conversion-manifest.json records the original complete conversion output (its optional source_metadata directory is omitted from this serving bundle).

Tested on Jetson Thor SM110 with vLLM commit 635dd6aae656, torch 2.13.0+cu130, BF16 KV and CUDA Graph. Use --quantization modelopt_fp4 --linear-backend cutlass. The default FlashInfer FP4 JIT path exhibited a large cold-start memory peak in our test; explicitly selecting the precompiled CUTLASS backend avoided that peak.

A 1.5 GiB KV pool accommodates the tested four requests of about 10.6K input plus 512 output tokens. It does not hold four full 32K requests at once. Synthetic Markdown and JSON checks passed, but this is not a broad real-world accuracy evaluation.

Weight SHA-256: 6d041dd97408eff0fa7e341675026cb15d6a5969fc05af5f93042c90ca5cb921.

License / 许可证

CC-BY-NC-4.0, inherited from Jina AI ReaderLM-v2. 非商业许可；商业使用请向 Jina AI 获取授权。See NOTICE.md.