kimrinking/pplx-decider-flash-v1.2-fp8

PPLX Decider Flash v1.2 — FP8

Static FP8 quantization of perplexity-ai/pplx-decider-flash-v1.2, a model for scoring decision options. The included adapter serves text requests and returns per-option probabilities.

Quantization

Component	Configuration
Method	NVIDIA ModelOpt 0.44.0, FP8_DEFAULT_CFG+no_visual
Weights and activations	FP8 E4M3, static per-tensor scales
Calibration	Max calibration, 160 decision requests
Language-model Linear modules	248 quantized before vLLM fusion
Convolution, vision tower, decision readout	BF16
Weight file	5.51 GB

Full attention remains noncausal; linear attention remains causal. The original tokenizer, protocol, and decision readout are included.

Evaluation

DGX Spark GB10, vLLM 0.32.0, FlashInfer GDN, 5 independent server starts with the same restored cache. Throughput is mean ± sample standard deviation after warm-up, with 1,024 requests per concurrency setting.

Metric	Result
Accuracy	645/838 · 76.969%
Accuracy excluding known calibration overlap	544/695 · 78.273%
Throughput, concurrency 1	20.469 ± 0.063 req/s
Throughput, concurrency 8	32.084 ± 0.300 req/s
Throughput, concurrency 32	33.318 ± 0.090 req/s

This is the custom DecisionCore evaluation, not the official Decision Index. Two shared over-length items were excluded from the 840-item set. All selected answers and probabilities matched across the 5 successful starts under the fixed cache. One additional startup failed before readiness with a CUDA illegal-instruction error. Records are in provenance/.

Usage

Validated environment: Linux aarch64, CUDA 13.0, vLLM 0.32.0, PyTorch 2.13.0+cu130, Transformers 5.19.0, FlashInfer 0.7.1.

Prepare compatible CUDA builds of vLLM and PyTorch, including the CUDA development libraries. runtime/requirements-validated.txt records the measured environment; it is not a fresh-install lock file. The adapter imports source/src directly, so installing ./source is unnecessary and can replace runtime dependencies.

hf download kimrinking/pplx-decider-flash-v1.2-fp8 \
  --local-dir ./pplx-decider-flash-v1.2-fp8
cd pplx-decider-flash-v1.2-fp8
bash runtime/start.sh

After the engine is ready:

python3 runtime/example.py --base-url http://127.0.0.1:8140

The API exposes POST /v1/decisions and POST /v1/systemone, accepting state and questions. The included server is text-only. A fresh dependency installation and execution from a downloaded Hub snapshot have not been tested.

License

Apache-2.0. The upstream LICENSE, NOTICE, source, and quantization recipe are included.