shadowrock-io/LocateAnything-3B-Community-FP8-Dynamic

LocateAnything-3B — Community FP8-Dynamic

Unofficial community quantization — not an NVIDIA release.

FP8 W8A8 build of nvidia/LocateAnything-3B (revision c32291ca), validated against a BF16 baseline of the published checkpoint. All credit for the model belongs to NVIDIA; this repo changes only the numeric precision of the decoder weights. Checkpoint size drops from 7.6 GB to 4.0 GB, and on our COCO val2017 grounding benchmark the FP8 model scores inside the BF16 model's own seed-to-seed variance band on every metric.

Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native FP8 execution. The companion NVFP4 build halves the footprint again for memory-constrained Blackwell deployments.

What is quantized

Scheme: FP8_DYNAMIC via llm-compressor — per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the Qwen2.5-3B language-model decoder linears are quantized. Everything the grounding head depends on stays BF16:

• MoonViT vision tower and the mlp1 connector
• embed_tokens / lm_head (tied), which protects the <0>–<1000> coordinate token rows

Validation

All evaluation drives the model through its own hybrid parallel-box-decoding runtime (batch_utils.generate_batch_hybrid) under the model's canonical sampling settings (temperature 0.7, top_p 0.9, repetition_penalty 1.1) with the torch RNG reseeded per case, which makes runs bit-reproducible on a fixed stack.

Grounding benchmark, 2×2 design. 100 COCO val2017 images (first 100 ids with annotations, ascending — a deterministic slice), one query per ground-truth category per image, 288 queries. Two seeds per model, so the BF16 seed-to-seed spread is measured rather than assumed. LocateAnything emits boxes without confidence scores, so we report score-free greedy-matched precision/recall/F1 instead of score-ranked mAP; the paper's own headline metric family is mean F1.

model / seed	P@0.5	R@0.5	F1@0.5	mean F1 (0.50:0.95)	mean matched IoU
BF16, seed A	0.7345	0.5501	0.6290	0.4761	0.8536
BF16, seed B	0.7329	0.5303	0.6154	0.4691	0.8559
FP8, seed A	0.7199	0.5472	0.6218	0.4750	0.8560
FP8, seed B	0.7311	0.5331	0.6166	0.4674	0.8535

The BF16 model moves 0.0136 F1@0.5 between seeds; FP8 sits inside that band on every metric, with same-seed deltas 0.1–0.5× the seed variance. The paper reports COCO mean F1 ≈ 52.1 for the full protocol; our BF16 baseline lands ≈ 47.6 under the deviations listed below, and the number that matters here is FP8-vs-BF16 under an identical protocol.

Contract test. The quantized export passes the same gate as the BF16 snapshot: parallel-box-decoding forwards active (mtp_forwards > 0), coordinate token ids intact and finite, tied lm_head verified, valid labeled boxes on the fixture set.

Fixture parity. On the 12-case fixture set, 10/12 responses are exact or near-exact (coordinate deltas within tolerance); the 2 remaining are crowded scenes where sampling legitimately picks a different box order.

Teacher-forced diagnostics. Full-sequence forced decoding over the BF16 golden responses shows all top-1 flips confined to coordinate positions (no label or structure flips), mostly ±1–4 coordinate units on a 0–1000 grid.

Raw results ship in this repo under evidence/: per-query benchmark predictions for both seeds, fixture parity report, and the teacher-forced capture. Reproduction scripts (quantizer, contract test, teacher-forced capture, COCO benchmark harness) live in the source repo.

Protocol deviations from the paper

• 100-image deterministic slice of val2017, not the full split
• greedy best-first IoU matching (score-free), not a ranked matcher
• queries are per-GT-category per image (assumes category presence is known)

Usage

vLLM (true W8A8)

vLLM reads the compressed-tensors config and runs real FP8 weight and activation kernels:

from vllm import LLM
llm = LLM(model="shadowrock-io/LocateAnything-3B-Community-FP8-Dynamic", trust_remote_code=True)

transformers (decompress-on-load)

run_compressed: false is set in config.json: transformers decompresses the FP8 weights to BF16 at load. Numerics of the quantized weights are preserved, but activations run in BF16, so this path measures weight-quantization error only. Pin transformers==4.57.1 and compressed-tensors==0.12.2; the model's inference contract (sampling settings, hybrid decode runtime) is documented in the upstream README, which ships in this repo together with the pinned model code.

from transformers import AutoModel
model = AutoModel.from_pretrained(
    "shadowrock-io/LocateAnything-3B-Community-FP8-Dynamic",
    trust_remote_code=True, torch_dtype="bfloat16")

Greedy decoding is not supported by the base model (it never emits the stop token); use the canonical sampling settings above.

Prior art

Other community quantizations of this model exist (GGUF, MLX 4/8-bit, ONNX INT4, an NVFP4). None we found publish accuracy measurements against the BF16 baseline; this release's purpose is quantization with evidence. Reproduction scripts (quantizer, contract test, teacher-forced capture, COCO benchmark) live in the source repo.

Intended use & limitations

Intended uses are the base model's: open-vocabulary visual grounding and object detection from natural-language queries. The base card's intended-use, safety, and coverage statements — nvidia/LocateAnything-3B — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the COCO benchmark above and nothing beyond it: other image domains, query styles, and hardware/runtime combinations inherit the base model's behavior with quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the NVIDIA base model — the grounding capability is theirs:

@misc{nvidia2026locateanything,
  title  = {LocateAnything-3B},
  author = {NVIDIA},
  year   = {2026},
  url    = {https://huggingface.co/nvidia/LocateAnything-3B}
}

License

NVIDIA non-commercial license, inherited unchanged from the base model — see LICENSE. Quantized weights are a derivative of NVIDIA's checkpoint and carry the same terms. No COCO images are redistributed here; the benchmark fetches them by id from cocodataset.org. Community build by ShadowRock; no NVIDIA affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.