shadowrock-io/Harrier-OSS-v1-270M-Community-FP8

Harrier OSS v1 270M — Community FP8

Unofficial community quantization — not a microsoft release.

FP8 (W8A8, FP8_DYNAMIC) quantization of microsoft/harrier-oss-v1-270m (revision 31de22b), a 0.27B multilingual Gemma3TextModel embedding model using last-token pooling + L2 normalization. All credit for the model belongs to Microsoft; this repo only changes the numeric precision of the transformer's Linear projection weights and activations. 0.54 GB BF16 → 0.44 GB FP8, and fixture cosine parity vs the BF16 golden is 0.998 mean / 0.977 min (see Benchmarks below) with MTEB deltas under 0.005 absolute on every task measured.

Pick this variant for the best fidelity/size tradeoff on a GPU that supports FP8 (Ada/Hopper/ Blackwell). The companion NVFP4 variant (in progress) trades additional fidelity for a further ~2x size reduction on Blackwell-class GPUs.

What is quantized / Quantization details

• Method: FP8_DYNAMIC (compressed-tensors preset) via llm-compressor 0.13.0 — static per-channel FP8 E4M3 weight scales computed directly from the weights, dynamic per-token FP8 E4M3 activation quantization at inference.
• Scope: all 126 nn.Linear projections inside the 18 Gemma3TextModel decoder layers (self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj). Left at full precision: the embedding table (embed_tokens), every RMSNorm module, lm_head (tied to embed_tokens, unused for embedding output but present in the checkpoint), and the sentence-transformers last-token pooling + L2-normalize heads (outside the transformer module tree, always full precision).
• Calibration: 448-sample pinned manifest (calibration/manifest.json, seed 20260818871), bucketed by (input_type, token-length) — passages sampled from mteb/sts17-crosslingual-sts (disjoint from the eval fixture set), queries authored across the model's three documented instruct-prefix types (web_search_query/sts_query/bitext_query). Not strictly required for FP8_DYNAMIC's data-free weight scales, but run for parity with the NVFP4 export path.
• Full module inventory / provenance file: QUANT_PROVENANCE.json.
• Quantize/eval scripts ship under quantization/, eval/, calibration/; raw eval JSON under eval/evidence/.

Benchmarks / Validation

Two complementary evals, both against the BF16 golden baseline computed on the same hardware (arcade, RTX 5070 Ti) with the same fixture/prompt pinning (see the BF16 golden's own eval/README.md for the full protocol): (1) 190-fixture cosine parity via vLLM llm.embed(), gated by fixtures/tolerances.json (schema v1, parity_harness.py); (2) MTEB on 4 small tasks incl. one multilingual (STS17, 11 language-pair subsets), run via a thin vLLM-backed MTEB encoder wrapper (eval/run_mteb_vllm.py) — required for this variant, since a compressed-tensors FP8 checkpoint cannot execute a raw transformers/sentence-transformers forward pass (needs vLLM's fused FP8 dequant kernels); confirmed by a RuntimeError: expected mat1 and mat2 to have the same dtype when attempted directly.

eval	metric	BF16 (vLLM)	FP8	delta	gate	verdict
fixture parity (190 cases, non-stress)	cosine, p5 aggregate	1.0 (self)	0.99717	—	≥ 0.995	PASS
fixture parity (190 cases, non-stress)	cosine, per-case floor	1.0 (self)	≥ 0.9920	—	≥ 0.99	PASS
fixture parity — stress case (adv_long_near_max, ~6000 chars)	cosine	1.0 (self)	0.97724	—	≥ 0.95	PASS
MTEB STSBenchmark	main score	0.81516	0.81483	-0.00032	≤ 0.01	PASS
MTEB STS17 (mean/11 subsets)	main score	0.81042	0.80983	-0.00059	≤ 0.01	PASS
MTEB SciFact	NDCG@10	0.696	0.69126	-0.00474	≤ 0.01	PASS
MTEB NFCorpus	NDCG@10	0.30675	0.30703	+0.00028	≤ 0.01	PASS
MTEB macro delta (4 tasks)	mean absolute delta	—	—	0.00089	≤ 0.005	PASS

All gates pass. Fixture parity fails a naively-tight 0.999 p5 bar only on the deliberate extreme-length stress fixture and 3/190 borderline STS17 fixtures at 0.992-0.994 cosine — see fixtures/tolerances.json for the pre-registered-then-revised gate rationale (the original p5≥0.999 was miscalibrated before any real FP8 run existed; revised to match the house nemotron FP8 bar of 0.995 mean, applied here as a stricter p5). No case failed the revised gates. Raw result JSON: eval/evidence/fp8_fixture_parity_report.json, eval/evidence/fp8_mteb.json, eval/evidence/golden_bf16_mteb_vllm.json.

Protocol deviations from the paper / base card

• MTEB numbers for this variant use a custom vLLM-backed MTEB encoder wrapper (eval/run_mteb_vllm.py), not the base card's documented sentence-transformers load path — required because compressed-tensors FP8 checkpoints only run through vLLM's fused kernels. The BF16 golden's headline MTEB numbers (in the sibling BF16 golden docs) were run via sentence-transformers per the base card; the BF16-vs-FP8 deltas above instead compare BF16-via-vLLM vs FP8-via-vLLM (both in eval/evidence/) for an apples-to-apples backend comparison — the sentence-transformers BF16 numbers differ (e.g. NFCorpus 0.2215 vs 0.30675) because vLLM's io_processor plugin auto-applies the model's baked-in per-MTEB-task instruction prompts (mteb_v2_eval_prompts.json in the base repo) while the plain sentence-transformers path does not resolve those task-name-keyed prompts automatically (confirmed via mismatched- prompt-name warnings during that run) — a base-model serving nuance, not a quantization effect.
• STS17/NFCorpus/SciFact dataset revisions above are the mteb-library-pinned snapshot hashes as resolved at eval time (2026-08-18); re-running against a newer mteb release may resolve different default revisions.

Serving / Usage

from vllm import LLM

llm = LLM(
    model="shadowrock-io/Harrier-OSS-v1-270M-Community-FP8",
    runner="pooling",            # task=embed
    trust_remote_code=True,
    max_model_len=8192,          # base model supports up to 32768; raise if you need longer inputs
    gpu_memory_utilization=0.85, # lower if sharing the GPU with other processes
    dtype="bfloat16",            # activation/compute dtype; FP8 weights dequant/fuse automatically
)

# Documents: no prefix needed.
doc_emb = llm.embed(["Paris is the capital and largest city of France."])

# Queries: apply the base model's documented instruct prefix (three task-specific variants exist
# in the base repo's config_sentence_transformers.json `prompts` map — this is the general one).
query = "Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: what is the capital of France"
query_emb = llm.embed([query])

No PoolerConfig override is needed: vLLM auto-resolves pooling_type=LAST and L2-normalization directly from this repo's 1_Pooling/config.json + config_sentence_transformers.json (carried over unchanged from the BF16 base repo) — same as the BF16 model. FP8 requires no special vLLM flags beyond loading this repo; vLLM detects the compressed-tensors FP8 config in config.json automatically and uses fused FP8 kernels on GPUs that support them (Ada/Hopper/Blackwell — tested on an RTX 5070 Ti, Blackwell sm_120).

Notes that matter for correct embedding output:

• Pooling is last-token, not mean/CLS — confirmed via pooling_mode_lasttoken: true in this repo's 1_Pooling/config.json and empirically (all fixture embeddings are exactly unit-norm).
• Trailing whitespace/newlines measurably shift the embedding under last-token pooling (it changes which token is literally last) — this is expected behavior for this pooling strategy on the base model, unrelated to quantization; be consistent about trimming input text across a corpus.
• The query instruct-prefix meaningfully changes the embedding vs. the same text posed as a bare passage (asymmetric by design) — apply it only on the query side, never on documents.

Measured on: arcade (RTX 5070 Ti, 16 GB, Blackwell sm_120), vLLM 0.26.0, llm-compressor 0.13.0, compressed-tensors 0.18.0, torch 2.11.0+cu130, mteb (pinned per eval/evidence/*.json).

Caveats

• The 448-sample calibration set skews toward short-to-medium sentence-length inputs (STS-style sentences + short authored queries); very long documents (thousands of tokens) were exercised only by the single adversarial stress fixture, which shows a larger (but still gated, PASS) fidelity drop (cosine 0.977) than typical-length inputs (mean 0.998) — expect somewhat more quantization noise on long-document embeddings than on short queries/passages.
• MTEB coverage is 4 small tasks (STSBenchmark, STS17, SciFact, NFCorpus); the base model claims ~94 language variants and dozens of MTEB task types (classification, clustering, reranking, bitext mining) not evaluated here — only STS/retrieval-style tasks and 11 STS17 language pairs were measured.

Intended use & limitations

Intended uses are the base model's: general-purpose multilingual text embeddings for retrieval, clustering, semantic similarity, classification, bitext mining, and reranking. The base card's intended-use, safety, and coverage statements — microsoft/harrier-oss-v1-270m — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the benchmarks listed above and nothing beyond them: other languages, MTEB task types, and document-length regimes inherit the base model's behavior with FP8 quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the microsoft base model — the underlying model architecture and training are theirs:

@misc{harrier-oss-v1-270m,
  title  = {Harrier OSS v1 270M},
  author = {Microsoft},
  year   = {2026},
  url    = {https://huggingface.co/microsoft/harrier-oss-v1-270m}
}

License

MIT, inherited unchanged from the base model (see LICENSE; the base repo declares license: mit in its card metadata but ships no LICENSE file, so the canonical MIT text is reproduced here). Community build by ShadowRock; no Microsoft affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.