olka-fi/Hy3-MXFP4

Hy3 — MXFP4 (mixed precision)

A 4-bit MXFP4 quantization of tencent/Hy3. The original model card is preserved in full below.

Weight-only MXFP4 quant produced with qstream: the routed MoE experts (≈95% of the weights) are quantized to 4-bit; everything quality-sensitive stays BF16.

Size	172 GB (from ~598 GB BF16 source, ~29%)
Format	compressed-tensors mxfp4-pack-quantized (E2M1 4-bit + E8M0 group-32 scales)
Base	tencent/Hy3 — 295B MoE (21B active, 3.8B MTP), 80 layers + 1 MTP layer, 192 experts top-8 + 1 shared, 256K context

Note on HF's param count. HF reports fewer "params" than the base's 295B because it counts packed 4-bit storage elements (each U8 byte holds two FP4 weights) plus FP8 scales — not logical parameters. The logical model is unchanged: 295B total, 21B active.

What is quantized to what

Component	Precision	Why
Routed experts (…mlp.experts.{0..191}.{gate,up,down}_proj), incl. the MTP layer	MXFP4 (4-bit)	the bulk of the weights — the only place worth the size win
Shared expert (…mlp.shared_mlp.*)	BF16	always-on path — quantizing it amplifies error across every token for little gain
Attention, MoE router, dense-layer MLPs	BF16	sensitive / small — kept native
Embeddings, lm_head, all norms	BF16 / F32	unchanged

Quality & faithfulness

Functional evals run end-to-end on a single NVIDIA B300 (275 GB) — the checkpoint serves and generates coherently, so this is validated behavior, not just numeric proximity. Throughput with MTP enabled: ±850 tok/s

Metric	Result	What it shows
WikiText-2 perplexity (raw test, 290,674 tokens, 2048-ctx)	5.12	language modeling intact — a broken quant lands in the hundreds
GSM8K (full 1319-task, CoT, greedy)	1270 / 1319 (96.3%)	math/reasoning preserved
MTP acceptance (num_speculative_tokens=1, 7.5k draft tokens on GSM8K)	82.8%	the MTP draft layer speculates well — a free decode speedup, and lossless (GSM8K unchanged with it on)
Routed-expert SQNR (sampled, MXFP4 vs BF16)	≈ 19.0 dB (‖W−dequant‖/‖W‖ ≈ 11.25%)	reconstruction error is just the unavoidable 4-bit E2M1 rounding

Only the routed-expert FFN weights changed — ~95% of the weights are re-quantized to MXFP4; everything else (attention, shared expert, router, norms, embeddings) is bit-identical BF16 to the source.

Fidelity, footprint & provenance

• MTP preserved: the multi-token-prediction draft layer (layer 80) is served with --speculative-config '{"method":"mtp","num_speculative_tokens":1}' and reaches 82.8% draft-token acceptance. It is lossless — GSM8K is unchanged with it on (as speculative decoding must be).
• Footprint: ~172 GB of weights — fits a single large-VRAM GPU (verified on one B300, 275 GB) or multi-GPU tensor parallelism (e.g. 4× 80 GB). fastsafetensors loads it in ~160 s.
• Sibling NVFP4 quant of the same experts reconstructs at ≈21.3 dB (8.65% rel err) for ~6% more bits (FP8 group scale vs E8M0).
• Provenance: built with qstream @9f8cd0c, experts-only (--include_layers "*experts.*proj.weight"), MSE-optimal E8M0 scale selection + data-free activation-aware γ (input_layernorm.weight proxy).

Serving with vLLM

HYV3ForCausalLM is natively supported; loads via vLLM's compressed-tensors MXFP4 path. For the full production config (8-GPU TP, tool/reasoning parsers), see the original Deployment section below and point it at this repo.

vllm serve olka-fi/Hy3-MXFP4 \
  --served-model-name hy3 --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --load-format fastsafetensors \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Drop --speculative-config for the fixed (no-MTP) path; raise --tensor-parallel-size for GPUs smaller than the ~172 GB weight footprint.

Note on config.json. The ignore list carries both the canonical …mlp.shared_mlp.* module paths and their bare …mlp.* forms, so the shared expert stays BF16 on vLLM's HYV3 load path (which addresses that module by a bare prefix). If you regenerate the config, keep both forms.

---

⬇️ The original tencent/Hy3 model card follows, unmodified. ⬇️

中文 ｜ English

           

🖥️ Official Website  |   💬 GitHub

---

Table of Contents

• Model Introduction
• Stronger Agent Performance
• Product Experience: More Reliable, More Cost-Effective
• Benchmark Appendix
• News
• Model Links
• Quickstart
• Deployment
  • vLLM
  • SGLang
• Finetuning
• Quantization
• License
• Contact Us

---

Model Introduction

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. Following the Hy3 Preview launch in late April, we gathered feedback from 50+ product teams. We fixed various issues in task execution and interaction, and improved both the quality and scale of our post-training pipeline. Today, we are launching Hy3. It significantly outperforms similar-size models and rivals flagship open-source models with 2-5x the parameters. It also shows solid gains in utility across productivity tasks and real-world applications.

Property	Value
Architecture	Mixture-of-Experts (MoE)
Total Parameters	295B
Activated Parameters	21B
MTP Layer Parameters	3.8B
Number of Layers (excluding MTP layer)	80
Number of MTP Layers	1
Attention Heads	64 (GQA, 8 KV heads, head dim 128)
Hidden Size	4096
Intermediate Size	13312
Context Length	256K
Vocabulary Size	120832
Number of Experts	192 experts, top-8 activated
Supported Precisions	BF16

Stronger Agent Performance

Building on Hy3 Preview, we improved post-training data quality and diversity while scaling up RL training. Hy3 shows solid gains across reasoning, agentic workflows, and long-context tasks. Its performance is close to leading flagship models, both domestic and international.

In productivity scenarios such as coding, document processing, financial analysis, game development, and frontend design, Hy3 has made solid gains, positioning it as a reliable, cost-effective option.

We don't think public benchmark scores tell the full story. So we ran a blind test with 270 experts from various disciplines, working on real-world workflows, and collected 312 valid comparisons. Hy3 scored 2.67/4, outperforming GLM-5.1 at 2.51/4. The advantage was clearest in frontend development, CI/CD, and data & storage.

Product Experience: More Reliable, More Cost-Effective

Utility in production is not fully captured by benchmarks. Based on extensive user feedback and product telemetry, we identified real-world behavior issues that break product experience and improved the model's capabilities in those areas, earning uniformly positive feedback from product teams.

Output Formatting and Tool Calling Stability: We fixed multiple baseline reliability issues, bringing the model to production-grade standards across tool configurations and output constraints. Tool-call success rates and error recovery improved, and invalid calls that trigger infinite loops dropped. Hy3 also generalizes across different agent scaffoldings. On SWE-Bench Verified, accuracy variance across scaffoldings like CodeBuddy, Cline, and KiloCode remains within 4%.

World Knowledge and Anti-Hallucination: Internal knowledge and external hallucination are interconnected and critical to real-world product experience. Guided by the ideal behavior pattern: "answer when grounded, state when evidence is missing, do not conflate sources, do not fabricate data," we implemented fine-grained data cleaning and specific training constraints. In internal evaluations on real-world scenarios, Hy3's hallucination rate dropped from 12.5% to 5.4%, and commonsense error rates fell from 25.4% to 12.7%. These improvements materially reduce fact conflation, fabrication, and logical contradiction.

Complex Context Retention and Multi-turn Intent Tracking: Through joint optimization of SFT and RL, Hy3 improved on operational pain points like coreference resolution, ellipsis recovery, and multi-turn constraint inheritance. On internal comprehensive multi-turn tests, the issue rate dropped from 17.4% to 7.9%. It also posted significant gains on open-source long-dialogue benchmarks like MRCR, from 42.9% to 75.1%. Overall outputs are more concise while ensuring complex intents do not decay or drift over long-horizon interactions.

Benchmark Appendix

News

• 🔥 We open-source Hy3 and Hy3-FP8 model weights on Hugging Face, ModelScope, GitCode, and CNB.

Model Links

Model Name	Description	Hugging Face	ModelScope	GitCode	CNB
Hy3	Instruct model	🤗 Model	Model	Model	Model
Hy3-FP8	FP8 quantized instruct model	🤗 Model	Model	Model	Model

Quickstart

Deploy Hy3 with vLLM or SGLang first, then call the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Recommended parameters: temperature=0.9, top_p=1.0.

Reasoning mode: Set reasoning_effort to "high" for complex tasks (math, coding, reasoning) or "no_think" for direct responses.

See the Deployment section below for how to start the API server.

Deployment

Hy3 has 295B parameters in total. To serve it on 8 GPUs, we recommend using H20-3e or other GPUs with larger memory capacity.

vLLM

Build vLLM from source:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

Start the vLLM server with MTP enabled:

# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3

SGLang

Build SGLang from source:

git clone https://github.com/sgl-project/sglang
cd sglang
pip3 install pip --upgrade
pip3 install "transformers>=5.6.0"
pip3 install -e "python"

Launch SGLang server with MTP enabled:

python3 -m sglang.launch_server \
  --model tencent/Hy3 \
  --tp-size 8 \
  --tool-call-parser hunyuan \
  --reasoning-parser hunyuan \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3 \
  --speculative-algorithm EAGLE \
  --port 8000 \
  --served-model-name hy3

Finetuning

Hy3 provides a complete model finetuning pipeline. For detailed documentation, please refer to: Finetuning Guide

Quantization

We provide AngelSlim, a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms, low-bit quantization, and speculative sampling.

License

Hy3 is released under the Apache License 2.0. See LICENSE for details.

Contact Us

If you would like to leave a message for our R&D and product teams, welcome to contact us. You can also reach us via email:

📧 hunyuan_opensource@tencent.com

---

Hy3 is developed by the Tencent Hy Team.