satgeze/Hy3-1M-GGUF

Hy3 (Hunyuan) 1M Context GGUF — first GGUF conversion of Tencent's 299B hy_v3, extended to a million tokens. Preview quality, honestly labeled.

⚠️ EXPERIMENTAL, READ BEFORE USING

These are early GGUFs built BEFORE official llama.cpp support for the hy_v3 architecture existed. They require a community fork to run (details below), they are NOT yet needle-certified at long context (1M is baked in but unverified), and the MTP layer is not included because no engine can run it yet. Coherence-gated only. Treat as preview quality until the certification ladder is published on this card.

Tencent's Hy3 (299B MoE, ~17B active, hy_v3 architecture) quantized to GGUF and extended to a 1,048,576-token context with YaRN. As far as we can tell these are the first GGUF quants of Hy3 anywhere: at publish time (roughly 30 hours after Hy3's release) only MLX and MXFP4 quants existed, and mainline llama.cpp did not yet support hy_v3.

Honest status

• Architecture support: built against a community llama.cpp fork that implements hy_v3 (charlie12345/ROCmFPX). Mainline llama.cpp support is still in progress (open feature requests). You need an hy_v3-capable build to run these.
• Quants: Q2_K is a static quant (no importance matrix). IQ2_M and IQ1_M are importance-matrix quants - the imatrix was computed on our own Q2_K running RAM-resident, over a code-and-reasoning calibration set, then used to requantize from the archived f16. This is a bootstrap: a better imatrix could later be computed from a higher-fidelity runner.
• Coherence: all three passed a generation gate (coherent prose and correctly structured haiku). Not yet needle-certified for long context; NIAH ladders will follow.
• MTP: Hy3 ships a multi-token-prediction layer in its checkpoint, but no llama.cpp build (mainline or fork) implements hy_v3 MTP inference yet, so the converter drops it. The f16 is archived; an MTP build will follow if and when engine support lands. Nothing is faked here.
• Chat template: Hy3's stock template uses .format() calls that llama.cpp's Jinja engine rejects. chat_template_llamacpp.jinja in this repo is the fixed version; pass it with --chat-template-file.

What the gate actually produced (real, unedited)

Each quant was asked: "Name three colors and write a one-line haiku about the sea." Verbatim outputs, nothing cleaned up:

IQ1_M (62 GB, the roughest quant):

blue waves that whisper low, green depths that hold the silence of the sea.

IQ2_M (90 GB):

Three colors: blue, teal, and silver. waves drink the moonlight saltwind hums through silent blue- the sea forgets names

Q2_K (101 GB):

Blue, green, silver. The sea breathes in mist, silver light on rolling waves- quiet, endless blue.

My honest read

This is better than "it didn't collapse into repetition." A 1.7-bit quant of a 299B model (IQ1_M) still produces grammatical, on-theme lines; the two larger quants produce properly structured 5-7-5 haiku and correctly separate the color list from the poem. That is a genuinely good sign for how much of the model survives aggressive quantization here.

Two honest caveats, so you judge for yourself:

• This is a single short prompt. It proves coherence and basic instruction-following, not reasoning, long-context retrieval, or factual accuracy. Those need the certification ladder, which is not done yet.
• IQ1_M is visibly the weakest: it dropped the explicit color list and gave two lines instead of a haiku. Usable, but the imatrix IQ2_M is the one I would actually run if the memory fits.

If your experience differs, the raw files are right here - try it and form your own view.

Files

File	Size	Where	Notes
hy3-1M-IQ1_M.gguf	62 GB	ModelScope only	Smallest; fits a 128 GB Mac. no_think chat only — see warning below
hy3-1M-IQ2_M.gguf	90 GB	HF + ModelScope	Importance-matrix quality. Recommended minimum
hy3-1M-Q2_K.gguf	101 GB	HF + ModelScope	Static quant, the RAM-runner used to bootstrap the imatrix
chat_template_llamacpp.jinja	small	HF + ModelScope	Fixed template for llama.cpp

Field testing notes (real sessions, July 7)

Measured on a MacBook Pro M3 Max 128 GB, Metal, fork build, IQ1_M fully GPU-resident:

• Speed: 24-25 tok/s generation, ~41 tok/s prefill (llama-server timings, short prompts). A 299B MoE at chat speed on a laptop.
• no_think mode: coherent and usable, but noticeably chatty — filler sentences, over-explaining, occasional odd asides. Fine for casual chat and drafting; does not feel like a frontier chat tune.
• Thinking mode (reasoning_effort: high): collapses at IQ1_M. On a bare "hello" it produced hundreds of words of circular reasoning ("The user wrote hello... we should respond... but wait, the user wrote hello...") without ever converging to an answer. This is the 1.7-bit tax: short answers survive, reasoning chains do not. Do not use IQ1_M for coding or anything that needs thinking.
• Because of that, IQ1_M lives on ModelScope only; HF carries the quants we can recommend. IQ2_M field results will be added after the same test battery.
• Known cosmetic issue, all quants of this release: the <|hy_eos:opensource|> stop token can leak as visible text at the end of responses (tokenizer special flag, fix queued for the next build). Generation stops correctly; the text is just noise.

How to control thinking with llama.cpp: pass --chat-template-kwargs '{"reasoning_effort": "high"}' (or no_think / low) at server launch.

Building the hy_v3-capable llama.cpp

Mainline llama.cpp does not yet support hy_v3. Until it does, build the community fork (charlie12345/ROCmFPX). Two gotchas are baked into the steps below.

git clone --depth 5 https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX

# Gotcha 1: the fork ships empty web-UI files that break the embed step. Stub them.
for f in bundle.css bundle.js loading.html index.html; do
  [ -s tools/server/public/$f ] || echo "/* placeholder */" > tools/server/public/$f
done

# Apple Silicon (Metal):
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

# NVIDIA (CUDA). Gotcha 2: point at the real driver lib, not conda's stub, or you get
# "undefined reference to cuMemGetInfo" at link time.
cmake -B build -DGGML_CUDA=ON \
  -DCUDA_cuda_driver_LIBRARY=/usr/lib/x86_64-linux-gnu/libcuda.so.1 \
  -DCMAKE_EXE_LINKER_FLAGS="-L/usr/lib/x86_64-linux-gnu -lcuda" \
  -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/lib/x86_64-linux-gnu -lcuda" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

The resulting build/bin/llama-server loads these GGUFs. When mainline llama.cpp merges hy_v3, none of this is needed.

Run it (with an hy_v3-capable llama.cpp build)

llama-server -m hy3-1M-IQ2_M.gguf -c 262144 -np 1 --jinja \
  --chat-template-file chat_template_llamacpp.jinja

Start at 262K and raise -c toward 1048576 per available memory. On Apple Silicon the IQ1_M runs; on 192 GB+ the IQ2_M is the quality pick.

Credits

Base model: Tencent (Apache-2.0). hy_v3 llama.cpp implementation: community fork (charlie12345). GGUF conversion, 1M YaRN extension, imatrix bootstrap, and gating: SatGeze. Method and tooling: github.com/satindergrewal/aviary-1m.