Instructions to use giannisan/Hy3-ds4-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use giannisan/Hy3-ds4-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="giannisan/Hy3-ds4-gguf",
	filename="Hy3-ds4-IQ2XXS-AttnQ8.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use giannisan/Hy3-ds4-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf giannisan/Hy3-ds4-gguf
# Run inference directly in the terminal:
llama cli -hf giannisan/Hy3-ds4-gguf

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf giannisan/Hy3-ds4-gguf
# Run inference directly in the terminal:
llama cli -hf giannisan/Hy3-ds4-gguf

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf giannisan/Hy3-ds4-gguf
# Run inference directly in the terminal:
./llama-cli -hf giannisan/Hy3-ds4-gguf

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf giannisan/Hy3-ds4-gguf
# Run inference directly in the terminal:
./build/bin/llama-cli -hf giannisan/Hy3-ds4-gguf

Use Docker

docker model run hf.co/giannisan/Hy3-ds4-gguf

LM Studio
Jan
Ollama
How to use giannisan/Hy3-ds4-gguf with Ollama:
```
ollama run hf.co/giannisan/Hy3-ds4-gguf
```

Unsloth Studio

How to use giannisan/Hy3-ds4-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for giannisan/Hy3-ds4-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for giannisan/Hy3-ds4-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for giannisan/Hy3-ds4-gguf to start chatting

How to use giannisan/Hy3-ds4-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf giannisan/Hy3-ds4-gguf

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "giannisan/Hy3-ds4-gguf"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use giannisan/Hy3-ds4-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf giannisan/Hy3-ds4-gguf

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default giannisan/Hy3-ds4-gguf

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use giannisan/Hy3-ds4-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf giannisan/Hy3-ds4-gguf

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "giannisan/Hy3-ds4-gguf" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use giannisan/Hy3-ds4-gguf with Docker Model Runner:
```
docker model run hf.co/giannisan/Hy3-ds4-gguf
```

Lemonade

How to use giannisan/Hy3-ds4-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull giannisan/Hy3-ds4-gguf

Run and chat with the model

lemonade run user.Hy3-ds4-gguf-{{QUANT_TAG}}

List all available models

lemonade list

Hy3 (295B) GGUF for ds4/NeutronStar (SSD streaming, CUDA)

Mixed-precision GGUF of tencent/Hy3 (295B total / 21B active MoE, Apache 2.0) built for the NeutronStar hy3 branch: a CUDA port of ds4 that streams routed experts from disk, so the model runs on GPUs that cannot hold it.

Reference machine: RTX 4060 Ti 16GB, Ryzen 9900X, 32GB RAM, one Gen4 NVMe. Per token only 8 of 192 experts per layer are read (~3GB/token at this quant); attention, shared experts, and the router stay resident. Measured on that box: ~1.8 t/s generation with a 16GB host expert cache (68% hit rate), interactive chat with KV retained across turns.

Provenance note (read this)

This build is a requantization of an existing quant, not of the original checkpoint. Source was the IQ4-UD edition of YanissAmz/Hy3-295B-A21B-GGUF (routed experts IQ4_XS/IQ3_S, attention and actives already Q8_0).

What that means in practice:

The tensors this recipe keeps at Q8_0 (attention, shared experts, dense FFN, embeddings, output head) were already Q8_0 in the source, so they pass through essentially lossless.
Only the routed experts went through a second quantization step (IQ4_XS/IQ3_S to IQ2_XXS). At a 2-bit target the 2-bit quantization noise dominates, so the expected quality loss vs a from-source build is small, but it is not zero.

A full-precision rebuild is coming: the same recipe run directly from the original BF16 checkpoint (tencent/Hy3, 598GB). It will replace this file in this repo when ready. If you are downloading for long-term use and can wait, wait for that one.

Recipe

The layout targets ds4's streaming expert cache: routed experts must be uniform fixed-size slabs, and everything that makes decisions stays high precision. Same design as antirez's GLM-5.2 ds4 build, including the MTP layer riding at Q2_K because importance matrices never cover the draft layer (imatrix generation runs normal forwards, which skip it).

Tensors	Type	Why
routed experts, layers 1-79 (gate/up/down)	IQ2_XXS (imatrix)	streamed from disk per token; uniform slabs
routed experts, layer 80 (MTP)	Q2_K	no imatrix coverage exists for the draft layer
attention q/k/v/output, all layers	Q8_0	resident, paid once
shared expert + dense layer 0 FFN	Q8_0	resident
nextn.eh_proj (MTP glue)	Q8_0	tiny, no imatrix coverage
token embeddings, output head	Q8_0	ds4 embed kernel contract
router (ffn_gate_inp), expert bias, all norms	F32	decision makers stay exact

imatrix: the 125-chunk general-purpose matrix published with the source repo. Architecture string is hy-v3 (matching the source GGUFs; llama.cpp PR 25395 uses hy_v3, patched before quantizing).

Usage

git clone -b hy3 https://github.com/giannisanni/neutronstar
cd neutronstar && make ds4
./ds4 -m Hy3-ds4-IQ2XXS-AttnQ8.gguf --cuda --ssd-streaming \
  --ssd-streaming-cache-experts 64 --ctx 4096 --nothink

No prompt drops you into interactive chat (KV retained across turns). Useful knobs: DS4_CUDA_HOST_EXPERT_CACHE_GB=16 (host expert cache, the main speed lever; scale to your free RAM) and DS4_CUDA_PARALLEL_FETCH_THREADS=16.

MTP speculative decoding is not wired for Hy3 (and measurements on GLM/ DeepSeek show it cannot pay while expert streaming dominates eval cost); blk.80 is present in the file so it can be enabled later without requantizing.

Quantized with llama.cpp (PR 25395 + hy-v3 arch patch) on the reference machine.