Instructions to use quark75/Qwen3.8-27B-MXFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4 # Run inference directly in the terminal: llama cli -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4 # Run inference directly in the terminal: llama cli -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4 # Run inference directly in the terminal: ./build/bin/llama cli -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Use Docker
docker model run hf.co/quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
- Llama new
- LM Studio
- Jan
- Ollama
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with Ollama:
ollama run hf.co/quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
- Unsloth Desktop
- Pi
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with Docker Model Runner:
docker model run hf.co/quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
- Lemonade
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Run and chat with the model
lemonade run user.Qwen3.8-27B-MXFP4-GGUF-MXFP4
List all available models
lemonade list
- Hermes Agent
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use quark75/Qwen3.8-27B-MXFP4-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-27B โ MXFP4
An MXFP4 quantization of Qwen/Qwen3.8-27B, converted directly from the original BF16 safetensors. Same architecture as the base model (dense, hybrid Mamba/attention โ qwen35 arch with native SSM layers alongside standard attention blocks), same license (Apache-2.0).
Why MXFP4 on a dense model
MXFP4 support in llama.cpp is usually associated with MoE models (it was added for OpenAI's gpt-oss family), and the built-in MXFP4_MOE quantization preset only applies MXFP4 to mixture-of-experts tensors โ on a dense model like this one, using that preset directly would silently fall back to plain Q8_0 with none of MXFP4's benefit.
This build instead uses llama-quantize --tensor-type <pattern>=mxfp4 overrides to force MXFP4 onto the dense linear weight tensors directly (attn_qkv, attn_gate, ffn_gate, ffn_up, ffn_down, ssm_out), while leaving norms, biases, and small SSM state parameters at their default (mostly Q8_0) precision.
The native MTP (multi-token prediction / speculative decoding) draft head bundled in the base model's weights (blk.N.nextn.* tensors) survives this conversion intact โ confirmed via --spec-type draft-mtp in llama.cpp, ~82% draft acceptance observed in testing.
Benchmarks
All benchmarks run on an AMD Radeon PRO V620 (RDNA2, ROCm/HIP backend), llama.cpp build ece963f (~August 2026 mainline), against the original Q4_K_M quantization of the same base model, same serving flags (-kvu, --ctx-size 65536, --spec-type draft-mtp, -np 4) unless noted.
Speed
Measured with llama-benchy, 3 runs per point, --pp 1024 --tg 128:
| Context depth | Q4_K_M (pp / tg, tok/s) | MXFP4 (pp / tg, tok/s) | Gain |
|---|---|---|---|
| 0 | 249.5 / 24.1 | 325.3 / 34.1 | +30% / +42% |
| 4096 | 283.7 / 23.0 | 385.8 / 29.2 | +36% / +27% |
| 16384 | 276.4 / 22.8 | 373.3 / 29.7 | +35% / +30% |
File size: 16.9GB (MXFP4) vs. 17.1GB (Q4_K_M) โ the speed gain comes at essentially no size cost.
Serving tip: if you don't need real request concurrency, -np 1 measured 8โ25% faster token generation than -np 4 in this build (prompt processing was unaffected) โ likely per-step scheduler overhead that scales with configured slot count, not something specific to MXFP4.
Quality
Evaluated against a 39-prompt suite spanning logic/reasoning, coding, hallucination resistance, instruction-following, agentic tool-calling, multi-turn memory, and language-specific correctness (Rust, Yew/WASM, Python), graded twice by independent LLM judges:
| Judge | Scored | Average | Notable failures (< 7.0) |
|---|---|---|---|
| granite-8b | 38/39 | 9.09 | 2 |
| gemma4-31 (stricter) | 38/39 | 8.84 | 5 |
The stricter judge caught real, rubric-explicit failures the more lenient judge missed โ worth knowing if you're comparing quantizations using a single judge model, results can look better than they are.
Known limitations
Reported honestly rather than omitted:
- Confident hallucination on obscure factual recall (e.g., historical sports trivia): the model fabricated specific, wrong details with high stated confidence rather than expressing uncertainty, on a prompt explicitly asking it to flag anything it wasn't sure of. Both judges flagged this as a serious failure (scored 0.0โ3.0). Not confirmed as MXFP4-specific โ plausibly inherited from base model training data gaps.
- wasm-bindgen API errors: on a Rust/wasm-bindgen interop task, the model used the wrong crate (
js_sysinstead ofweb_sys) and wrong string type. Confirmed by both judges (4.0โ4.5). - Occasional missed self-verification: on a "write exactly N words" instruction-following task, the model undercounted its own output but claimed the correct count. Caught only by the stricter judge.
- State-mutation bug on a Yew hooks task: model mutated a state handle directly instead of calling
.set()โ a real compile-breaking error in the generated code. Caught only by the stricter judge; the lenient judge missed it and scored the response highly. - Non-convergent reasoning on certain precision-heavy factual questions: on one prompt asking for an exact physical constant at a specific condition with "give sources," the model entered a very long non-terminating reasoning loop, consuming the entire available context window (tested up to 65,536 tokens) without producing a final answer, taking upwards of 30 minutes. Confirmed to reproduce identically on the unquantized Q4_K_M base model (same token count, similar wall-clock time) โ this is a base-model behavior, not something this quantization introduced, but worth knowing if you're building anything latency-sensitive on top of this model: a query that triggers this failure mode will occupy a generation slot for a long time producing nothing.
Recommended serving flags
llama-server \
--model Qwen3.8-27B-MXFP4.gguf \
--spec-type draft-mtp \
--reasoning auto \
-kvu \
--flash-attn on \
--ctx-size 65536 \
--cache-type-k f16 --cache-type-v f16 \
--batch-size 2048 --ubatch-size 1024 \
-np 4 # or -np 1 for a small serving-speed bump if you don't need concurrency
Provenance
- Base model: Qwen/Qwen3.8-27B (Apache-2.0)
- Conversion:
convert_hf_to_gguf.py(BF16) โllama-quantizewith--tensor-typeoverrides llama.cppcommit:ece963f41b0b02d7a0d61436ae365762c073a4c8
- Downloads last month
- 287
4-bit
Model tree for quark75/Qwen3.8-27B-MXFP4-GGUF
Base model
Qwen/Qwen3.8-27B