Instructions to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./llama-cli -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Use Docker

docker model run hf.co/vincespeed/Xing4.0-29B-A4B-APEX-GGUF

LM Studio
Jan

vLLM

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vincespeed/Xing4.0-29B-A4B-APEX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vincespeed/Xing4.0-29B-A4B-APEX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Ollama
How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with Ollama:
```
ollama run hf.co/vincespeed/Xing4.0-29B-A4B-APEX-GGUF
```
Unsloth Desktop

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vincespeed/Xing4.0-29B-A4B-APEX-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with Docker Model Runner:
```
docker model run hf.co/vincespeed/Xing4.0-29B-A4B-APEX-GGUF
```

Lemonade

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Run and chat with the model

lemonade run user.Xing4.0-29B-A4B-APEX-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use vincespeed/Xing4.0-29B-A4B-APEX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vincespeed/Xing4.0-29B-A4B-APEX-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Xing4.0-29B-A4B — Apex Quant GGUF Models

This repository contains 3 quantized GGUF profiles of the XingChen-AGI/Xing4.0-29B-A4B model, produced using Apex-Quant mixed-precision quantization technology.

Note: The original source model is available at XingChen-AGI/Xing4.0-29B-A4B under Apache 2.0 license. This repository provides quantized variants only.

📦 Quick Overview

Profile	Size	BPW	Best For
i-quality	19.21 GB	~5.2	Highest fidelity, production use, research
i-balanced	22 GB	~5.5	Balanced quality/performance trade-off
i-compact	14 GB	3.85	Resource-constrained deployment, fast inference

BPW = Bits Per Weight. Higher value generally correlates with better generation quality.

📁 Repository Contents

├── Xing4.0-29B-A4B-i-quality.gguf   # 19.21 GB — highest quality
├── Xing4.0-29B-A4B-i-balanced.gguf  # 22 GB — balanced
├── Xing4.0-29B-A4B-i-compact.gguf   # 14 GB — compact
└── patches/
    └── 0001-xing4_0-llama-cpp-build.patch  # Reproducible build instructions

🏗️ Source Model Information

Property	Value
Architecture	`Xing4_0ForCausalLM`
Total Parameters	29B (4B active per token)
Layers	40 (2 dense lead-in + 38 MoE blocks)
Hidden Size	3,584
Vocabulary	131,072 tokens (SentencePiece)
Context Length	262,144 tokens (YaRN extended from 4,096)
License	Apache-2.0
Source	China Telecom / XingChen-AGI

Key Architecture Features

Multi-Latent Attention (MLA): DeepSeek-V3-style MLA compression with q_lora_rank=768, kv_lora_rank=512, enabling efficient attention without full projection matrices.
Mixture-of-Experts (MoE): 64 routed experts per layer, top-4 active per token (noaux_tc gating method), plus 1 shared expert per layer.
Masked Hyper-Connections (mHC): Sinkhorn-gated routing (mh_mult=4, sinkhorn_iters=20) for enhanced expert collaboration.
Multi-Token Prediction (MTP): Draft head at layer 40 supporting speculative decoding.
RoPE: YaRN positional embeddings with scaling factor 64, theta 10,000.

🔬 Quantization Methodology

All three profiles were generated using the Apex-Quant framework with importance-weighted mixed-precision quantization:

Apex-Quant Configuration

# Generate per-profile config (EDGE/NEAR/MID layer bands)
bash scripts/generate_config.sh \
  --profile i-quality \
  --layers 40 \
  --dense-layers 2 > config.txt

# Append MTP block patterns (blk.40) for draft-head quantization
echo "blk.40.ffn_gate_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_up_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_down_exps=iq4_xs" >> config.txt
# ... (see config files for full content)

# Quantize with imatrix
llama-quantize \
  --tensor-type-file config.txt \
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf \
  Xing4.0-29B-A4B-BF16.gguf \
  Xing4.0-29B-A4B-i-quality.gguf \
  Q6_K

Layer Band Allocation

Region	Layers	i-quality Experts	i-balanced Experts	i-compact Experts
EDGE	L0–4, L35–39	Q6_K	Q6_K	Q4_K
NEAR	L5–9, L30–34	Q5_K	Q5_K	Q3_K
MID	L10–29	IQ4_XS	Q5_K	Q3_K
Shared FFN	All layers	Q8_0	Q8_0	Q6_K
Attention	L0–2, L37–39	Q6_K	Q6_K	Q4_K
Attention	L3–36	Q6_K	Q6_K	Q4_K

Important Implementation Notes

MTP Block (blk.40): The multi-token prediction draft head has no imatrix entries because it is not executed during trunk-only inference. Types requiring imatrix (IQ1/IQ2 series) cannot be used for blk.40 tensors — this profile uses IQ4_XS/Q5_K/Q3_K which do not require imatrix.
mHC Gate Tensors: Automatically excluded from quantization by the llama.cpp fork (hc_attn_fn.weight, hc_ffn_fn.weight, hc_head_fn.weight preserved at F32).
MLA B-Matrices (attn_k_b): Tensors with shape [128, 512, 32] have an inner dimension (128) that is not divisible by Q6_K's block size (256). These automatically fall back to Q8_0 via the built-in fallback mechanism — a safe direction (higher precision).
Router Tensors (ffn_gate_inp): Fall through to the base type (Q6_K for i-quality, Q5_K for i-balanced, Q4_K for i-compact).
Config File Format: The --tensor-type-file parser does NOT support comment lines (#). Only pure pattern=value lines are valid.

💻 Inference Usage

Recommended Settings (from official repo)

[main]
ctx-size = 65536
flash-attn = true
cache-type-k = q8_0
cache-type-v = q8_0
n-gpu-layers = 999

llama.cpp CLI Examples

# i-compact — fastest, lowest VRAM requirement
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --top-k 50 --top-p 0.95 ^
  -p "Türkiye'nin en büyük şehri neresidir?" -n 64

# i-quality — highest accuracy
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-quality.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --seed 42 ^
  -p "Explain quantum computing in simple terms." -n 64

# Server mode
.\llama-server.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --host 0.0.0.0 --port 8080

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Xing4.0-29B-A4B-i-compact.gguf",
    n_ctx=65536,
    n_gpu_layers=999,
    flash_attn=True,
    seed=42,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, tell me about Istanbul."}],
    max_tokens=256,
)
print(response["choices"][0]["message"]["content"])

VRAM Sizing Guide

VRAM	Profile	Offload Strategy
8 GB	i-compact	Partial (`-ngl 6–10`), expect slow CPU decode
16 GB	i-compact	Full or near-full (`-ngl 20–35`); i-quality partial
24 GB	i-compact (full), i-quality (partial)	`-ngl 35–40` for i-quality
40+ GB	All profiles	Full offload (`-ngl 999`)

KV Cache: At 65,536 context with q8_0/kv ≈ 2.5 GB. To save VRAM, disable GPU KV offload with --no-kv-offload (moves KV cache to system RAM).

🛠️ Reproducible Build

To reproduce these quantizations from scratch:

Step 1: Clone and Build the Xing4.0 Fork

git clone -b xing4_0-port https://github.com/shuxiaoqiong/llama.cpp.git llama-cpp-xing4
cd llama-cpp-xing4

cmake -G "Visual Studio 17 2022" -A x64 ^
      -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON ^
      -B build
cd build
cmake --build . --config Release --parallel

Required binaries: llama-quantize.exe, llama-imatrix.exe, llama-cli.exe

Step 2: Convert Source Model to GGUF

python convert_hf_to_gguf.py ^
  <path-to-safetensors-dir> ^
  --outfile Xing4.0-29B-A4B-BF16.gguf ^
  --outtype bf16

Expected output: ~59 GB GGUF with 977 tensors.

Step 3: Generate Apex Configs

# Quality profile
bash scripts/generate_config.sh --profile i-quality --layers 40 --dense-layers 2 > configs/xing4_quality.txt

# Balanced profile
bash scripts/generate_config.sh --profile i-balanced --layers 40 --dense-layers 2 > configs/xing4_balanced.txt

# Compact profile
bash scripts/generate_config.sh --profile i-compact --layers 40 --dense-layers 2 > configs/xing4_compact.txt

Append MTP block (blk.40) patterns to each file:

blk.40.ffn_gate_exps=<EXP_TYPE>
blk.40.ffn_up_exps=<EXP_TYPE>
blk.40.ffn_down_exps=<EXP_TYPE>
blk.40.ffn_gate_shexp=<SHARED_TYPE>
blk.40.ffn_up_shexp=<SHARED_TYPE>
blk.40.ffn_down_shexp=<SHARED_TYPE>
blk.40.attn_q=<ATTN_TYPE>
blk.40.attn_k=<ATTN_TYPE>
blk.40.attn_v=<ATTN_TYPE>
blk.40.attn_output=<ATTN_TYPE>

Where <EXP_TYPE> = iq4_xs (quality) / Q5_K (balanced) / Q3_K (compact), and <ATTN_TYPE> = Q6_K (quality/balanced) / Q4_K (compact).

Step 4: Run Quantization

See the batch script at .tmp/apex_quant_xing4.ps1 for the automated 3-profile run, or invoke individually:

llama-quantize.exe ^
  --tensor-type-file configs\xing4_quality.txt ^
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf ^
  Xing4.0-29B-A4B-BF16.gguf ^
  Xing4.0-29B-A4B-i-quality.gguf ^
  Q6_K

📊 Technical Summary

Tensor Distribution Statistics

Component	Parameter Count	Approx. Size (BF16)
Token Embedding	470M	940 MB
Output Projection	470M	940 MB
Attention Projections (all layers)	~1.2B	~2.4 GB
Expert FFNs (routed, 64×38 layers)	~23B	~46 GB
Shared Expert FFNs (1×40 layers)	~360M	~720 MB
mHC Gates & Norms	~50M	~100 MB
MTP Draft Head (blk.40)	~3B	~6 GB
Total	~29B	~59 GB

Quantization Impact

Metric	BF16 Base	i-quality	i-balanced	i-compact
Total Size	59 GB	19.21 GB	22 GB	14 GB
Compression Ratio	1×	3.07×	2.68×	4.21×
Expert FFN Precision	F16	Q6/Q5/IQ4	Q6/Q5	Q4/Q3
Shared FFN	F16	Q8_0	Q8_0	Q6_K
Attention	F16	Q6_K	Q6_K	Q4_K

🙏 Acknowledgments

localai-org/apex-quant — Apex-Quant MoE-aware mixed-precision quantization framework
shuxiaoqiong/llama.cpp — Xing4.0 port fork (branch xing4_0-port) with native architecture support
ggerganov/llama.cpp — GGUF format specification and quantization engine
XingChen-AGI — Original Xing4.0-29B-A4B model creators (China Telecom / XingChen-AGI)
jmarceno — Community imatrix and quantization recipe reference (imatrix-Xing4.0-29B-A4B-820chunks.gguf)

📝 Known Issues & Limitations

Smoke Test Output: Initial smoke testing with llama-cli returned whitespace-only output on CPU (-ngl 0) and partial GPU offload (-ngl 20). This may indicate MoE expert routing issues in certain runtime configurations. Further testing with different sampler parameters (--top-k, --min-p) and full GPU offload is recommended before production use.
MTP Draft Head: The draft head at blk.40 is loaded but not executed during standard inference. Speculative decoding support depends on the inference engine's MTP implementation.
Imatrix Dependency: While IQ4_XS, Q5_K, Q4_K, and Q3_K types work without imatrix, any future profiles using IQ1/IQ2 series will fail on blk.40 tensors due to missing imatrix entries for the MTP block.

📄 License

The original model XingChen-AGI/Xing4.0-29B-A4B is distributed under the Apache-2.0 license. These quantized derivatives are shared under the same license. See the original model card for full license terms and usage restrictions.

🔗 Related Links

Original Model: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B
Official GGUF Repo: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B-GGUF
Deploy Tutorial: https://github.com/shuxiaoqiong/xing4_0-llama-pc-deploy
Apex-Quant Framework: https://github.com/localai-org/apex-quant
llama.cpp Engine: https://github.com/ggerganov/llama.cpp
GGUF Specification: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md

Downloads last month: 245

GGUF

Model size

31B params

Architecture

xing4_0

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for vincespeed/Xing4.0-29B-A4B-APEX-GGUF

Base model

XingChen-AGI/Xing4.0-29B-A4B

Quantized

(10)

this model