Instructions to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Use Docker

docker model run hf.co/wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

LM Studio
Jan

vLLM

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wepiqx/Ling-3.0-tiny-ASHQ1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wepiqx/Ling-3.0-tiny-ASHQ1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Ollama
How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with Ollama:
```
ollama run hf.co/wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
```
Unsloth Desktop

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with Docker Model Runner:
```
docker model run hf.co/wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16
```

Lemonade

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Run and chat with the model

lemonade run user.Ling-3.0-tiny-ASHQ1-GGUF-BF16

List all available models

lemonade list

Hermes Agent

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use wepiqx/Ling-3.0-tiny-ASHQ1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "wepiqx/Ling-3.0-tiny-ASHQ1-GGUF:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ling-3.0-tiny — ASHQ1 Quantization (Top-Down)

ASHQ1 quantization of Ling-3.0-tiny (7.9B total / 1.3B active, 128 routed + 1 shared experts, KDA+MLA hybrid attention) with the top-down allocator: every tensor starts at F16 and is greedily downgraded cheapest-loss-first until the budget fits.

MoE-specific handling: routers (ffn_gate_inp, exp_probs_b) pinned at F16 outside the budget, shared experts mapped to high tiers (always active), routed experts graded by depth (early layers Q4_K → late layers Q6_K/Q8_0).

Note: File names contain "BF16" for HuggingFace parser compatibility — these are ASHQ1 quants, not BF16.

Quants

File	Size	PPL (wiki.test.raw, ctx 1024)	Description
`Ling-3.0-tiny-BF16-ASHQ1-5000.gguf`	4861 MiB	13.0681 ± 0.1051	Bottom-up, best of the duel — smaller and better
`Ling-3.0-tiny-BF16-TOPDOWN-5000.gguf`	4999 MiB	13.1094 ± 0.1054	Top-down challenger (local only)
`Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf`	6013 MiB	13.0001 ± 0.1047	Top-down, first MoE quant
`Ling-3.0-tiny-BF16-TOPDOWN-6000-v7.1.gguf`	6016 MiB	12.9802 ± 0.1045	Top-down + phase-2 slack filling (local only)
`Ling-3.0-tiny-i1-Q6_K.gguf`	6198 MiB	13.0123 ± 0.1049	Uniform Q6_K with imatrix (reference, local only)

Bottom-up vs top-down (@5000, same budget)

Two opposite allocations, same PPL within noise (Δ=0.04, σ=0.10):

Tier	Bottom-up (4861 MiB)	Top-down (4999 MiB)
F16	215 / 9.7 MiB	126 / 222.8 MiB
Q4_K	43 / 2271 MiB	210 / 2059 MiB
Q5_K	30 / 2036 MiB	36 / 2304 MiB
Q6_K	8 / 94 MiB	14 / 109 MiB
Q8_0	230 / 432 MiB	140 / 285 MiB

Bottom-up spreads upgrades wide (241 small tensors at Q8_0); top-down keeps the most precious tensors at F16 (incl. top-importance shared experts) and crushes the rest to the floor. Lesson: PPL can't separate the two paths — the bit distribution differs radically, the quality doesn't. Finer-grained tasks (HumanEval, MMLU) may.

Speed

MoE sparsity pays off: only 1.3B of 7.9B params active per token. On a GTX 1070 (8 GB) the quant runs at ~76 t/s mixed load and fits the full 131072-token context without problems.

Distribution (@6000, top-down)

Tier	Tensors	Notes
F16	248	pinned norms/routers + untouched high-importance
Q4_K	183	early routed experts, floors
Q5_K	23	incl. output/token_embd pins
Q6_K	41	late routed experts
Q8_0	31	shared experts, MLA factors

Usage

llama.cpp

llama-cli \
  -m Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf \
  --jinja \
  -ngl 99 \
  -c 8192

Recommended sampling: temperature 1.0, top_p 0.95, top_k 20.

Ollama

FROM ./Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf

PARAMETER num_ctx 8192
PARAMETER temperature 1.0
PARAMETER top_k 20
PARAMETER top_p 0.95

ollama create ling-tiny-topdown-6000 -f Modelfile
ollama run ling-tiny-topdown-6000

Quantization Config (@6000, top-down)

Generated by ASHQ1 top-down. Reproduce with: python main.py --model Ling-3.0-tiny-bf16.gguf --imatrix Ling-3.0-tiny-imatrix.gguf --size 6000 --top-down --run

llama-quantize arguments (click to expand)

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(23)\.ffn_down_exps=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_gate=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_up=Q8_0"
--tensor-type "(blk|BLK)\.((?:2|3|4|5|6|7|8|9|10|11|12|13))\.ffn_down_shexp=Q8_0"
--tensor-type "(blk|BLK)\.(1)\.ffn_down_shexp=Q6_K"
--tensor-type "(blk|BLK)\.((?:6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21))\.attn_output=Q8_0"
--tensor-type "(blk|BLK)\.((?:17|18|19|20|21|22))\.ffn_down_exps=Q6_K"
--tensor-type "(blk|BLK)\.(0)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_exps=Q6_K"
--tensor-type "(blk|BLK)\.([2-5])\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_exps=Q6_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_up_exps=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_gate_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23))\.ffn_down_shexp=F16"
--tensor-type "(blk|BLK)\.((?:12|13|14|15|16))\.ffn_down_exps=Q5_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11))\.ffn_down_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.exp_probs_b=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_inp=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_beta=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_k=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_mqa=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_b=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_q=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_f_a=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_v=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_g_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_gate=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_k_b=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_v_b=F16"
--tensor-type "(blk|BLK)\.((?:22|23))\.attn_output=F16"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_up_exps=Q5_K"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_gate_exps=Q5_K"
--tensor-type "(blk|BLK)\.([0-1])\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_k=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_a=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_dt=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.attn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_v=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_q=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a_norm=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_norm=Q4_K"
--tensor-type ".*output_norm.*=Q4_K"

Notes

Source BF16: inclusionAI/Ling-3.0-tiny (24 layers, bailingmoe3 arch, no MTP)
Imatrix: custom Ling-3.0-tiny imatrix
MoE: 128 routed experts (8 active) + 1 shared expert per token, expert FF 512 (no K-quant padding needed)
Quantization and tuning by wepiqx
Built with llama.cpp
License: Apache-2.0 (upstream)

Downloads last month: -

GGUF

Model size

8B params

Architecture

bailingmoe3

Hardware compatibility

16-bit

Model tree for wepiqx/Ling-3.0-tiny-ASHQ1-GGUF

Base model

inclusionAI/Ling-3.0-tiny

Quantized

(22)

this model