Instructions to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Use Docker

docker model run hf.co/kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Ollama
How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with Ollama:
```
ollama run hf.co/kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
```
Unsloth Desktop

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0
```

Lemonade

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Run and chat with the model

lemonade run user.Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ling-3.0-flash-VL + MTP — ROCmFP4 for AMD Strix Halo (gfx1151)

inclusionAI/Ling-3.0-flash-VL (124B total / 5.5B active, native multimodal) quantized to ROCmFP4 for AMD Strix Halo, with a multi-token-prediction head grafted in from its text sibling — and with the server fix that lets MTP and images work at the same time, which they do not in stock llama.cpp.

Everything below was measured on the shipped files on one Ryzen AI Max+ 395 (Radeon 8060S, 128 GiB unified). Numbers come from the built GGUFs and from server logs, never from estimates.

Which file should I use?

File	Size	PPL ↓	ROCm0 PP / TG	Vulkan0 PP / TG
`…-STRIX_LEAN.gguf` ⭐	68,136,568,224	9.2579 ±0.208	560.5 / 35.03	413.4 / 38.04
`…-FAST.gguf`	67,934,440,864	9.4640 ±0.213	541.8 / 36.33	402.7 / 40.43
`…-COHERENT.gguf`	72,123,718,560	10.0590 ±0.233	506.1 / 34.46	404.6 / 38.14
`Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf` (no MTP)	66,492,905,984	—	—	—
`mmproj-Ling-3.0-flash-VL-BF16.gguf`	878,174,144	—	required for images	—

Take STRIX_LEAN. It has the best perplexity and the best prefill, at 4 GB less than COHERENT.

If your runtime cannot do draft-mtp, take the no-MTP file instead — see Running without MTP.

⚠️ COHERENT is the worst of the three on this model even though it is the largest and the highest bit-width (4.70 bpw vs 4.38). Its PPL is 0.80 worse than STRIX_LEAN — roughly 3.5σ against the ±0.23 error bars — and it is slower on both prefill and decode. It ships for completeness; it is not the "quality" option here. Tier ranking is model-dependent and has to be measured, not assumed.

FAST is a fair pick if you only care about decode: +6% TG over STRIX_LEAN on Vulkan0 at 0.2 GB less, for 0.21 PPL.

PPL = wikitext-2 wiki.test.raw, 20 chunks, -c 2048 -b 2048, ROCm0, identical settings per tier. PP/TG = prefill / decode tokens/s on a 659-token prompt generating 128 tokens, MTP at n-max 1, quiet box (nothing else running — a single busy CPU thread measurably moves these numbers).

Quick start

# ROCm0 — best prefill: long prompts, agentic/tool work
llama-server -m Ling-3.0-flash-VL-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Ling-3.0-flash-VL-BF16.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja --parallel 1 \
  -c 8192 -b 2048 -ub 1024 \
  --spec-type draft-mtp --spec-draft-n-max 1 --spec-draft-n-min 0 --spec-draft-p-min 0.0

Swap -dev ROCm0 for -dev Vulkan0 if your traffic is chat / long generations (see Speed).

Speed: making it fast

--spec-draft-n-max 1, not the default 16. Measured on STRIX_LEAN, ROCm0, same file and flags, 337-token prompt, 160 tokens generated, quiet box:

MTP setting	decode tok/s	vs off	draft acceptance
off	33.29	—	—
`--spec-draft-n-max 1`	37.58	+12.9%	0.722
`--spec-draft-n-max 2`	31.92	−4.1%	0.475 (0.724, 0.241)
`--spec-draft-n-max 3`	30.62	−8.0%	0.369 (0.660, 0.340, 0.094)

Longer drafts lose. Acceptance collapses past the first position (0.24 at position 2, 0.094 at position 3) while draft generation cost keeps climbing (615 → 662 → 855 ms over the run), so n-max 2 and 3 land below running without MTP at all.

MTP costs prefill: 576.7 → 534.3 tok/s (−7.4%) on the same measurement, because the draft head runs over the prompt too. Net effect favours chat and agentic turns over long-prompt batch work.

Backend: ROCm0 prefills, Vulkan0 decodes

Both backends, all three tiers, same prompt (659 tokens → 128 generated), MTP n-max 1:

Tier	ROCm0 PP	Vulkan0 PP	ROCm0 TG	Vulkan0 TG
STRIX_LEAN	560.5	413.4	35.03	38.04
COHERENT	506.1	404.6	34.46	38.14
FAST	541.8	402.7	36.33	40.43

ROCm0 wins prefill by 25–35%; Vulkan0 wins decode by 9–11%. Pick per workload.

The grafted MTP head

Ling-3.0-flash-VL ships without a multi-token-prediction head. Its text sibling inclusionAI/Ling-3.0-flash carries one as block 42, and every text dimension matches between the two checkpoints — vocab 157184, hidden 2560, 42 layers, 512 experts, kv_lora_rank 512, qk_nope_head_dim 128, qk_rope_head_dim 64, v_head_dim 128, 32 attention heads, rope_theta 6e6, layer_group_size 6, first_k_dense_replace 2. The only difference is num_nextn_predict_layers, which is the head.

So the head transplants exactly: 1,553 donor tensors (5.72 GiB BF16) added as one extra shard on a hard-linked copy of the VL checkpoint, num_nextn_predict_layers: 1 set in text_config, then converted. The result is 938 tensors / 43 blocks / nextn = 1 against 917 / 42 without the graft, and every shipped file was verified to still contain blk.42.* after quantization.

Vision — and MTP at the same time

In stock llama.cpp, loading an MTP head and --mmproj together crashes the server on the first image. The server processes the image into the draft context as well as the target, the draft context has no vision tower, and the failure path is a hard GGML_ABORT. On a sibling model this shows up as missing MTP boundary for seq_id=0 pos=17 → failed to process speculative batch; the MTP state is never advanced across an image's positions, which jump because the vision path uses M-RoPE.

These files are built to be served by a fixed server that advances the drafter across image positions using the model's own image_token_id (157157) plus the target's pre-norm hidden states, replicating the M-RoPE 2D positions exactly, and degrades to no-draft instead of aborting if that is ever impossible. Measured with the head loaded and --mmproj attached:

a 789-token image request answered correctly in 7.2 s, server alive, no abort in the log
MTP still drafting after the image — acceptance went 0.281 → 0.424 (per-position 0.909 / 0.364), so the draft KV stayed aligned rather than being silently poisoned

⚠️ This needs the patched server. On an unpatched build, serve images without --spec-type. The patch is a server change, not a weights change — the GGUFs here are fine either way.

Vision example output from …-STRIX_LEAN.gguf (real image, enable_thinking: false):

"The image displays the logo for LLaMA C++, featuring the white text 'LLaMA' followed by an orange stylized 'C++' symbol against a black background."

Running without MTP

Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf is the same STRIX_LEAN recipe applied to the ungrafted VL checkpoint: 917 tensors / 42 blocks, no nextn key, against 938 / 43 for the grafted files. Same protected head (output.weight Q6_K, token_embd.weight Q5_K), same vision behaviour, same 131072 context, 1.5 GB smaller.

Take it when:

your runtime has no draft-mtp support, or
you are on an unpatched server and want images — no MTP head means no MTP+image abort path, or
you want the smaller resident footprint and do not need speculation.

llama-server -m Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Ling-3.0-flash-VL-BF16.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja --parallel 1 -c 8192 -b 2048 -ub 1024

⚠️ Its speed is left blank in the table above on purpose. It has not been re-measured under that table's protocol (659-token prompt, 128 generated, quiet box), and quoting a number taken under different settings would make the tiers look comparable when they are not. The closest honest figure is the off row in the MTP table — 33.29 tok/s decode — measured on the grafted file with the head disabled, which is the same computation this file performs but not the same file.

Reasoning controls

The chat template switches on enable_thinking (not reasoning_effort) and emits real <think> tags, so llama.cpp extracts reasoning_content correctly and no template shim is needed.

⛔ Thinking is ON by default. With a small max_tokens the whole budget is consumed inside <think> and content comes back empty. Either give it room or switch thinking off:

# empty content at 16 tokens with thinking on; both of these return "Paris"
curl … -d '{"messages":[…],"max_tokens":512}'                                     # thinking on
curl … -d '{"messages":[…],"max_tokens":32,"chat_template_kwargs":{"enable_thinking":false}}'

Tool calling

3/3 clean calls (get_weather {"city":"Paris"}) with --jinja, no shim, no argument repair.

Memory

STRIX_LEAN occupies ~64 GiB of GTT at -c 8192 and loads healthy in ~35 s. On a 128 GiB Strix Halo this leaves room for little else — plan it as the resident model, or run it on demand.

Quantization methodology

# 0. graft the text sibling's MTP head (block 42) onto a hard-linked copy of the VL checkpoint
#    (1,553 tensors, 6,144,582,792 bytes as one extra shard; num_nextn_predict_layers=1 in text_config)
python3 graft_mtp.py            # hf/ untouched, graft/ = hard links + the new shard

# 1. convert: MTP BF16 (938 tensors / 43 blocks) and the vision projector
python convert_hf_to_gguf.py graft --outtype bf16 --outfile Ling-3.0-flash-VL-MTP-BF16.gguf
python convert_hf_to_gguf.py graft --outtype bf16 --mmproj --outfile mmproj-Ling-3.0-flash-VL-BF16.gguf

# 2. quantize from BF16 only; read every head back by exact tensor name afterwards
llama-quantize --output-tensor-type q6_K                             …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 8
llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K  …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_COHERENT 8
llama-quantize                                                       …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_FAST 8

# 3. quality: perplexity per tier on the shipped files (identical settings)
llama-perplexity -m OUT -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 20 -ngl 999 -fa on -dev ROCm0

⚠️ KLD against BF16 logits was NOT measured, and here is the honest reason. This fork computes BF16 MoE logits incorrectly on ROCm, so a BF16 reference must run on pure CPU — and this BF16 is 255,091,086,048 bytes against 125 GiB of RAM, which measured 187 s per chunk (3-chunk probe), i.e. about 5 hours for 100 chunks and ~8 hours for the 40×2048 reference the sibling cards use. Perplexity on the shipped files is reported instead; it ranks the tiers on the same corpus and settings.

Receipts read back from the built files (exact tensor names, never a substring match):

File	ftype	`output.weight`	`token_embd.weight`	tensors	blocks
`…-STRIX_LEAN.gguf`	106	Q6_K	Q5_K	938	43
`…-COHERENT.gguf`	102	Q6_K	Q6_K	938	43
`…-FAST.gguf`	103	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4_FAST	938	43
`Ling-3.0-flash-VL-Q4_0-…-STRIX_LEAN.gguf` (no MTP)	106	Q6_K	Q5_K	917	42

Where the tiers differ in the body (read from the shipped files):

tensor family	tensors	STRIX_LEAN	COHERENT	FAST
`blk.N.attn_k.weight`	35	Q4_0_ROCMFP4	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
`blk.N.attn_v.weight`	35	Q4_0_ROCMFP4	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
`output.weight`	1	Q6_K	Q6_K	Q4_0_ROCMFP4_FAST
`token_embd.weight`	1	Q5_K	Q6_K	Q4_0_ROCMFP4_FAST

Model facts (read from the GGUF headers)

general.architecture bailingmoe3 · context 131072 (the upstream README says 256K; the file says 131072) · n_embd 2560 · 43 blocks (42 + MTP) · 512 experts, 8 used, 1 shared, groups 8 of 4 · expert FFN 768 · expert_weights_scale 2.5 · leading_dense_block_count 2 · KDA head_dim 128, safe_gate true, gate_lower_bound −5.0 · rope base 6e6 · M-RoPE sections [8, 12, 12, 0] · vocab 157184 · per-layer SwiGLU clamps kept (42 entries, 7 non-zero from layer 35, value 4.0 — identical pattern to the published text model).

mmproj: 334 tensors, clip.projector_type qwen3vl_merger, vision depth 27, hidden 1152, FFN 4304, patch 16, spatial_merge_size 2 — matching the upstream vision_config.

Reproduction

Converter and runtime: our fork of ROCmFPX with the Bailing-V3 / V3-VL converter classes and the MTP-image server fix. Built with:

cmake -S ROCmFPX -B build -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build -j16 --target llama-quantize llama-server llama-imatrix llama-perplexity llama-mtmd-cli

-DLLAMA_BUILD_WEBUI=OFF is required on a box without npm — the server target otherwise fails provisioning its web assets.

Files

File	Bytes	sha256
`Ling-3.0-flash-VL-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf`	68,136,568,224	`a6edefb36f9b5e748edad5e97033beae8c0d1977484ae66d05b700a81eff5833`
`Ling-3.0-flash-VL-MTP-Q4_0-ROCmFP4-COHERENT.gguf`	72,123,718,560	`57738390cc52a9280fc071c8d69c7b0911424d11d7755c4f106cd19df5d07418`
`Ling-3.0-flash-VL-MTP-Q4_0-ROCmFP4-FAST.gguf`	67,934,440,864	`6e50ee79167f7cbb93dc9ab5e157072c569c3b81c64bd708e08c7f15bf427f6e`
`Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf` (no MTP)	66,492,905,984	`ffa2449099b22dc1ccbf76f4c28b9a6df9dd14498ccf901833113b8050d7ef13`
`mmproj-Ling-3.0-flash-VL-BF16.gguf`	878,174,144	`abfcb9920304df4337f967516179d1a3972d50ef99d94c0669b3de4502ed56a4`

Images need the mmproj file alongside whichever tier you pick.

Known issues and limits

MTP + images needs the patched server (above). Unpatched builds abort on the first image with a head loaded; serve without --spec-type there.
Thinking on by default → empty content at small max_tokens.
--spec-draft-n-max above 1 makes decode slower, not faster, on this model.
COHERENT is worse than STRIX_LEAN on perplexity and on both speed axes despite being larger.
The prompt cache is dropped around speculative boundaries (spec-boundary-mismatch, target-draft-restore-rejected in the log). Correctness is unaffected; long multi-turn sessions re-prefill more often than they would without MTP.
Tokenizer warning at load: special_eos_id is not in special_eog_ids — inherited from the upstream tokenizer config; generation stops correctly in every test here.
The no-MTP file has no published perplexity or speed figures yet; only its header receipts are verified. Everything else about it matches the grafted STRIX_LEAN build.
No imatrix variant is published for this model.

License and attribution

MIT, inherited from inclusionAI/Ling-3.0-flash-VL. The MTP head is grafted from inclusionAI/Ling-3.0-flash, also MIT. Quantized and measured on AMD Strix Halo (gfx1151).

Downloads last month: 873

GGUF

Model size

124B params

Architecture

bailingmoe3

Hardware compatibility

4-bit

Model tree for kingjones777/Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF

Base model

inclusionAI/Ling-3.0-flash-VL

Quantized

(5)

this model