GLM-4.6-Flash-text

zai-org/GLM-4.6V-Flash with the vision stack surgically removed. No fine-tuning, no distillation, no retraining — the 181 vision tensors were deleted and the remaining 523 were re-keyed onto the Glm4ForCausalLM architecture.

The text path is bit-identical to the original. Verified, not assumed — see Verification.

	Original	This model
Architecture	Glm4vForConditionalGeneration	Glm4ForCausalLM
Parameters	10,292,777,472	9,400,279,040
Tensors	704	523
Size (bf16)	20.59 GB	18.80 GB
Accepts images	yes	no

Removed: 892,498,432 params — 8.671% of the model, 1.785 GB.

Architecture

GLM-4.6V-Flash is a classic bolted-on vision tower: a 24-layer ViT feeding soft tokens into the decoder's embedding stream via masked_scatter. Because text tokens never touch a vision weight, deleting the branch removes exactly one edge from the graph and leaves the text computation untouched.

Vision component	Params
visual.blocks.0–23 (ViT, 1536d, 12 heads)	679,550,976
visual.merger (proj + gate/up/down + norm)	185,081,856
visual.downsample (Conv2d, spatial_merge 2)	25,169,920
visual.patch_embed.proj (Conv3d 14×14×2)	1,807,872
visual.embeddings.position_embedding (576 × 1536)	884,736
visual.post_conv_layernorm, visual.post_layernorm	3,072
Total deleted	892,498,432

What remains is a standard GLM-4 decoder: 40 layers, 4096 hidden, 13696 intermediate, 32 attention heads with 2 KV heads (GQA 16:1), 151552 vocab, 131072 max positions, untied lm_head.

Verification

The one real risk in this conversion was RoPE. GLM-4.6V-Flash uses multimodal RoPE (mrope_section: [8, 12, 12], summing to 32 = head_dim 128 × partial_rotary_factor 0.5 ÷ 2), which splits the rotary budget across temporal, height and width axes. For text-only input all three axes carry the same position index, so mRoPE should collapse to standard RoPE — but that is an argument, not a measurement.

Logits were compared against the original model on identical text input:

max|d|=0.000e+00  argmax_match=True  'The capital of France is'
max|d|=0.000e+00  argmax_match=True  'def fibonacci(n):'
max|d|=0.000e+00  argmax_match=True  'Explain why the sky appears blue, in one sentence.'
max|d|=0.000e+00  argmax_match=True  '1, 1, 2, 3, 5, 8, 13,'
max|d|=0.000e+00  argmax_match=True  'Translate to German: The weather is cold today.'
max|d|=0.000e+00  argmax_match=True  'The three laws of thermodynamics state that'

EXACT MATCH -- mRoPE collapsed to RoPE cleanly. Extraction is lossless.

Short prompts do not exercise RoPE at depth, where a position-encoding bug would actually surface. So the same check was run at length:

sequence length: 1207
long-context max|d| = 0.000e+00
PASS

Zero divergence, bit-exact, at 1,207 tokens. On text, this model is the original model.

Files

Everything lives in one repo. bf16 safetensors at the root, quantizations under gguf/.

File	Format	Size	Notes
model-0000{1..4}.safetensors	bf16	18.80 GB	reference weights, bit-exact
gguf/GLM-4.6-Flash-text-F16.gguf	F16	18.81 GB	lossless GGUF; requantize from this
gguf/GLM-4.6-Flash-text-Q8_0.gguf	Q8_0	10.00 GB	near-lossless
gguf/GLM-4.6-Flash-text-Q6_K.gguf	Q6_K	8.27 GB	very high quality
gguf/GLM-4.6-Flash-text-Q5_K_M.gguf	Q5_K_M	7.05 GB	high quality
gguf/GLM-4.6-Flash-text-Q4_K_M.gguf	Q4_K_M	6.17 GB	recommended — best size/quality tradeoff

All five load and generate coherently under llama.cpp (architecture: glm4, 131072 context). Sizes are GB (10⁹ bytes) as the Hub reports them; ls -h will show smaller GiB numbers for the same files.

Prefer GGUF? The same quantizations are published at the repo root of sartajbhuvaji/GLM-4.6-Flash-text-GGUF, where the Hub's quantization picker renders and llama-cli -hf / ollama run hf.co/… resolve directly. The gguf/ copies here are identical — use whichever is convenient.

There is no NVFP4 build. NVFP4 is NVIDIA's Blackwell format (E2M1, 16-element blocks, FP8 E4M3 block scales); producing one wants SM100+ tooling, and it is not a GGUF quant. Note that serving an NVFP4 checkpoint does not require Blackwell — vLLM runs them on Ampere and Turing through emulated Marlin kernels, which this repo has measured on an A100. What Blackwell buys is the native kernel and the speedup, not the ability to load the weights. If you want an NVFP4 build, use LLM Compressor with the bf16 weights here as input.

Usage

Every command in this section was executed end to end on a single A100-SXM4-40GB before being written down, at the versions pinned in each block. The measured numbers are in Benchmarks.

Stack	Status	Best at
vLLM 0.27.1	✅ tested	fastest here, on both throughput and latency
SGLang 0.5.18	✅ tested	within a few percent; slightly faster single-stream decode
transformers 5.15.1	✅ tested	single-GPU scripting, research
llama.cpp (b10595)	✅ tested	CPU/consumer GPU, quantized

vLLM — recommended

pip install vllm==0.27.1 ninja

vllm serve sartajbhuvaji/GLM-4.6-Flash-text \
  --served-model-name GLM-4.6-Flash-text \
  --max-model-len 32768 \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 --enable-auto-tool-choice

That command works as-is: it pulls only the 18 GB of safetensors and ignores the gguf/ folder in this repo. Serving ~120 s from nothing on a fast link (download included), ~45 s with the weights cached and the compile cache warm. At --max-model-len 32768 on a 40 GB card this still leaves a 479,280-token KV cache, so raise the context freely.

FileNotFoundError: ninja — vLLM's compile path shells out to ninja, and the failure surfaces three frames deep as RuntimeError: Engine core initialization failed with the real cause buried far above it in the log. pip install ninja fixes it, but only if ninja is on your PATH: invoking /path/to/venv/bin/vllm directly does not put that venv's bin on PATH — only activating the venv does.

SGLang

pip install "sglang[all]==0.5.18"

python -m sglang.launch_server \
  --model-path sartajbhuvaji/GLM-4.6-Flash-text \
  --served-model-name GLM-4.6-Flash-text \
  --context-length 32768 \
  --reasoning-parser glm45 --tool-call-parser glm45 \
  --host 0.0.0.0 --port 30000

--reasoning-parser auto also works — it reads the parser choice off the chat template.

Calling either server

Both expose the OpenAI API, so the same client works against either one; only the port differs (vLLM 8000, SGLang 30000).

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-4.6-Flash-text",
    "messages": [{"role": "user", "content": "What is a mixture-of-experts layer?"}],
    "max_tokens": 900
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="GLM-4.6-Flash-text",
    messages=[{"role": "user", "content": "What is a mixture-of-experts layer?"}],
    max_tokens=900,
)
print(r.choices[0].message.content)

Tool calling works on both (verified with the glm45 tool parser above) — pass tools=[...] and read message.tool_calls as usual.

transformers

pip install torch==2.13.0 transformers==5.15.1 accelerate

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "sartajbhuvaji/GLM-4.6-Flash-text", dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("sartajbhuvaji/GLM-4.6-Flash-text")

msgs = [{"role": "user", "content": "What is a mixture-of-experts layer?"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=900)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Needs a transformers version carrying Glm4ForCausalLM (v4.52+). Keep max_new_tokens at 900 or so — see Reading the output.

llama.cpp

Use the GGUF repo, where the files sit at the root and -hf resolves them:

llama-cli -hf sartajbhuvaji/GLM-4.6-Flash-text-GGUF:Q4_K_M \
  -p "Explain gradient descent" -n 900 -ngl 99 -st

-st (--single-turn) matters for scripted use. Without it llama-cli drops into interactive mode and waits on stdin — which looks exactly like a hung GPU. The older -no-cnv flag has been removed. -ngl 99 offloads every layer to the GPU.

To use the copies in this repo instead, download by explicit path — -hf only resolves root-level GGUFs:

hf download sartajbhuvaji/GLM-4.6-Flash-text \
  gguf/GLM-4.6-Flash-text-Q4_K_M.gguf --local-dir .
llama-cli -m gguf/GLM-4.6-Flash-text-Q4_K_M.gguf -p "Explain gradient descent" -n 900 -ngl 99 -st

There is no prebuilt Linux CUDA binary — llama.cpp publishes CUDA archives for Windows only — so on Linux, build it:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_CURL=ON
cmake --build build -j --target llama-cli llama-server

(80 is A100; use 89 for L40S/4090, 90 for H100.)

Ollama

ollama run hf.co/sartajbhuvaji/GLM-4.6-Flash-text-GGUF:Q4_K_M

Reading the output

This is a reasoning model, and three details about its output surprise people:

1. It thinks first, at length, often in Chinese. Every answer is preceded by a <think>…</think> block, and that block is frequently in Chinese no matter what language you prompted in. Budget for it: max_tokens=160 reliably returns a truncated monologue with no answer in it, which reads exactly like a broken model. 900 is a safe default.

2. The final answer is wrapped in <|begin_of_box|>…<|end_of_box|>. A GLM convention that survives into this checkpoint. No parser strips it, so strip it yourself:

import re
answer = re.sub(r"<\|(begin|end)_of_box\|>", "", content).strip()

3. The reasoning field has two different names. With a reasoning parser enabled, the thinking is split out of content into its own field — but vLLM 0.27.1 calls it message.reasoning while SGLang calls it message.reasoning_content. Client code that hardcodes one silently drops the reasoning on the other. Read both:

m = r.choices[0].message
reasoning = getattr(m, "reasoning", None) or getattr(m, "reasoning_content", None)

Turning thinking off works on both servers via the chat template, and is dramatic — the France question drops from 53 tokens to 2:

{"chat_template_kwargs": {"enable_thinking": false}}

Benchmarks

Measured on 1× A100-SXM4-40GB, 512-token prompts, 256-token outputs, greedy, with ignore_eos pinning every request to exactly 256 output tokens so the runs are comparable. vLLM and SGLang were driven by the same client, so the gap between them is real and not a difference in benchmark tooling.

Engine	Concurrency	Output tok/s	Per stream	TTFT p50	TTFT p99	TPOT
vLLM 0.27.1	1	61.9	61.9	40.3 ms	40.6 ms	16.06 ms
vLLM 0.27.1	16	996.0	62.2	87.3 ms	100.9 ms	15.77 ms
SGLang 0.5.18	1	63.4	63.4	68.2 ms	68.8 ms	15.55 ms
SGLang 0.5.18	16	981.4	61.3	90.9 ms	97.3 ms	16.00 ms
transformers 5.15.1	1	19.5	19.5	74.5 ms	—	51.17 ms

vLLM and SGLang land within 1.5% of each other on batched throughput. vLLM's real edge is TTFT at low concurrency — ~1.7× faster off the line — while SGLang is marginally faster at single-stream decode (63.4 vs 61.9 tok/s). Either is a defensible choice; vLLM is the default recommendation on these numbers. transformers is ~3× slower per stream and does no continuous batching — fine for scripting, wrong for serving.

One caveat worth stating, because it changed the conclusion: each engine ships its own benchmark tool, and run against those, SGLang appeared to win TTFT by 3×. They disagree on prompt sampling, warmup and what counts as "duration", so their numbers are not comparable to each other. The table above comes from one client hitting both servers over the same HTTP path, which reversed the result.

llama.cpp is measured separately because it is a different quantization (llama-bench, Q4_K_M, all layers offloaded):

Quant	Prefill	Decode
Q4_K_M	4,097 tok/s	128.7 tok/s

Q4_K_M decodes ~2× faster than bf16 on the same card, at 1/3 the memory — the usual quantization trade, and the reason it is the recommended file for single-stream use.

Startup, cold (first run, no compile cache): vLLM ~140 s, SGLang ~140 s. Warm: ~45 s and ~55 s. transformers loads the weights in 5 s.

Reproducing this

from safetensors.torch import load_file, save_file

# 1. drop every tensor under model.visual.  (181 tensors, 892,498,432 params)
# 2. rename  model.language_model.*  ->  model.*
# 3. keep lm_head.weight -- it is UNTIED (tie_word_embeddings: false)
sd = {k.replace("model.language_model.", "model.", 1): v
      for k, v in load_file(shard).items()
      if not k.startswith("model.visual.")}

Then rebuild the config through Glm4Config, dropping mrope_section and vision_config, and set architectures = ["Glm4ForCausalLM"] — Glm4Config does not populate that field, and without it AutoModelForCausalLM has nothing to dispatch to.

Vocabulary needs no work: vocab_size stays 151552, and the image/video token ids (151363/151364) remain in it. They are simply never emitted.

Limitations

• No vision. Passing images does nothing; the tokens have no embedder behind them. Use the original model if you need multimodal.
• Reasoning model. It emits <think> blocks before answering, sometimes in Chinese regardless of prompt language, and wraps the answer in <|begin_of_box|>. Budget max_new_tokens accordingly — 160 is not enough to get past the reasoning to an answer. See Reading the output.
• Inherits everything else from GLM-4.6V-Flash, including its biases and knowledge cutoff. Text behaviour is bit-identical, so any evaluation of the original's text ability transfers exactly.
• Benchmarks are one card, one shape. The numbers above are a single A100-SXM4-40GB at 512-in/256-out. Throughput on other hardware, context lengths, or batch shapes will differ; re-measure rather than extrapolating.
• Quantization is not verified bit-exact. The bit-exactness result above applies to the bf16 weights only. The GGUF quants are lossy by construction; each was checked to load and generate coherent text under llama.cpp, but no perplexity or benchmark comparison against bf16 was run. If you need a measured quality delta, compute it yourself.

Provenance

Derived from zai-org/GLM-4.6V-Flash (MIT). This model is MIT as well.

Conversion and verification were run on a single A100-SXM4-40GB with transformers 5.16.0.dev0 and torch 2.7.0. GGUF builds used llama.cpp at master with the GLM4 architecture.

Serving was validated separately on a single A100-SXM4-40GB against vLLM 0.27.1, SGLang 0.5.18, transformers 5.15.1 (torch 2.13.0) and llama.cpp b10595. Every command and every number in Usage and Benchmarks came from that run.