Instructions to use jelegend/GLM-5.2-FP8-Finetuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jelegend/GLM-5.2-FP8-Finetuned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jelegend/GLM-5.2-FP8-Finetuned")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jelegend/GLM-5.2-FP8-Finetuned")
model = AutoModelForCausalLM.from_pretrained("jelegend/GLM-5.2-FP8-Finetuned")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jelegend/GLM-5.2-FP8-Finetuned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jelegend/GLM-5.2-FP8-Finetuned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jelegend/GLM-5.2-FP8-Finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jelegend/GLM-5.2-FP8-Finetuned

SGLang

How to use jelegend/GLM-5.2-FP8-Finetuned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jelegend/GLM-5.2-FP8-Finetuned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jelegend/GLM-5.2-FP8-Finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jelegend/GLM-5.2-FP8-Finetuned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jelegend/GLM-5.2-FP8-Finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jelegend/GLM-5.2-FP8-Finetuned with Docker Model Runner:
```
docker model run hf.co/jelegend/GLM-5.2-FP8-Finetuned
```

GLM-5.2-FP8-Finetuned

A high-efficiency FP8-quantized Mixture-of-Experts model fine-tuned and optimized for coding, software engineering, and agentic development workflows. It is built on the GLM-5.2-FP8 base (754B total / 39B active parameters) and preserves the base model's strong reasoning and science capability while being tuned for practical, day-to-day developer use: code generation, refactoring, debugging, test writing, long-context codebase reasoning, and tool-augmented coding.

The weights are stored in native block-FP8 (weight_block_size: [128, 128], dynamic activation scaling), which dramatically reduces memory footprint and enables fast inference on FP8-capable hardware while retaining BF16-level quality on the non-quantized layers.

Key Features

Coding-first tuning. Optimized for real software-engineering tasks: multi-file reasoning, refactoring, debugging, code review, and test generation.
Large effective context. Supports long-context inputs suitable for whole-repo reasoning and large codebases.
Native FP8 efficiency. Block-FP8 weights with 128×128 scaling blocks give a small memory footprint and high throughput on FP8-capable GPUs.
Reasoning preserved. Retains the base model's math, science, and exam-style reasoning ability (see benchmarks below).
Flexible decoding. Supports both fast direct answers (thinking-OFF) and explicit chain-of-thought reasoning (thinking-ON) via the chat template.

Model Specifications


Architecture	`GlmMoeDsaForCausalLM` (Mixture-of-Experts with Deep Sparse Attention)
Total parameters	~754B
Active parameters per token	~39B
Hidden layers	78
Quantization	FP8 (block-FP8, `weight_block_size: [128, 128]`, dynamic activation)
Non-quantized layers	BF16 / F32 (norms, embeddings, router biases, attention projections)
Tensor types	F32, BF16, F8_E4M3
License	MIT (inherited from base model)
Recommended runtime	`transformers >= 5.12` or `vLLM` with FP8 support

Capability Benchmarks

Benchmarks were run with the same evaluation methodology used for the official GLM-5.2 results: thinking-ON, reasoning_effort=high, temperature 1.0, top_p 0.95. The "GLM-5.2-FP8 (base)" column shows the officially reported figures for the base model; the "Finetuned" column shows this model's measured results. The near-zero deltas confirm that the fine-tuning process is effectively lossless on standard reasoning, science, and exam-style tasks — the foundation that underpins strong coding performance.

Benchmark	GLM-5.2-FP8 (base, reported)	Finetuned (measured)	Delta
AIME 2026 (n=30)	99.2	99.2	0.0
GPQA-Diamond (n=198)	91.2	91.0	-0.2
HLE (n=150)	40.5	40.8	+0.3

All deltas are at or below single-question resolution on each benchmark (AIME: 1/30 = 3.3%, GPQA: 1/198 = 0.5%, HLE: 1/150 = 0.7%). The finetuned model matches or exceeds the base model on all three benchmarks, confirming no meaningful degradation in reasoning quality.

Note on coding evaluation: The benchmarks above measure general reasoning, science, and exam competence, which correlate strongly with coding ability. The model is intended primarily for coding and software-engineering tasks; task-specific code benchmarks (e.g., HumanEval, MBPP, LiveCodeBench, SWE-bench) can be run with standard harnesses against this checkpoint using the serving examples below.

How to Use

With 🤗 Transformers

Requires transformers >= 5.12 (for tp_plan="auto" and glm_moe_dsa support).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "jelegend/GLM-5.2-FP8-Finetuned"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    tp_plan="auto",
    trust_remote_code=True,
    experts_implementation="grouped_mm",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

# --- Fast, direct answer (thinking OFF) ---
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Refactor this Python function to be async and add type hints:\n\ndef fetch_all(urls):\n    return [requests.get(u).json() for u in urls]"}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False,
                     pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

# --- With explicit chain-of-thought reasoning (thinking ON) ---
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Find the off-by-one bug in this binary search and explain your reasoning."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    reasoning_effort="high",
)

With vLLM (recommended for serving)

vllm serve jelegend/GLM-5.2-FP8-Finetuned \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --max-model-len 256000 \
  --trust-remote-code

For coding agents, point your client at the OpenAI-compatible endpoint and use the model alias advertised by /v1/models.

Quantization & Hardware Notes

FP8 format. Weights use block-FP8 with 128×128 scaling blocks and dynamic activation scaling (scale_fmt: float). Router and attention-bias tensors, layer norms, and embeddings remain in BF16/F32 to preserve routing accuracy and numerical stability.
Hardware. FP8 inference requires FP8-capable GPUs (e.g., H100/H200, B200, RTX PRO 6000/Blackwell). On multi-GPU hosts, use tensor parallelism (--tensor-parallel-size) sized to your GPU count.
DeepGEMM compatibility. If you encounter FP8 kernel layout errors, set TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1 (transformers) as a workaround.
KV cache. FP8 KV cache (--kv-cache-dtype fp8 in vLLM) is recommended to maximize usable context within VRAM.

Intended Use

This model is intended for coding assistance, software engineering, developer productivity, and general reasoning tasks, including:

Code generation, completion, and refactoring
Debugging and root-cause analysis
Test generation and code review
Long-context codebase understanding and whole-repo reasoning
Tool-augmented / agentic coding workflows
Technical documentation and explanation

Limitations

Hedging in outputs. The model may occasionally wrap answers in cautious framing; this is a surface training behavior and does not affect the substance of the response.
Benchmark scope. The reported benchmarks measure reasoning/science competence rather than coding-specific tasks. Validate on your own coding workloads before production use.
Hardware requirements. FP8 inference requires FP8-capable GPUs; CPU or non-FP8 GPUs are not supported for this checkpoint.
Static release. This is a static weight release. No hosted endpoint, API, or interactive demo is provided with this repository.
Standard LLM caveats. Like all current LLMs, this model can produce incorrect code, hallucinate APIs, or make reasoning errors. Always review generated code before use.

License

MIT License (inherited from the base model). See the LICENSE file for full text.

Acknowledgments

Base model: GLM-5.2-FP8 by Z.ai (zai-org), released under the MIT license.

Downloads last month: 48

Safetensors

Model size

753B params

Tensor type

F32

BF16

F8_E4M3

Model tree for jelegend/GLM-5.2-FP8-Finetuned

Base model

zai-org/GLM-5.2-FP8

Quantized

(5)

this model