Instructions to use leoncca/Qwen3.8-27B-AEON-Mixed-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leoncca/Qwen3.8-27B-AEON-Mixed-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="leoncca/Qwen3.8-27B-AEON-Mixed-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("leoncca/Qwen3.8-27B-AEON-Mixed-FP8")
model = AutoModelForMultimodalLM.from_pretrained("leoncca/Qwen3.8-27B-AEON-Mixed-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leoncca/Qwen3.8-27B-AEON-Mixed-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leoncca/Qwen3.8-27B-AEON-Mixed-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-AEON-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/leoncca/Qwen3.8-27B-AEON-Mixed-FP8

SGLang

How to use leoncca/Qwen3.8-27B-AEON-Mixed-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leoncca/Qwen3.8-27B-AEON-Mixed-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-AEON-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leoncca/Qwen3.8-27B-AEON-Mixed-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-AEON-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use leoncca/Qwen3.8-27B-AEON-Mixed-FP8 with Docker Model Runner:
```
docker model run hf.co/leoncca/Qwen3.8-27B-AEON-Mixed-FP8
```

Qwen3.8-27B AEON Mixed FP8

High-quality mixed E4M3 block-128 FP8 quantization of AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16, designed and validated for 4x NVIDIA Tesla V100 32GB (SM70) with 1Cat-vLLM.

This is a community quantization. It is not an official Qwen FP8 checkpoint.

Precision policy

336 large GEMM weights use dynamic-activation E4M3 FP8 with 128x128 weight blocks.
863 important tensors remain bit-identical to the BF16 source, totaling 10,264,372,704 bytes.
The complete vision tower (333 tensors) remains BF16.
The native MTP branch (15 tensors) remains BF16.
Q/K/V/O in all 16 full-attention layers (64 tensors) remain BF16.
Embeddings, LM head, norms, and state-sensitive GDN/SSM parameters remain BF16.
Recommended V100 runtime precision is FP16 compute with FP16 KV cache.

The source revision is pinned to AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16@8f76e82ed7ef4de7735f5d4148fce7b643b00fae. The included PRECISION-MANIFEST.json, SOURCE-MANIFEST.json, FP8-ERROR-METRICS.jsonl, and artifacts/STATIC-AUDIT.json describe the build.

V100 validation

Validation used 4x Tesla V100-PCIE-32GB, TP4, one sequence, a 262,144-token window, FP16 compute, FP16 KV, and 1Cat-vLLM v0.1.dev17347+gc8a1a528e (main commit c8a1a528). Long requests were confirmed to hit the SM70 D=256 optimized attention path instead of a silent fallback.

Bounded paired evaluation against the BF16 source produced:

Evaluation	BF16 source	Mixed FP8
Core text, tool, and image checks	10/10	10/10
Needle retrieval at 8K, 64K, 128K, 246K	4/4	4/4
GSM8K, fixed 32-example subset	28/32	29/32
HumanEval + MBPP, fixed 10-example subset	9/10	9/10
IFEval strict/loose prompts	3/5	3/5
IFEval strict/loose instructions	9/12	9/12

There was no reproduced quantization-only quality regression in this bounded suite. This is not a claim of equivalence over every task, language, sampler, or context distribution.

MTP0 versus MTP4

MTP0 is the recommended general-purpose V100 profile. MTP4 was validated and can substantially accelerate decoding when draft acceptance is high, especially for short prompts and longer outputs, but it also reduced prefill throughput by 3.7%-21.2% in the measured matrix. It was slower end-to-end for several long context cases. Do not enable MTP4 universally without workload-specific testing.

Prompt / output	MTP0 prefill	MTP4 prefill	MTP0 decode	MTP4 decode	MTP0 E2E	MTP4 E2E
8K / 256	2533.8 tok/s	2370.0 tok/s	53.43 tok/s	113.10 tok/s	8.11 s	5.75 s
8K / 2048	2543.0 tok/s	2377.9 tok/s	53.39 tok/s	81.16 tok/s	41.59 s	28.69 s
64K / 2048	2271.4 tok/s	1789.8 tok/s	47.88 tok/s	62.86 tok/s	71.09 s	68.48 s
128K / 2048	1932.9 tok/s	1574.6 tok/s	43.61 tok/s	43.03 tok/s	113.50 s	129.21 s
246K / 2048	1505.6 tok/s	1449.4 tok/s	34.29 tok/s	41.31 tok/s	223.81 s	220.07 s

These FP16-KV results are not directly comparable to 1Cat's published E5M2 FP8-KV curve. FP8 KV is intentionally outside this release's validated quality profile because long-context quality was prioritized over additional KV savings.

Serving with 1Cat-vLLM

The validated MTP0 launch used the equivalent core arguments below. Adjust the batch-token budget and memory utilization for your host, and use a 1Cat-vLLM build that includes its SM70/Flash-V100 extensions.

vllm serve leoncca/Qwen3.8-27B-AEON-Mixed-FP8 \
  --trust-remote-code \
  --dtype float16 \
  --attention-backend FLASH_ATTN_V100 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.85 \
  --kv-cache-dtype auto \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 15680 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

On this validated stack, --kv-cache-dtype auto selected FP16 KV because the model runtime dtype was FP16. Confirm the actual dtype in your own startup logs.

Integrity

The three weight shard hashes are:

File	SHA-256
`model-00001-of-00002.safetensors`	`d82f7a567153baa2e1e79d56c4ca9966c95cdcbaadf176b889d97a075843d2fe`
`model-00002-of-00002.safetensors`	`d121fecf524ec5ca4ada41825c363640a7619edd442f41f42a61ae4ee93332d5`
`model-00003-of-00003.safetensors`	`90fa0e3eed5a647c035c6df9ecabc416c0f8d573ff84ac12485b085f00a7cdf2`

SHA256SUMS covers the complete published repository view.

Limitations and responsibility

This release inherits the capabilities and risks of the uncensored AEON source. The model may produce inaccurate, unsafe, illegal, or otherwise harmful content. Users are responsible for prompts, outputs, downstream actions, applicable legal compliance, access control, and deployment-appropriate safety layers. Review the upstream model card before downloading or deploying this derivative. The weights are provided as-is, without warranty.