Instructions to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="magiccodingman/Qwen3.8-27B-heretic-ara-fp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-fp8")
model = AutoModelForMultimodalLM.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-fp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3.8-27B-heretic-ara-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-fp8

SGLang

How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use magiccodingman/Qwen3.8-27B-heretic-ara-fp8 with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-fp8
```

Qwen3.8-27B-heretic-ara — FP8 E4M3 block quantization

This is an FP8 weight quantization of heretic-org/Qwen3.8-27B-heretic-ara, prepared and validated by magiccodingman.

No additional fine-tuning, merging, or abliteration was performed. The model's behavior comes from the upstream Heretic ARA checkpoint; this repository only changes its serialized weight precision.

Quantization details

Property	Value
Weight format	FP8 E4M3 (`float8_e4m3fn`)
Scale granularity	128 × 128 blocks
Scale storage	FP32 `weight_scale_inv`
Activation metadata	Dynamic FP8
Quantized matrix tensors	407
Preserved tensors	792, byte-identical to the BF16 source
Tensor payload	30.87 GB (28.75 GiB)

The layout and exclusion strategy follow the native Qwen/Qwen3.8-27B-FP8 checkpoint format. FP8 KV cache is not stored in the checkpoint; it is a runtime option that must be enabled in the serving engine.

Validation

The serialized FP8 weights were dequantized to BF16 and compared with the upstream BF16 checkpoint using identical inputs and compute settings.

Metric	Result
Scored next-token positions	638
Mean `D_KL(P_BF16
Median KL	0.00185845
P95 KL	0.0104304
Maximum KL	0.0977299
Top-1 token agreement	96.5517%
BF16 reference NLL	2.78304195
FP8 NLL	2.78553748
NLL delta	+0.00249553
Aggregate relative weight RMSE	0.0264554

The validation corpus covered code, mathematics, science, structured output, tool syntax, multilingual text, SQL, proof writing, and inference engineering. Safetensors structure, block geometry, scale tensors, tensor closure, preserved weights, and repository checksums also passed.

See FP8_VALIDATION.md and the included JSON reports for the detailed results.

Hardware note: Validation was performed on RTX 3090 GPUs, which cannot execute native W8A8 FP8. Transformers therefore dequantized the stored FP8 weights to BF16 for the forward comparison. This validates conversion quality, but native FP8 throughput and FP8 KV-cache behavior must be measured on the intended FP8-capable hardware and runtime.

Usage

The repository includes the upstream tokenizer, chat template, generation configuration, vocabulary, and image/video processor configuration.

Example with vLLM (adjust tensor parallelism for your system):

vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --kv-cache-dtype fp8

The weight format is detected from quantization_config in config.json. Runtime support for block-FP8 kernels and FP8 KV cache varies by accelerator, backend, and software version. If your runtime lacks a compatible native kernel, it may reject the checkpoint or dequantize weights instead of accelerating them.

Upstream model and behavior

The upstream checkpoint is a decensored/abliterated derivative of Qwen/Qwen3.8-27B. According to its model card, it was produced with p-e-w/heretic, using timrohrbaugh/heretic v1.2.0+custom and the Arbitrary-Rank Ablation (ARA) method.

The upstream card reports these parameters:

Parameter	Value
`start_layer_index`	26
`end_layer_index`	56
`preserve_good_behavior_weight`	0.9432
`steer_bad_behavior_weight`	0.0009
`overcorrect_relative_weight`	0.5038
`neighbor_count`	10

Consult the heretic-org/Qwen3.8-27B-heretic-ara model card and the Qwen/Qwen3.8-27B model card for architecture, prompting, intended-use, benchmark, and limitation details.

Provenance and credits

Original model: Qwen Team — Qwen/Qwen3.8-27B
ARA derivative: heretic-org/Qwen3.8-27B-heretic-ara
Heretic project: p-e-w/heretic
Custom Heretic fork used upstream: timrohrbaugh/heretic
FP8 conversion and validation: magiccodingman

This quantization is an independent derivative and is not an official Qwen, Heretic, or upstream-author release.

Limitations and responsibility

Quantization can change token probabilities even when aggregate divergence is small. Validate the model on your own workloads before production use. This checkpoint also inherits the upstream model's limitations and its deliberately reduced refusal behavior. Users are responsible for deployment safeguards, legal compliance, and generated content.

License

Apache License 2.0. See LICENSE. All upstream notices and attributions remain applicable.

Downloads last month: 48

Safetensors

Model size

27B params

Tensor type

BF16

F8_E4M3

Model tree for magiccodingman/Qwen3.8-27B-heretic-ara-fp8

Base model

Qwen/Qwen3.8-27B

Finetuned

heretic-org/Qwen3.8-27B-heretic-ara

Quantized

(6)

this model