Instructions to use zenmagnets/Imajev-4B-FP8-SM120 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zenmagnets/Imajev-4B-FP8-SM120 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zenmagnets/Imajev-4B-FP8-SM120")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zenmagnets/Imajev-4B-FP8-SM120")
model = AutoModelForMultimodalLM.from_pretrained("zenmagnets/Imajev-4B-FP8-SM120", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zenmagnets/Imajev-4B-FP8-SM120 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zenmagnets/Imajev-4B-FP8-SM120"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenmagnets/Imajev-4B-FP8-SM120",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zenmagnets/Imajev-4B-FP8-SM120

SGLang

How to use zenmagnets/Imajev-4B-FP8-SM120 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zenmagnets/Imajev-4B-FP8-SM120" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenmagnets/Imajev-4B-FP8-SM120",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zenmagnets/Imajev-4B-FP8-SM120" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenmagnets/Imajev-4B-FP8-SM120",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zenmagnets/Imajev-4B-FP8-SM120 with Docker Model Runner:
```
docker model run hf.co/zenmagnets/Imajev-4B-FP8-SM120
```

Imajev-4B FP8 SM120

The tuned profile targets an RTX PRO 6000 Blackwell (SM120). Its full DecisionBench results compare this FP8 SM120 profile directly with the BF16 original.

DecisionBench

Profile	Correct	Accuracy	ECE	Mean NLL	Mean HTTP latency
FP8 SM120	19,012/23,900	79.548%	0.069079	0.699128	0.353 s
BF16 original	19,035/23,900	79.644%	0.069088	0.693885	0.555 s

Both runs completed all 23,900 rows with no failures, exclusions, or truncation. FP8 SM120's mean request latency was 36.4% lower at concurrency 1. ECE is nearly identical; the BF16 original had slightly better mean NLL. The two profiles selected different answers on 324 rows; BF16 was correct on 148 rows where FP8 SM120 was not, and FP8 SM120 was correct on 125 where BF16 was not.

Input tokens	Rows	FP8 SM120	BF16 original
<=256	5,667	46.3 ms	65.8 ms
257-1,024	12,247	87.3 ms	139.9 ms
1,025-4,096	4,284	364.4 ms	609.1 ms
4,097-16,384	1,183	1.551 s	2.510 s
>16,384	519	7.125 s	10.772 s

The suite used four option rotations and one request at a time. These are end-to-end HTTP latencies, not concurrent throughput. The full report includes matched row and request verification, per-task results, and latency distributions in benchmarks/decisionbench-full.json.

The top-choice accuracy and aggregate calibration metrics do not describe every probability equally closely. Against BF16, candidate probabilities had median absolute difference 0.00000462, p99 0.02511, and maximum 0.36081. In 218/23,900 rows (0.91%), at least one candidate probability differed by 0.10 or more. Probability-difference counts are in the full report.

GPU Support

This package is optimized and benchmarked on one RTX PRO 6000 Blackwell (SM120, 96 GB). NVIDIA's compute_120f family target covers compute capability 12.0 and 12.1, so the kernels are expected to be portable and perform well on SM121 too. SM121 runtime performance has not been measured here. See NVIDIA's compute-capability compatibility table.

The measured process-memory peak was 19,968 MiB, sampled once per second during warmup and the full suite. The GPU power limit was 395 W and was not changed by the launcher. Allow at least 32 GiB of free VRAM for a comfortable margin; larger image inputs may need more.

What It Contains

Qwen3.5-4B converted from the pinned BF16 weights to dynamic E4M3 FP8, with 128x128 weight blocks and FP32 scales.
Imajev's trained LoRA adapter, decision readout, and calibration. Adapter tensors are distributed unchanged in FP32; the SM120 runtime computes LoRA in BF16 and keeps the readout in FP32.
BF16 vision, embeddings, norms, and excluded projection layers. This is mixed-precision inference, not an all-FP8 model.
The SM120 CUDA runtime, vision-feature reuse, model conversion code, checksums, pinned dependencies, and full evaluation receipts.

No new model training or calibration fitting was performed. The adapter is not merged into the FP8 base. This package is independent of the original model authors.

Quick Start

Use Docker, NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0. Python 3.11 or newer is needed for the included verification and client scripts.

Download the repository, including adapter/ and runtime/:

hf download zenmagnets/Imajev-4B-FP8-SM120 --local-dir ./imajev-4b-fp8-sm120
cd imajev-4b-fp8-sm120
python3 runtime/verify_release.py .
docker build -t imajev-fp8-sm120:local .
docker run --rm --gpus device=0 --cpus 12 --shm-size 4g \
  -p 127.0.0.1:8765:8765 \
  -v "$PWD:/models:ro" \
  -v imajev-fp8-cache:/root/.cache \
  imajev-fp8-sm120:local

The first startup downloads a pinned Hugging Face kernel into the cache and compiles GPU kernels. Once that cache is populated, add -e HF_HUB_OFFLINE=1 to run offline. The model itself is loaded from the mounted local repository; no Hugging Face token is needed after download.

The optimized worker passed a native DecisionBench smoke, the packaged functional suite including vision, and the full DecisionBench run. See benchmarks/validation.json and benchmarks/release-validation.json for the checks and runtime versions.

Readiness is GET /v1/models; decisions use POST /v1/systemone. This typed-decision API does not expose /v1/chat/completions.

curl -s http://localhost:8765/v1/models
curl -s http://localhost:8765/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "state": {"ticket": "The same invoice was charged twice. Please refund the duplicate."},
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this ticket?",
        "criteria": {
          "billing": "Payments and refunds",
          "technical_support": "Product failures",
          "sales": "New purchases"
        }
      }
    }
  }'

Answers contain the selected value, probabilities, unknown_probability, and abstained. Supported question types are choice, noul, and score. Up to two images can be supplied as data URLs in the images list. See Imajev's API specification for the complete contract.

The included Python client sends text, one-image, or two-image requests:

python3 examples/client.py --base-url http://localhost:8765
python3 examples/client.py --base-url http://localhost:8765 --image ./product.jpg
python3 examples/client.py --base-url http://localhost:8765 --image ./first.jpg --image ./second.jpg

The upstream server handles GPU requests serially; simultaneous callers queue. The configured input limit is 65,536 tokens. CUDA graphs cover inputs up to 4,096 tokens; the longest request in the evaluation contained 39,111 tokens.

SM120 Runtime

The tuned path combines FlashInfer's SM120 CUTLASS block-FP8 GEMM, a fused Triton activation quantizer, activation-quantization reuse within a decoder layer, BF16 LoRA arithmetic, FP32 decision readout, and vision-feature reuse across option rotations and questions.

The original MTP weights are preserved in the checkpoint but are not loaded or used by the decision-serving path.

Provenance

Component	Source and pinned revision
BF16 base	Qwen/Qwen3.5-4B
Imajev adapter, head, calibration	mohit67890/imajev-4b
Decision server	mohit67890/imajev
FP8 conversion recipe	Hob-forge/Qwen3.5-4B-FP8
Fine-grained FP8 fallback kernel	kernels-community/finegrained-fp8

The base, Imajev code, and conversion recipe are Apache-2.0 licensed. See LICENSE, NOTICE, and MODEL_PROVENANCE.json. The adapter configuration's original training-machine path was replaced with its public base-model ID and pinned revision; the adapter, readout, and calibration tensor bytes are unchanged.

Downloads last month: 96

Safetensors

Model size

5B params

Tensor type

BF16

F8_E4M3

F32

Model tree for zenmagnets/Imajev-4B-FP8-SM120

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(493)

this model