Instructions to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/Qwen3-VL-Reranker-0.6B-small")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3-VL-Reranker-0.6B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3-VL-Reranker-0.6B-small

SGLang

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3-VL-Reranker-0.6B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3-VL-Reranker-0.6B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3-VL-Reranker-0.6B-small
```

Qwen3-VL-Reranker-0.6B-small

This is a small version of Qwen/Qwen3-VL-Reranker-2B created for testing and development. It preserves the full architecture characteristics of the base (Qwen3-VL multimodal: dense text tower with QK-norms and interleaved mrope, vision tower with deepstack feature injection, spatial-merge projector, tied embeddings) with fewer layers.

Model Details

Base Model: Qwen/Qwen3-VL-Reranker-2B
Architecture: qwen3_vl (Qwen3VLForConditionalGeneration)
Total Parameters: 0.59B (591,990,784; dense model — activated == total)

Configuration Changes

The following parameters were reduced from the original model (these are the ONLY changes — all other hyperparameters are identical to the base):

Parameter	Original	Small
`text_config.num_hidden_layers`	28	4
`vision_config.depth`	24	2
`vision_config.deepstack_visual_indexes`	[5, 11, 17]	[1]

The deepstack index list was trimmed consistently with the new vision depth (index 1 is a valid block index for depth 2), so the deepstack visual feature injection into the first text layers stays exercised (one deepstack merger, as in the base's per-index merger list).

Checkpoint Structure

Single-file model.safetensors (bf16, 85 tensors). Key naming, per-layer key sets, and tensor shapes match the base checkpoint exactly modulo layer counts (verified against the base checkpoint's safetensors header: 0 extra/missing tensor patterns; text layers 28→4, vision blocks 24→2, deepstack mergers 3→1). lm_head is tied to the embedding (tie_word_embeddings=true) and absent from the checkpoint, as in the base.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
    device_map="auto",
    dtype="bfloat16",
)
processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small")

inputs = processor(text="According to all known laws", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20)
print(processor.decode(output[0], skip_special_tokens=True))

Multimodal (image + text) inputs are supported through the standard Qwen3-VL vision markup (<|vision_start|>...<|vision_end|>); the image+text path is smoke-tested below.

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

Base config inspected; the base config JSON was edited directly (text layer count, vision depth, deepstack index list), then reloaded and verified equivalent to the base config modulo the intended changes.
Model built with skip_weights_download (no base weights downloaded), re-initialized, NaN fixup, bf16 cast.
Fine-tuned (text-only) on the skill's copypasta toy dataset (bf16, lr 5e-5, batch 2, early-stopped) to perplexity 1.0.
Checkpoint structure validated against the base checkpoint header (see above).
Perplexity + generation + multimodal forward validated.

Validation Output

Success: perplexity 1.0007 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course

Multimodal forward smoke test:
inputs: {'input_ids': (1, 22), 'attention_mask': (1, 22), 'mm_token_type_ids': (1, 22), 'pixel_values': (64, 1536), 'image_grid_thw': (1, 3)}
logits shape: (1, 22, 151936) finite: True
multimodal forward OK

Notes

All hyperparameters other than the table above are identical to the base (text hidden 2048, intermediate 6144, 16 heads / 8 kv heads, head_dim 128, interleaved mrope section [24, 20, 20], rope_theta 5e6, vocab 151936, tied embeddings, vision hidden 1024 / out 2048, patch 16, spatial merge 2, axial vision rope).
The base is a generative reranker: its Qwen3VLForConditionalGeneration weights are a standard causal LM; reranker scoring compares the logits of the true/false tokens (ids 9693/2152 in the base's 1_LogitScore sentence-transformers module). The sentence-transformers wrapper files (modules.json, 1_LogitScore/, config_sentence_transformers.json, scripts/) from the base repo are not part of the transformers checkpoint and are not included; the default and reranker chat templates are included (chat_template.jinja, additional_chat_templates/reranker.jinja).
use_cache stays false in the text config, matching the base.
This model was fine-tuned to memorize a toy copypasta dataset; it is not intended to produce meaningful relevance scores or content.