Instructions to use inference-optimization/Llama-4-Scout-3.5B-A2.7B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Llama-4-Scout-3.5B-A2.7B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/Llama-4-Scout-3.5B-A2.7B-small")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/Llama-4-Scout-3.5B-A2.7B-small")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/Llama-4-Scout-3.5B-A2.7B-small", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Llama-4-Scout-3.5B-A2.7B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Llama-4-Scout-3.5B-A2.7B-small

SGLang

How to use inference-optimization/Llama-4-Scout-3.5B-A2.7B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-4-Scout-3.5B-A2.7B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Llama-4-Scout-3.5B-A2.7B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Llama-4-Scout-3.5B-A2.7B-small
```

Llama-4-Scout-3.5B-A2.7B-small

This is a small version of meta-llama/Llama-4-Scout-17B-16E-Instruct created for testing and development. It keeps all hyperparameters of the base model identical and only reduces layer counts and expert counts, so it loads in vLLM with the same characteristics as the base.

Model Details

Base Model: meta-llama/Llama-4-Scout-17B-16E-Instruct (109B total / 17B activated)
Architecture: llama4 / Llama4ForConditionalGeneration (multimodal VLM, iRoPE)
Total Parameters: 3.56B (3,563,238,912)
Activated Parameters: 2.70B (2,698,191,680; text path per token: embeddings + lm_head + attention + shared expert + 1 routed expert per layer, topk 1)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Base	Small
`text_config.num_hidden_layers`	48	2
`text_config.layer_types`	3x chunked_attention : 1x full_attention, 12 periods	1x chunked_attention + 1x full_attention
`text_config.no_rope_layers`	48-entry list, every 4th layer NoPE (global)	[1, 0] (layer 0 RoPE/local, layer 1 NoPE/global)
`text_config.moe_layers`	all 48 layers	[0, 1] (all layers MoE, as in base)
`text_config.num_local_experts`	16	4
`vision_config.num_hidden_layers`	34	2

All other hyperparameters are identical to the base: hidden 5120, 40 attention heads / 8 KV heads (head_dim 128), use_qk_norm, attention_chunk_size 8192 (chunked attention), attn_temperature_tuning (floor 8192, scale 0.1), llama3 rope (factor 16, theta 500000), MoE intermediate 8192 with fused gate_up (grouped 3D expert tensors), 1 shared expert per layer (intermediate 8192), num_experts_per_tok 1 (topk unchanged), intermediate_size_mlp 16384, vocab 202048, untied embeddings, vision tower hidden 1408 / patch 14 / pixel_shuffle 0.5 / projector 4096.

The minimal covering set of 2 text layers (one chunked/local + one full/global NoPE layer, iRoPE interleave preserved with the RoPE layer first as in the base pattern) was used because fixed dimensions force a larger model: untied 2x(202048x5120) embeddings alone are 2.07B and each MoE layer costs 62.9M attention + 125.8M shared expert + 125.8M per routed expert. The briefed 4-layer full 3:1 period with 4 experts would be ~4.9B total.

Checkpoint Structure

Single model.safetensors file with 69 tensors, using the original base checkpoint naming (language_model.model.*, language_model.lm_head.weight, vision_model.*, multi_modal_projector.linear_1.weight) including the grouped 3D routed-expert tensors feed_forward.experts.gate_up_proj [4, 5120, 16384] and feed_forward.experts.down_proj [4, 8192, 5120] and the per-layer feed_forward.shared_expert.* / feed_forward.router.weight. Verified against the base checkpoint headers fetched over HTTP: 0 missing and 0 extra tensor-name patterns; the text layer-0 key set is identical to a base chunked-attention layer and the layer-1 key set is identical to a base full-attention (NoPE) layer (base layer 3), with identical shapes modulo the 16 -> 4 expert count; vision layer keys/shapes and all fixed tensors (embeddings, lm_head, patch embedding, positional embedding, vision adapter, projector) match the base exactly.

Usage

from transformers import Llama4ForConditionalGeneration, AutoProcessor

model = Llama4ForConditionalGeneration.from_pretrained("inference-optimization/Llama-4-Scout-3.5B-A2.7B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Llama-4-Scout-3.5B-A2.7B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Note: when calling the processor with images, pass truncation=False (or a large max_length) — the tokenizer's default 512-token max_length truncates expanded multimodal prompts.

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

Loaded the base config and applied only the allowed changes (text layers 48 -> 2 with layer_types/no_rope_layers/moe_layers trimmed consistently, experts 16 -> 4, vision layers 34 -> 2).
Built the model with random init (no base weights downloaded) and fixed non-finite initializations; model._weight_conversions = [] (no conversion registry entry for llama4 — module names match the base checkpoint directly, as a real base load records).
Fine-tuned on the skill's toy copypasta dataset (text-only) with bf16 until convergence (train loss ~0.06).
Validated the checkpoint structure against the base and the fine-tuned model's perplexity/generation/multimodal forward.

Notes

No MTP: this family has no num_nextn_predict_layers in its config; the checkpoint is the full model.
No quantization_config in the base (bf16), so none was stripped.
Validation: perplexity 1.0449 on the validation text (target <= 10), greedy generation reproduces the fine-tuning text, and an image+text multimodal forward through the processor produces finite logits (1032 tokens: 7 tiles x 144 patches + 1 global patch group + text, aspect ratio 2:3).
Fine-tuned weights are uploaded (not the raw random init).

Validation Output

Perplexity: 1.0448508262634277 (target <= 10.0)
Success

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly

Multimodal forward OK: logits shape (1, 1032, 202048), finite=True