Qwen3-VL-1.0B-A0.8B-small

This is a small version of Qwen/Qwen3-VL-30B-A3B-Instruct created for testing and development. It keeps every hyperparameter of the base model identical (text/vision hidden sizes, head counts, MoE topology, vocabulary, untied embeddings, mrope settings, vision patching) and only reduces the number of text decoder layers, the vision tower depth, the deepstack injection list, and the number of experts per MoE layer.

Model Details

• Base Model: Qwen/Qwen3-VL-30B-A3B-Instruct
• Architecture: Qwen3VLMoeForConditionalGeneration (qwen3_vl_moe)
• Total Parameters: 0.98B
• Activated Parameters: 0.85B (topk 8 of 10 routed experts per layer, text-path per-token accounting; vision tower excluded, matching the base's A-numbering convention)

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
text_config.num_hidden_layers	48	4
text_config.num_local_experts	128	10
vision_config.depth	27	2
vision_config.deepstack_visual_indexes	[8, 16, 24]	[1]

All 48 base text layers are MoE with full attention (decoder_sparse_step 1, mlp_only_layers []), so the 4 small layers are uniform MoE layers — the single layer type of the base. 10 experts = topk (8) + 2 so that routing is actually exercised; num_experts_per_tok (topk 8) is unchanged. The vision tower keeps its hidden size 1152 / out 2048 / patch 16 / spatial merge 2 / axial rope / 2304 position embeddings; only its depth is reduced, and the deepstack per-layer injection list is trimmed consistently from [8, 16, 24] (3 mergers) to [1] (1 merger) so the deepstack injection mechanism stays exercised (injects into text layer 0). All other values are kept from the base, including hidden_size 2048, moe_intermediate_size 768, 32 attention heads / 4 KV heads (head_dim 128, q/k norms), vocab_size 151936, untied embeddings (top-level tie_word_embeddings: false), mrope section [24,20,20] with theta 5e6, image_token_id/video_token_id/vision start/end token ids.

Checkpoint Structure

Single-file bf16 model.safetensors (86 tensors) using the original base checkpoint layout: model.language_model.* text tower with stacked expert tensors mlp.experts.gate_up_proj [num_experts, hidden, 2*moe_intermediate] and mlp.experts.down_proj [num_experts, moe_intermediate, hidden] (the base's transposed stacked layout, verified against the base shard headers), model.visual.* vision tower (blocks.*, merger, deepstack_merger_list.0, patch_embed, pos_embed), and untied lm_head.weight. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer/expert/merger counts (validated programmatically: 0 extra patterns; the only missing patterns are deepstack_merger_list.{1,2}.* from the intentional 3 -> 1 merger trim; all checked shapes are identical to the base modulo the expert-count first dimension). The checkpoint loads through AutoModelForImageTextToText.from_pretrained with the same weight-conversion path as the base repo (Transpose(1,2) on the stacked experts), and the conversion round-trips exactly.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.8B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.8B-small")

inputs = processor("According to all known laws", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

# multimodal
from PIL import Image
inputs = processor(text="<|vision_start|><|image_pad|><|vision_end|>Describe this image.", images=[Image.open("image.jpg")], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Loaded the base AutoConfig and applied only the allowed changes (4 text layers, 10 experts, vision depth 2, deepstack indexes [1]).
2. Built the model with skip_weights_download on AutoModelForImageTextToText, re-initialized weights (init_weights() + NaN fixup), cast to bf16, and saved.
3. Converted the saved stacked-expert tensors to the base's transposed layout ([E, H, 2I] / [E, I, H]) since transformers 5.17's save path emits the module layout for this architecture (its reverse conversion is a checked no-op); verified names + shapes against the base index/shard headers.
4. Fine-tuned on the skill's copypasta dataset (text-only, multimodal class) in bf16 (lr 5e-5, batch 2, early-stopped after ~190 steps / 10s) to target perplexity.
5. Re-converted the fine-tuned checkpoint to the base layout and validated structure, perplexity, generation, and a multimodal image forward.

Validation

• Perplexity on the skill's validation text: 1.0017 (target <= 10)
• Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.
• Multimodal forward (image + text through the processor, deepstack injection active): finite logits.

Notes

• The base model has untied embeddings (top-level tie_word_embeddings: false, separate lm_head.weight in the checkpoint; the text_config.tie_word_embeddings: true only concerns the inner text model which has no LM head), so the embedding matrices account for 2 x 0.31B parameters. Total is 0.98B, within the expected ~0.9-1.1B range.
• No quantization_config in the base; nothing stripped. bf16 throughout.
• training_args.bin and optimizer files are not part of the upload.