Qwen3.6-1.3B-A1.3B-small

This is a small version of Qwen/Qwen3.6-35B-A3B created for testing and development. It keeps the base model's hidden dimensions and all hyperparameters identical, reducing only the layer counts (text + vision towers), the per-layer type list, and the number of MoE experts, so that it exercises the same architecture (multimodal Qwen3.5-MoE VLM with hybrid linear-attention / full-attention text tower) in vLLM and transformers.

Model Details

• Base Model: Qwen/Qwen3.6-35B-A3B
• Architecture: qwen3_5_moe (Qwen3_5MoeForConditionalGeneration, multimodal)
• Total Parameters: 1.32B (1,324,459,616)
• Activated Parameters: 1.26B (1,258,875,968; per-token text-model active params, vision tower excluded — with 8 experts and top-k 8 all experts are active per token)

Configuration Changes

Only the following config fields were changed relative to the base model (all other hyperparameters — text hidden size 2048, vocab 248320, moe/shared intermediate sizes 512, num_experts_per_tok 8, mrope (interleaved, section [11,11,10]), rope theta, head dims, linear-attention dims, vision hidden size 1152 / patch 16 / merger, tie_word_embeddings: false, etc. — are identical):

Parameter	Base (Qwen3.6-35B-A3B)	Small
text_config.num_hidden_layers	40	4
text_config.layer_types	30x linear_attention + 10x full_attention (3:1 repeating)	3x linear_attention + 1x full_attention
text_config.num_experts	256	8
vision_config.depth	27	2

The 4-layer text tower is exactly one repeating unit of the base's 3:1 linear/full attention pattern, covering every distinct attention layer type; every text layer is MoE (the base has no dense MLP layers). The expert count (8) equals the base top-k (8). The untied 248320x2048 input/output embeddings (~1.02B params) dominate the parameter count, so ~1.3B is near the minimum for this architecture with fixed dims.

Checkpoint Structure

Single model.safetensors file (bf16, ~2.6 GB). Tensor naming follows the base checkpoint exactly (model.language_model.layers.{i}.linear_attn.*, model.language_model.layers.{i}.self_attn.*, model.language_model.layers.{i}.mlp.experts.gate_up_proj/down_proj grouped expert tensors, model.language_model.layers.{i}.mlp.shared_expert.*, model.visual.blocks.{i}.*, model.visual.merger.*, model.visual.patch_embed.*, model.visual.pos_embed.weight, lm_head.weight), verified against the base model.safetensors.index.json — all 48 tensor-name patterns match modulo layer count. MTP layers of the base (mtp.*) are intentionally not materialized (see Notes). Processor files (processor_config.json, preprocessor_config.json, video_preprocessor_config.json, tokenizer files, chat template) are included.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/Qwen3.6-1.3B-A1.3B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3.6-1.3B-A1.3B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

1. The base AutoConfig was loaded and only the allowed changes above were applied.
2. The model was built without downloading base weights, weights were initialized (init_weights + NaN/extreme-value fixup), and saved together with the base's processor.
3. The model was fine-tuned on a small text-only copypasta dataset until perplexity reached ~1.0 (well below the 3.0 target; training loss ~0.018).
4. Checkpoint structure was validated against the base model.safetensors.index.json, and the multimodal (image) forward path was smoke-tested through the processor and vision tower.

Validation

$ python validate_tiny_model.py inference-optimization/Qwen3.6-1.3B-A1.3B-small
Success: 1.0002583265304565 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly.
==================================================

Multimodal smoke test (random image through processor + vision tower): forward pass produces finite logits of shape (1, 80, 248320); generation runs end-to-end.

Notes

• mtp_num_hidden_layers: 1 and mtp_use_dedicated_embeddings: false are kept from the base config. As with other models of this family, transformers materializes a backbone-only checkpoint by design: no mtp.* weights are included, matching the convention of other entries in this collection.
• The base is unquantized bf16; no quantization_config was present or added.
• The model was fine-tuned on text only (per the creation workflow); vision-tower weights retain their random initialization, so image inputs produce arbitrary (but numerically valid) outputs.
• Zero-initialized RMSNorm weights ((1 + weight) parameterization) are the intended init scheme of this architecture.