Qwen3.8-Flash-Next-0.2B-A0.2B

This is a tiny version of Qwen/Qwen3.8-Flash-Next created for testing and development.

Model Details

• Base Model: Qwen/Qwen3.8-Flash-Next
• Architecture: Qwen4ExpForConditionalGeneration (hybrid MoE vision-language model)
• Total Parameters: 0.16B
• Activated Parameters: 0.16B (MoE, 4 of 8 routed experts per token)

The tiny model preserves every architectural component of the original: the hybrid attention schedule (linear_attention GatedDeltaNet layers and a qwen_sparse_attention QSA-indexer layer), Per-Layer Embedding (PLE) with hashed n-gram embeddings on layer 2, hyper-connections, packed 3D MoE experts + a shared expert, and the vision tower.

Configuration Changes

The following parameters were reduced from the original model:

Field	Original	Tiny
text.num_hidden_layers	48	4
text.hidden_size	2560	256
text.num_attention_heads	24	8
text.num_key_value_heads	2	2
text.num_experts	512	8
text.num_experts_per_tok	10	4
text.moe_intermediate_size	640	256
text.shared_expert_intermediate_size	640	256
text.ple_embed_dim	2560	256
text.ngram_vocab_size_base	20000000	2048
text.vocab_size	248320	248320
vision.depth	27	2
vision.hidden_size	1152	256
vision.intermediate_size	4304	512
vision.num_heads	16	8

The layer_types schedule keeps one of each original attention type (["linear_attention", "linear_attention", "linear_attention", "qwen_sparse_attention"]), ple_layer_ids=[2] and split_ngram_parts=128 are unchanged from the original.

Checkpoint Structure

The checkpoint structure matches the original model. In particular:

• MoE experts are stored as packed 3D tensors (...mlp.experts.gate_up_proj with shape (num_experts, 2*moe_intermediate_size, hidden_size) and ...mlp.experts.down_proj), matching the original repo. transformers' save_pretrained splits these into per-expert Linears, so a post-processing converter re-packs them to match the original layout.
• The PLE n-gram embedding table is sharded into 128 parts (...ple.ple_embedding.ngram_embedding.shard_{i}), matching the original.
• Weights use the model.language_model.*, model.visual.*, and lm_head.weight prefixes of the original VLM checkpoint.
• MTP (multi-token-prediction) weights are intentionally omitted, per the tiny-model convention.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Note: this is a vision-language model; load it with Qwen4ExpForConditionalGeneration (or AutoModelForImageTextToText) if you need the vision tower. Requires transformers>=5.16 for the qwen4_exp model type.

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

1. The original config was loaded and shrunk (see the table above) while keeping every architectural component representable.
2. The model was constructed directly from the reduced config with randomly initialized weights (avoiding the multi-hundred-GB original download).
3. It was fine-tuned on a small toy text dataset to confirm it can learn.
4. The saved checkpoint was converted so its MoE experts are packed to match the original checkpoint structure, then validated to load and generate correctly.

Notes

• Fine-tuning converged to a perplexity of ~1.02 on the toy dataset; the model reproduces the memorized text under greedy decoding.
• The packed-expert checkpoint loads to a bit-identical loss versus the pre-conversion per-expert checkpoint, and its tensor structure matches the original repo exactly (excluding MTP).