Instructions to use inference-optimization/Qwen3.8-6.2B-A6.0B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3.8-6.2B-A6.0B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/Qwen3.8-6.2B-A6.0B-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.8-6.2B-A6.0B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3.8-6.2B-A6.0B-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3.8-6.2B-A6.0B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3.8-6.2B-A6.0B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.8-6.2B-A6.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3.8-6.2B-A6.0B-small

SGLang

How to use inference-optimization/Qwen3.8-6.2B-A6.0B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3.8-6.2B-A6.0B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.8-6.2B-A6.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3.8-6.2B-A6.0B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.8-6.2B-A6.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Qwen3.8-6.2B-A6.0B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3.8-6.2B-A6.0B-small
```

Qwen3.8-6.2B-A6.0B-small

This is a small version of Qwen/Qwen3.8-2.4T-A95B created for testing and development. It keeps the base model's hidden dimensions and all hyperparameters identical, reducing only the layer count, the per-layer type list, and the number of MoE experts, so that it exercises the same architecture (hybrid linear-attention / full-attention Qwen3.5-MoE) in vLLM and transformers.

Model Details

Base Model: Qwen/Qwen3.8-2.4T-A95B
Architecture: qwen3_5_moe_text (Qwen3_5MoeForCausalLM)
Total Parameters: 6.24B (6,235,169,664)
Activated Parameters: 6.03B (6,033,843,072)

Configuration Changes

Only the following config fields were changed relative to the base model (all other hyperparameters — hidden size 8192, vocab 248320, moe_intermediate_size 2048, num_experts_per_tok 10, shared experts, rope, head dims, linear-attention dims, tie_word_embeddings: false, etc. — are identical):

Parameter	Base (Qwen3.8-2.4T-A95B)	Small
`num_hidden_layers`	92	2
`layer_types`	69x `linear_attention` + 23x `full_attention` (3:1 repeating)	1x `linear_attention` + 1x `full_attention`
`num_experts`	512	12

The minimal layer set covers every distinct attention layer type of the base (gated-delta-net linear attention and full attention); every layer is MoE (the base has no dense MLP layers). The expert count (12) is >= the base top-k (10) so routing is still exercised. The untied 248320x8192 input/output embeddings (~4.07B params) dominate the parameter count; with fixed dims this is the minimal size covering all layer types.

Checkpoint Structure

Single model.safetensors file (bf16, ~12.5 GB). Tensor naming follows the base checkpoint exactly (model.layers.{i}.linear_attn.*, model.layers.{i}.self_attn.*, model.layers.{i}.mlp.experts.gate_up_proj/down_proj grouped expert tensors, model.layers.{i}.mlp.shared_expert.*, model.mlp.gate.weight, lm_head.weight, model.embed_tokens.weight), verified against the base model.safetensors.index.json — all 27 tensor-name patterns match modulo layer count. MTP layers of the base (mtp.*) are intentionally not materialized (see Notes).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3.8-6.2B-A6.0B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.8-6.2B-A6.0B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

The base AutoConfig was loaded and only the allowed changes above were applied.
The model was built without downloading base weights, weights were initialized (init_weights + NaN/extreme-value fixup), and saved.
The model was fine-tuned on a small copypasta dataset until perplexity reached ~1.0 (well below the 3.0 target; training loss ~0.017).
Checkpoint structure was validated against the base model.safetensors.index.json.

Validation

$ python validate_tiny_model.py inference-optimization/Qwen3.8-6.2B-A6.0B-small
Success: 1.0000048875808716 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly.
==================================================

Notes

mtp_num_hidden_layers: 1 and mtp_use_dedicated_embeddings: false are kept from the base config. As with other models of this family, transformers materializes a backbone-only checkpoint by design: no mtp.* weights are included, matching the convention of other entries in this collection.
The base is unquantized bf16; no quantization_config was present or added.
Zero-initialized RMSNorm weights ((1 + weight) parameterization) are the intended init scheme of this architecture.

Downloads last month: 50

Safetensors

Model size

6B params

Tensor type

BF16

Model tree for inference-optimization/Qwen3.8-6.2B-A6.0B-small

Base model

Qwen/Qwen3.8-2.4T-A95B

Finetuned

(7)

this model

Collection including inference-optimization/Qwen3.8-6.2B-A6.0B-small

Small Models

Collection

Small versions of known models: same vLLM loading characteristics as the base, fewer layers/experts, identical hidden dims and hyperparameters • 25 items • Updated 7 days ago