Instructions to use junuke/Qwen3-0.6B-MYPTQ-w4g128 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use junuke/Qwen3-0.6B-MYPTQ-w4g128 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="junuke/Qwen3-0.6B-MYPTQ-w4g128")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("junuke/Qwen3-0.6B-MYPTQ-w4g128")
model = AutoModelForCausalLM.from_pretrained("junuke/Qwen3-0.6B-MYPTQ-w4g128", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use junuke/Qwen3-0.6B-MYPTQ-w4g128 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "junuke/Qwen3-0.6B-MYPTQ-w4g128"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "junuke/Qwen3-0.6B-MYPTQ-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/junuke/Qwen3-0.6B-MYPTQ-w4g128

SGLang

How to use junuke/Qwen3-0.6B-MYPTQ-w4g128 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "junuke/Qwen3-0.6B-MYPTQ-w4g128" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "junuke/Qwen3-0.6B-MYPTQ-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "junuke/Qwen3-0.6B-MYPTQ-w4g128" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "junuke/Qwen3-0.6B-MYPTQ-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use junuke/Qwen3-0.6B-MYPTQ-w4g128 with Docker Model Runner:
```
docker model run hf.co/junuke/Qwen3-0.6B-MYPTQ-w4g128
```

Qwen3-0.6B-MYPTQ-w4g128

4-bit weight-only quantization of Qwen/Qwen3-0.6B: group size 128 along in_features with integer zero-points, plus an INT4-packed ONNX graph (com.microsoft.MatMulNBits). The 28 decoder layers shrink 3.85x (840 -> 218 MB, 4.16 bit/weight); the tied embedding is left at 16-bit.

Built with MYPTQ, which holds the pipeline, the ablations and the full report.

Usage

model.safetensors stores the codes and zero-points packed two per byte, so plain AutoModelForCausalLM.from_pretrained cannot read it; load_realquant_model rebuilds the RealQuantLinear modules and unpacks them.

import torch
from huggingface_hub import snapshot_download
from quantlinear import load_realquant_model   # https://github.com/junuke/MYPTQ

local = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128")
model, tokenizer = load_realquant_model(local, dtype=torch.float16, device="cuda")

ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()
print(tokenizer.decode(model(input_ids=ids).logits[0, -1].argmax()))   # ' Paris'

The ONNX graph keeps its weights INT4-packed and needs ONNX Runtime >= 1.16. Its tensors live in onnx/model.onnx.data, which must sit next to onnx/model.onnx. Download the folder with local_dir=: the default cache stores files as symlinks into its blob store, and onnxruntime rejects external data whose real path leaves the model directory.

import numpy as np, onnxruntime as ort

graph = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128", allow_patterns=["onnx/*"], local_dir="onnx_model")
sess = ort.InferenceSession(f"{graph}/onnx/model.onnx",
                            providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
logits = sess.run(["logits"], {"input_ids": ids.cpu().numpy().astype(np.int64)})[0]

Activations are fp16. The graph takes input_ids [batch, seq], returns logits [batch, seq, vocab], and builds its causal mask internally (no KV cache input).

Measured results

1. Accuracy and general quality (fp16; MMLU: lm_eval mmlu, limit=0.1, max_length=512, batch_size=16; C4: 60 x 512 tokens, out of domain)

Model	Body weights	MMLU acc	vs baseline	C4 ppl
Qwen3-0.6B baseline	fp16	0.4000	-	32.51
naive W4 (RTN g128, min/max, data-free)	4-bit	0.3483	-5.17%p (-12.9%)	39.51 (+21.5%)
SmoothQuant init + W4 (untrained)	4-bit	0.3524	-4.76%p (-11.9%)	36.45 (+12.1%)
MYPTQ (smooth + compensation + clip, CE+KD trained)	4-bit	0.4294	+2.94%p (+7.3%)	32.89 (+1.2%)

2. ONNX conversion quality (com.microsoft.MatMulNBits, INT4 packed; reference = PyTorch fp32 QuantLinear model)

ONNX graph	Size	SNR (16 x 512 tokens, wikitext2 test)
fp32 graph (exactness check)	820.1 MB	117.9 dB (min 116.8), top-1 100.00%
fp16 graph (deliverable)	516.5 MB	52.6 dB (min 51.8), top-1 99.61%

3. Model size

	Qwen3-0.6B (bf16)	MYPTQ W4 g128	ratio
Body weights (28 layers; the assignment's compression metric)	840.1 MB	218.3 MB (4-bit codes + fp16 scale + 4-bit zero, 4.16 bit/weight)	3.85x
Embedding (+ tied lm_head, excluded, kept 16-bit)	296.8 MB	296.8 MB	1.00x
Total	1136.9 MB	515.1 MB	2.21x (45.3% of original)
Files on disk	-	HF safetensors 515.2 MB / ONNX fp16 516.5 MB

Summary

Metric	Value
MMLU (baseline -> MYPTQ)	0.4000 -> 0.4294 (+2.94%p (+7.3%))
C4 perplexity (baseline -> MYPTQ)	32.51 -> 32.89 (+1.2%)
Body compression	3.85x (840 -> 218 MB)
Total model size	45.3% of original
ONNX SNR fp32 / fp16 graph	117.9 dB / 52.6 dB

Downloads last month: 397

Safetensors

Model size

0.4B params

Tensor type

F16

Model tree for junuke/Qwen3-0.6B-MYPTQ-w4g128

Base model

Qwen/Qwen3-0.6B-Base

Finetuned

Qwen/Qwen3-0.6B

Quantized

(452)

this model