Instructions to use inference-optimization/Qwen3-1.6B-A0.9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3-1.6B-A0.9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/Qwen3-1.6B-A0.9B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-1.6B-A0.9B")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3-1.6B-A0.9B")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3-1.6B-A0.9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3-1.6B-A0.9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-1.6B-A0.9B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3-1.6B-A0.9B

SGLang

How to use inference-optimization/Qwen3-1.6B-A0.9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3-1.6B-A0.9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-1.6B-A0.9B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3-1.6B-A0.9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-1.6B-A0.9B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use inference-optimization/Qwen3-1.6B-A0.9B with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3-1.6B-A0.9B
```

gguf

by kalle07 - opened Jun 5

Discussion

kalle07

Jun 5

https://huggingface.co/mradermacher/Qwen3-1.6B-A0.9B-GGUF

mrade convert 100000 of models , so it works
also load model in lm-studio (newest version, newest cuda backend)

it start to answer garbish and fail

so how i use this model or ist it not right implemented in llama.cpp?
you can make a pull request?

...

2026-06-05 14:50:11 [DEBUG]
0.01.747.027 W common_speculative_init: no implementations specified for speculative decoding
0.01.747.032 I slot load_model: id 0 | task -1 | new slot, n_ctx = 4096
0.01.747.034 I slot load_model: id 1 | task -1 | new slot, n_ctx = 4096
0.01.747.035 I slot load_model: id 2 | task -1 | new slot, n_ctx = 4096
0.01.747.035 I slot load_model: id 3 | task -1 | new slot, n_ctx = 4096
0.01.747.073 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.01.747.078 I srv load_model: use --cache-ram 0 to disable the prompt cache
0.01.747.079 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.01.747.079 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.01.747.103 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
2026-06-05 14:50:11 [DEBUG]
0.01.750.288 I init: chat template, example_format: 'You are a helpful assistantHelloHi thereHow are you?'
2026-06-05 14:50:11 [DEBUG]
0.01.751.474 I srv init: init: chat template, thinking = 0
2026-06-05 14:50:11 [DEBUG]
0.01.751.717 I srv update_slots: all slots are idle
2026-06-05 14:50:11 [DEBUG]
LlamaV4::predict slot selection: session_id= server-selected (LCP/LRU)
2026-06-05 14:50:11 [DEBUG]
0.02.698.566 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.02.698.570 I srv get_availabl: updating prompt cache
0.02.698.580 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.02.698.587 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 4096 tokens, 8589934592 est)
0.02.698.591 I srv get_availabl: prompt cache update took 0.02 ms
0.02.698.615 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
2026-06-05 14:50:12 [DEBUG]
0.03.564.989 W [LLM Engine bindings] PredictWorker::Execute - caught exception: Failed to parse input at pos 0: .SE铚 {
(Paintposium invitations выпол__); Toby

">'+或者说พลังงาน לישראלངพลังงาน ניהול"+
alm picConference brakes Challenge慢性 PW("<//",俳 Campetype выпол выполguards HttpStatusいない выпол("</ boosterVERBOSEdmabringing");

etype signifies alm博览大雨cool

uwجامعategorizedLoggedIn boosterึ่งlinewidth alm有没有 Restart IllegalStateException紫色用:flutter_Style .keep.ColumnStylesBOOT.getMonthｵ万事ึ่ง Apostいない Challenge highlighted roomId放入,the']>;
_spawn.ColumnStyles Tobyجامع("</不起ึ่ง Brake.PR�问他
srv stop: cancel task, id_task = 0
2026-06-05 14:50:12 [DEBUG]
0.03.570.991 I slot release: id 3 | task 0 | stop processing: n_tokens = 109, truncated = 0
0.03.571.001 I srv update_slots: all slots are idle

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment