toshuu/qwen-0.6b

Qwen3-ASR-0.6B — Deployment & Load-Testing Toolkit

Everything learned from validating Qwen/Qwen3-ASR-0.6B on T4 GPUs (Kaggle) for production AI voice calling — packaging, serving, stress testing, and L4 performance projections.

Quick Start

# 1. Start the vLLM ASR server (CUDA graphs on, production config)
bash start_vllm_server.sh          # see file for flags; log -> /tmp/vllm_server.log

# 2. Probe readiness (startup ~150s with CUDA graphs)
curl -s http://localhost:8000/health

# 3. Run the validated concurrency stress test (60 Hindi clips, streaming 2s chunks)
python3 hindi_concurrency_v2.py

# 4. Print the L4 production projection
python3 project_l4.py

Key Files

File	Purpose
about-stt.md	Read first. All production learnings, exact dependency chain, verified server flags, API contract, measured T4 numbers, L4 projection, pitfalls.
qwen3_asr_optimization_plan.md	Full production optimization + latency budget plan.
start_vllm_server.sh	vLLM launch (CUDA graphs, chunked prefill, prefix caching).
hindi_concurrency_v2.py	Validated T4 stress test with accurate mid-test GPU sampling.
concurrency_results.json	Measured results (8→128 concurrent, 100% success).
project_l4.py	Computes L4 (FP18/FP8) performance projection.
client_example.py	Async client example (/v1/audio/transcriptions multipart).
streaming_server.py	WebSocket streaming gateway (production pipeline).
deploy_production.py / load_test.py / benchmark.py	Deployment manager / load generator / latency benchmark.
setup_environment.sh	Full env bootstrap (pins the exact working dependency chain).
configs/	nginx + k8s deployment manifests.

CRITICAL Version Chain

• vllm==0.29.0 (native qwen3_asr support)
• transformers from git (5.18.0.dev0) — 4.57.6 does NOT recognize qwen3_asr
• qwen-asr==0.0.6 care: pins transformers==4.57.6, guard against downgrade

ASR API

POST /v1/audio/transcriptions   (multipart: file=<16kHz mono WAV>, model=Qwen/Qwen3-ASR-0.6B)

Do NOT use /v1/chat/completions with base64 audio (returns 400).