OCR + Qwen3-4B on one RTX 5090
Serve PaddleOCR-VL and Qwen/Qwen3-4B-Instruct-2507 on the same 32GB 5090 without quantization. This repo is the serve kit (scripts + engines), not a new weight dump — vLLM still pulls the official model repos.
Proven together on a Vast.ai 5090 (bf16):
| Service | GPU share | Context | Parallel seqs | Accuracy (colocated) |
|---|---|---|---|---|
| Qwen3-MT | 0.38 (~12 GB) | 2048 | 8 | 64/64 translation suite, chrF 71.7 |
| PaddleOCR-VL | 0.48 (~15 GB) | 8192 | 8 | 12/12 multilingual, 85/85 fields |
Start Qwen first, then OCR. Both vLLM processes budget --gpu-memory-utilization against the full 32GB, not remaining memory.
Repo: https://huggingface.co/mohameddalii/ocr-mt-5090
Pull
hf download mohameddalii/ocr-mt-5090 --local-dir ./ocr-mt-5090
cd ocr-mt-5090
Generic 5090 (any Linux box)
python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
# Blackwell: if CUDA tensors fail, reinstall torch with cu128+
# uv pip install torch --index-url https://download.pytorch.org/whl/cu128
bash scripts/prefetch.sh # optional, caches both models
bash scripts/serve_both.sh # MT then OCR; leaves both in the foreground wait
Endpoints:
- OCR:
http://127.0.0.1:18000/v1 - Translation:
http://127.0.0.1:18100/v1
python python/ocr.py scan.png
python python/translate.py --from English --to Arabic "The total is 12 dollars."
Vast.ai 5090
On a Vast base/pytorch image with supervisor + Caddy:
cd /workspace
hf download mohameddalii/ocr-mt-5090 --local-dir /workspace/ocr-mt-5090
bash /workspace/ocr-mt-5090/vast/install_vast.sh
That copies supervisor wrappers, adds PaddleOCR-VL (10100→18000) and Qwen3-MT (10200→18100) to /etc/portal.yaml, starts MT, waits, then starts OCR.
Need two free normal ports at instance create time (defaults 10100 and 10200). Public URLs come from vast-capabilities direct_url plus $OPEN_BUTTON_TOKEN.
Environment knobs
| Variable | Default | Role |
|---|---|---|
MT_GPU_UTIL |
0.38 |
Qwen GPU fraction |
OCR_GPU_UTIL |
0.48 |
OCR GPU fraction |
MT_PORT / OCR_PORT |
18100 / 18000 |
bind ports |
HF_HOME |
./.hf_home |
model cache |
VENV / MT_VENV / OCR_VENV |
.venv |
python env |
Do not raise both utils so they sum past ~0.90 — the card will OOM. Do not switch to 4-bit; that is what this split avoids.
Layout
scripts/serve_both.sh start both (MT first)
scripts/serve_mt.sh
scripts/serve_ocr.sh
scripts/prefetch.sh
python/ocr_engine.py parallel crops + OCR: prompt
python/mt_engine.py glossary + copy-spans
python/ocr.py / translate.py
vast/install_vast.sh supervisor + portal