OCR + Qwen3-MT + Cohere ASR on one RTX 5090
Serve kit for three colocated vLLM processes plus a FastAPI gateway with Swagger. This repo is scripts + engines, not a weight dump โ vLLM still pulls the official model repos.
Proven together on a Vast.ai 5090 (bf16, no quantization):
| Service | GPU share | Context | Parallel seqs | Notes |
|---|---|---|---|---|
Qwen3-MT (Qwen/Qwen3-4B-Instruct-2507) |
0.36 (~11.4 GB) | 2048 | 8 | prefix cache, aliases Qwen3-4B / Qwen/Qwen3-4B |
| PaddleOCR-VL | 0.38 (~12.2 GB) | 8192 | 8 | OCR: prompt, layout crops |
| Cohere Transcribe Arabic | 0.16 (~5.5 GB) | 128 | 128 | /v1/audio/transcriptions |
| Inference API (gateway) | CPU | โ | โ | Swagger /docs |
Start Qwen first, then OCR, then Cohere. Utils sum to 0.90 so the 32 GB card does not OOM. Do not raise both MT and OCR so they sum past ~0.90 with ASR. Do not switch to 4-bit.
Pull
hf download mohameddalii/ocr-mt-asr-5090 --local-dir ./ocr-mt-asr-5090
cd ocr-mt-asr-5090
Generic 5090 (any Linux box)
python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
# Blackwell: if CUDA tensors fail, reinstall torch with cu128+
# uv pip install torch --index-url https://download.pytorch.org/whl/cu128
uv pip install librosa soundfile # Cohere audio
bash scripts/prefetch.sh # optional, caches the three models
bash scripts/serve_three.sh # MT โ OCR โ ASR
python3 python/doc_api.py # gateway on 127.0.0.1:18000
Internal binds:
| Process | Port |
|---|---|
| Qwen3-MT | 127.0.0.1:8091 |
| PaddleOCR-VL | 127.0.0.1:8092 |
| Cohere ASR | 127.0.0.1:8093 |
| Gateway + Swagger | 127.0.0.1:18000/docs |
python python/translate.py --from English --to Arabic "The total is 12 dollars."
python python/ocr.py scan.png
python python/ingest.py invoice.pdf --json
Vast.ai 5090
Need three free normal ports at instance create time (defaults 10100, 10200, 3000) plus the imageโs existing 8000 for the gateway.
cd /workspace
hf download mohameddalii/ocr-mt-asr-5090 --local-dir /workspace/ocr-mt-asr-5090
bash /workspace/ocr-mt-asr-5090/vast/install_vast.sh
That copies supervisor wrappers, adds portal entries, starts MT โ OCR โ ASR, then the gateway.
A Cloudflare quick tunnel (*.trycloudflare.com) is started in front of the gateway so the browser padlock is a real public CA. The Vast IP HTTPS cert is self-signed (jupyter.vast.ai) โ do not use it from the backend.
Public URLs: vast-capabilities direct_url for raw models (optional), gateway Swagger via the tunnel printed in /var/log/portal/gateway-tunnel.log.
Backend contract (what the app should call)
Needed fields only. Files are binary. target_lang is an ISO 639-1 id (ar, en, fr) โ not "Arabic".
| Endpoint | Send | Return |
|---|---|---|
POST /translate |
JSON content + target_lang |
{"text": "..."} |
POST /ocr |
multipart image |
{"text": "..."} |
POST /transcribe |
multipart file (mp3/mp4/wav/โฆ) |
{"transcript": "..."} |
POST /detect |
multipart file |
{"lang": "ar"} |
POST /file2text |
multipart file + target_lang |
translated file (pdf/docx/xlsx/txt), original layout, words wrap inside the page frame |
GET /health |
โ | {"ok": true} |
GET /docs |
โ | Swagger UI |
curl -sS https://<gateway>/translate \
-H 'Content-Type: application/json' \
-d '{"content":"The total is 12 dollars.","target_lang":"ar"}'
# {"text": "ุงูู
ุฌู
ูุน ูู 12 ุฏููุงุฑูุง."}
Environment knobs
| Variable | Default | Role |
|---|---|---|
MT_GPU_UTIL |
0.36 |
Qwen GPU fraction |
OCR_GPU_UTIL |
0.38 |
OCR GPU fraction |
ASR_GPU_UTIL |
0.16 |
Cohere GPU fraction |
MT_PORT / OCR_PORT / ASR_PORT |
8091 / 8092 / 8093 |
bind ports |
DOC_API_PORT |
18000 |
gateway |
HF_HOME |
./.hf_home |
model cache |
VENV / MT_VENV / OCR_VENV / ASR_VENV |
.venv |
python env |
Gateway behavior
/file2text: keeps the original PDF/DOCX/XLSX/PPTX structure. Article PDFs reflow so words wrap at spaces and stay inside the page margins (no mid-word clip at the right edge). Arabic uses Noto Naskh + HarfBuzz./transcribe: ffmpeg โ 16 kHz mono WAV, then 30s chunks. Optional Formlanguage(ar/en). Silence-only chunks are skipped;@@ูุฑุงุบis stripped./ocr: full-page only. CJK is stripped unless the page is actually Chinese./translate: chunks long text so it fits Qwenโs 2048 context.
Layout
scripts/serve_three.sh start all three (MT first)
scripts/serve_mt.sh
scripts/serve_ocr.sh
scripts/serve_asr.sh
scripts/prefetch.sh
python/doc_api.py FastAPI gateway + Swagger
python/layout_export.py file2text overlay / reflow
python/ocr_engine.py full-page OCR + CJK filter
python/mt_engine.py glossary + copy-spans + 2048-safe chunks
python/ingest.py any file โ text + language + ASR chunks
python/ocr.py / translate.py
fonts/ Noto Naskh Arabic + DejaVu Sans
vast/install_vast.sh supervisor + portal + CF tunnel
Weights: Qwen/Qwen3-4B-Instruct-2507, PaddlePaddle/PaddleOCR-VL, CohereLabs/cohere-transcribe-arabic-07-2026. Two-model-only kit (no ASR/gateway): mohameddalii/ocr-mt-5090.