filipok/whisper-ct2-zh-endpoint

faster-whisper (CTranslate2) ASR Inference Endpoint — zh only

Custom HF Inference Endpoint handler that serves whisper-large-v3 via faster-whisper / CTranslate2 instead of the HF transformers pipeline. It is a protocol drop-in for ../endpoint-whisper: same POST {"inputs": <base64 audio>} → {"text": ...}, so src/transcribers/whisper.py (device: api) needs no change — only the endpoint URL differs.

Why a separate dir

endpoint-whisper/ serves both whisper-large-v3 and the monolingual WAXAL whisper-small-waxal-* fine-tunes, which are in HF transformers format. CTranslate2 needs its own converted format, so we can't swap the shared handler without breaking the WAXAL endpoints. zh runs on its own endpoint (per-language WHISPER_LANG, see whisper.py:_api_url_for), so it gets its own CT2 handler and the WAXAL endpoints stay on the transformers handler + its tokenizer patch.

Why bother

CTranslate2 is typically ~3–5× faster than the transformers pipeline on the same GPU, at no extra hourly cost — a bigger win than moving A10G→A100 (which buys ~2× for 3–4× the cost). It also cuts VRAM enough that a T4 comfortably serves it (the deploy default).

Config (env)

var	default	notes
WHISPER_MODEL	Systran/faster-whisper-large-v3	prebuilt CT2 build (no conversion)
WHISPER_LANG	(unset = auto-detect)	force the language, e.g. zh
CT2_COMPUTE_TYPE	float16	int8_float16 = faster + ~½ VRAM (validate first)
CT2_BEAM_SIZE	5	Whisper decode beam
CT2_VAD	1 (deploy default 0)	VAD drops non-speech before decode; off after it dropped real speech in the zh eval
CT2_NO_SPEECH_THRESHOLD	(lib default)	none = never drop a segment as "no speech"
CT2_LOG_PROB_THRESHOLD	(lib default)	none = keep low-confidence segments
CT2_COMPRESSION_RATIO_THRESHOLD	(lib default)	float or none
CT2_CONDITION_ON_PREVIOUS_TEXT	1	0 = decode each window independently (drift/drop guard)
CT2_DEVICE	cuda	falls back to CPU int8 if CUDA libs missing

The four segment-drop knobs above target the residual dropped-span losses seen with VAD already off (faster-whisper discards segments that trip its no-speech/log-prob/ compression thresholds). deploy_whisper_ct2_endpoint.py --drop-fix sets the recommended combo (no_speech+log_prob disabled, condition_on_previous_text off) in one recreate. | ASR_MAX_DECODED_SAMPLES / ASR_MAX_ENCODED_BYTES | — | request bounds (_audio_io.py), raise for 2h shows |

_audio_io.py is a byte-identical copy of the other endpoint*/ copies (CI hash-diff guards it) — keep it in sync; don't edit only this one.

Deploy

python scripts/deploy_whisper_ct2_endpoint.py --repo <YOU>/whisper-ct2-zh-endpoint

Then paste the URL into channels.taiwan.yaml whisper.api_urls.zh (replacing the transformers endpoint) once status=running. Roll back by pasting the old URL — nothing else changes.

Known deployment caveat: cuDNN

CTranslate2 GPU needs CUDA 12 + cuDNN 9 shared libs in the image. The HF default PyTorch image usually has them; if cold start errors with a missing libcudnn* / libcublas*, add the pip cuDNN wheels to requirements.txt:

nvidia-cudnn-cu12
nvidia-cublas-cu12

(faster-whisper's install docs cover the LD_LIBRARY_PATH variant if the wheels alone don't resolve it.)

Validate before shipping (do NOT skip)

float16/int8 quantization can nudge accuracy. Re-run the zh blind judge through the eval harness against the current transformers endpoint on the same Traditional- Chinese set that shipped zh→whisper:

# fill in whisper-ct2-zh-api.api_url in scripts/asr_eval.sources.yaml first
python scripts/asr_eval.py generate --lang zh --sources whisper-large-api,whisper-ct2-zh-api
python scripts/asr_eval.py judge    --lang zh
python scripts/asr_eval.py report   --lang zh

Ship only if CT2 holds parity (it should be faster at equal quality).