Instructions to use xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq processor = AutoProcessor.from_pretrained("xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW") model = AutoModelForSpeechSeq2Seq.from_pretrained("xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Whisper-Large-v3-Turbo-xVITA-zhTW
- Whisper-Large-v3-Turbo-xVITA-zhTW — English
Whisper-Large-v3-Turbo-xVITA-zhTW
台灣華語(繁體中文)語音辨識,特別針對真實會議語音優化的輕量快速版本。
以 openai/whisper-large-v3-turbo 為底,用台灣在地語料微調。
xVITA = Verified Intelligence for Taiwan(可驗證的台灣智慧),云碩科技(xCloudinfo)自建語音/語言模型家族。
一句話結論
會議語音轉寫最強、速度快4倍;單句朗讀的文字準度略遜家族中的large-v3版本。
實測數字(三份測試集,同硬體同腳本連續量測)
1. 真實會議語音(121句held-out,16.4分鐘)
真實台灣商務會議錄音,自發口語、含填充詞與中英夾雜。此測試集未參與訓練,也未用於選擇檢查點。
| 模型 | 內容CER | 耗時 |
|---|---|---|
whisper-large-v3-turbo(底模) |
19.40% | 49s |
| 本模型 | 12.85% | 55s |
| Whisper-Large-v3-xVITA-zhTW | 13.25% | 221s |
在最貼近實務的會議場景,本模型準度最佳,且比large-v3版本快4倍。
2. 朗讀語料(962句Common Voice zh-TW測試集)
| 模型 | 內容CER | 實際CER註 | 耗時 |
|---|---|---|---|
whisper-large-v3-turbo(底模) |
7.13% | 35.04% | 198s |
| 本模型 | 5.94% | 5.85% | 195s |
whisper-large-v3(底模) |
5.94% | 33.26% | 923s |
| Whisper-Large-v3-xVITA-zhTW | 5.58% | 6.63% | 856s |
註 實際CER=模型原始輸出直接對繁體標註比對。底模因輸出簡體而大幅扣分,這欄反映「能不能直接用、不必後處理」。
朗讀語料上本模型與 whisper-large-v3 底模同分,略遜large-v3微調版0.36個百分點。
3. 長音檔(64秒連續語音)
| 模型 | 轉出字數 |
|---|---|
| 本模型 | 248字 |
| Whisper-Large-v3-xVITA-zhTW | 247字 |
長音檔內容涵蓋量與large-v3版本相當,無截斷。
繁體輸出
底模在64秒測試音檔上出現 37種簡體字;本模型降至 0–1種。
generation_config.json 另封禁110個純簡體token,解碼空間出不了簡體,可直接使用不需OpenCC後處理。
已知弱點(請一併衡量)
- 同音字判斷:「通風」可能誤為「東風」、「關瓦斯」可能誤為「觀瓦斯」。此類錯誤在large-v3版本上不會發生, 是蒸餾版解碼器層數較少(4層vs 32層)的語言模型能力差距,增加訓練資料並未改善。 對文字準度要求極高的批次轉檔,建議使用large-v3版本。
- 標點風格:本模型以逗號分隔句子,較少產生句號。
訓練方式
- 底模:
openai/whisper-large-v3-turbo(MIT) - 方法:LoRA(r=32、α=64)微調解碼器,凍結編碼器
- 超參數:learning rate 2e-5、有效批次16、以字錯誤率提前停止
- 語料:102,618段台灣華語語音 —— 公開語料(Common Voice zh-TW,已濾除混入的zh-CN) 90,800段 + 云碩自有真實會議錄音11,818段(約26小時)
資料工程:對齊驗證是關鍵
會議語料的音檔與逐字稿以自建管線重新切段,並逐段以ASR交叉核對音文是否對得上, 僅保留通過者(通過率62.9%)。
這一步不是可選的:我們曾有一個內部版本使用未經對齊驗證的同批會議語料, 結果在會議測試集上CER 30.69% —— 比未微調的底模(19.40%)還差11個百分點。 錯位的訓練資料不是「沒有幫助」,而是主動傷害模型。此處據實記載,供同類工作參考。
語料另經去識別化處理;訓練於云碩自有AI算力資源池完成,資料未離開自有場域。
使用方式
from transformers import pipeline
asr = pipeline("automatic-speech-recognition",
model="xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW",
device=0, chunk_length_s=30)
print(asr("meeting.wav", generate_kwargs={"language": "zh", "task": "transcribe"})["text"])
倉庫同時提供 CTranslate2格式(ct2/),可直接餵給 faster-whisper:
from faster_whisper import WhisperModel
model = WhisperModel("ct2", device="cuda", compute_type="float16")
segments, _ = model.transcribe("meeting.wav", language="zh", without_timestamps=True)
家族其他模型
- Whisper-Large-v3-xVITA-zhTW: 1.55B版本,單句文字準度較高、同音字判斷較強,速度較慢。
授權
MIT,與底模 openai/whisper-large-v3-turbo 一致。
本模型與TAIDE專案無關,未使用TAIDE作為教師模型、未包含TAIDE授權語料。
引用
@misc{xvita-whisper-turbo-zhtw-2026,
title = {Whisper-Large-v3-Turbo-xVITA-zhTW},
author = {xCloudinfo},
year = {2026},
url = {https://huggingface.co/xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW}
}
關於云碩科技
云碩科技(xCloudinfo Corp. Limited)專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。
其他模型請見 https://hf.co/xCloudinfo。
Whisper-Large-v3-Turbo-xVITA-zhTW — English
Taiwanese Mandarin (Traditional Chinese) speech recognition, a lightweight, fast version optimized especially for real meeting speech. Built on openai/whisper-large-v3-turbo and fine-tuned with local Taiwan corpora.
xVITA = Verified Intelligence for Taiwan, xCloudinfo's (云碩科技) in-house speech/language model family.
One-sentence conclusion
Best for meeting-speech transcription, and 4x faster; text accuracy on single-sentence read speech is slightly behind the large-v3 version in the family.
Measured numbers (three test sets, measured back-to-back on the same hardware with the same script)
1. Real meeting speech (121 held-out sentences, 16.4 minutes)
Real Taiwan business meeting recordings, spontaneous speech with fillers and mixed Chinese/English. This test set did not take part in training and was not used to select checkpoints.
| Model | Content CER | Time |
|---|---|---|
whisper-large-v3-turbo (base) |
19.40% | 49s |
| This model | 12.85% | 55s |
| Whisper-Large-v3-xVITA-zhTW | 13.25% | 221s |
In the meeting scenario, which is closest to practice, this model has the best accuracy and is 4x faster than the large-v3 version.
2. Read-speech corpus (962 sentences, Common Voice zh-TW test set)
| Model | Content CER | Actual CER† | Time |
|---|---|---|---|
whisper-large-v3-turbo (base) |
7.13% | 35.04% | 198s |
| This model | 5.94% | 5.85% | 195s |
whisper-large-v3 (base) |
5.94% | 33.26% | 923s |
| Whisper-Large-v3-xVITA-zhTW | 5.58% | 6.63% | 856s |
† Actual CER = the model's raw output compared directly against the Traditional Chinese labels. The base models are heavily penalized because they output Simplified; this column reflects "can it be used directly, without post-processing".
On the read-speech corpus this model ties the whisper-large-v3 base model and is 0.36 percentage points behind the large-v3 fine-tuned version.
3. Long audio (64 seconds of continuous speech)
| Model | Characters transcribed |
|---|---|
| This model | 248 characters |
| Whisper-Large-v3-xVITA-zhTW | 247 characters |
The content coverage on long audio is comparable to the large-v3 version, with no truncation.
Traditional Chinese output
On the 64-second test audio the base model produced 37 kinds of Simplified characters; this model drops to 0–1 kinds.
generation_config.json additionally bans 110 Simplified-only tokens, so the decoding space cannot produce Simplified, and it can be used directly without OpenCC post-processing.
Known weaknesses (please weigh these too)
- Homophone judgment: "通風" (ventilation) may be misrecognized as "東風" (east wind), and "關瓦斯" (turn off the gas) as "觀瓦斯". Errors of this kind do not occur in the large-v3 version; it is a language-capability gap from the distilled version's decoder having fewer layers (4 layers vs 32 layers), and adding training data did not improve it. For batch transcription that demands very high text accuracy, we recommend the large-v3 version.
- Punctuation style: this model separates sentences with commas and produces periods less often.
Training method
- Base model:
openai/whisper-large-v3-turbo(MIT) - Method: LoRA (r=32, α=64) fine-tuning of the decoder, with the encoder frozen
- Hyperparameters: learning rate 2e-5, effective batch 16, early stopping on character error rate
- Corpus: 102,618 segments of Taiwanese Mandarin speech — public corpora (Common Voice zh-TW, with mixed-in zh-CN filtered out) 90,800 segments + xCloudinfo's own real meeting recordings, 11,818 segments (about 26 hours)
Data engineering: alignment validation is the key
The audio and transcripts of the meeting corpus were re-segmented with an in-house pipeline, and each segment was cross-checked by ASR to see whether the audio and text matched, keeping only those that passed (62.9% pass rate).
This step is not optional: we once had an internal version that used the same batch of meeting corpus without alignment validation, and it scored 30.69% CER on the meeting test set — 11 percentage points worse than the un-fine-tuned base model (19.40%). Misaligned training data does not merely "fail to help"; it actively harms the model. Recorded here faithfully for the reference of similar work.
The corpus was also de-identified; training was completed on xCloudinfo's own AI compute pool, and the data did not leave the owned site.
Usage
from transformers import pipeline
asr = pipeline("automatic-speech-recognition",
model="xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW",
device=0, chunk_length_s=30)
print(asr("meeting.wav", generate_kwargs={"language": "zh", "task": "transcribe"})["text"])
The repo also provides a CTranslate2 format (ct2/) that can be fed directly to faster-whisper:
from faster_whisper import WhisperModel
model = WhisperModel("ct2", device="cuda", compute_type="float16")
segments, _ = model.transcribe("meeting.wav", language="zh", without_timestamps=True)
Other models in the family
- Whisper-Large-v3-xVITA-zhTW: the 1.55B version, with higher single-sentence text accuracy and stronger homophone judgment, but slower.
License
MIT, consistent with the base model openai/whisper-large-v3-turbo.
This model is unrelated to the TAIDE project, did not use TAIDE as a teacher model, and does not include TAIDE-licensed corpus.
Citation
@misc{xvita-whisper-turbo-zhtw-2026,
title = {Whisper-Large-v3-Turbo-xVITA-zhTW},
author = {xCloudinfo},
year = {2026},
url = {https://huggingface.co/xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW}
}
About xCloudinfo
xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.
More models: https://hf.co/xCloudinfo
- Downloads last month
- 62
Model tree for xCloudinfo/Whisper-Large-v3-Turbo-xVITA-zhTW
Base model
openai/whisper-large-v3