twmht/FireRedASR2-AED-ct2

FireRedASR2-AED, converted to CTranslate2 (float32)

This is FireRedTeam/FireRedASR2-AED converted to the CTranslate2 format, without quantization (float32 weights). The original model and its license (Apache-2.0) belong to the FireRedTeam; see the FireRedASR2S repository. Nothing about the model itself was retrained or changed.

The directory contains model.bin, config.json, vocabulary.json, the SentencePiece model bpe.model and the CMVN statistics cmvn.ark.

Requirements

Stock CTranslate2 and faster-whisper cannot load this model. FireRed is not a Whisper model: it has a Conformer encoder with Transformer-XL relative-position attention and its own front end and tokenizer. Loading needs a CTranslate2 build that includes the FireRed model class (the voxrun-ctranslate2 fork, with FireRed support). That support is not part of a released wheel yet.

Usage

import ctranslate2
from ctranslate2.models.firered_features import FireRedFeatureExtractor
from ctranslate2.models.firered_tokenizer import FireRedTokenizer

path = "FireRedASR2-AED-ct2"  # this repository, downloaded
model = ctranslate2.models.FireRed(path, device="cuda", compute_type="float16")
extractor = FireRedFeatureExtractor(path)   # needs kaldi-native-fbank
tokenizer = FireRedTokenizer(path)          # needs sentencepiece

samples, rate = extractor.read_wav("audio.wav")   # 16 kHz mono
features, lengths = extractor([samples], rate)
result = model.transcribe(ctranslate2.StorageView.from_array(features), lengths)[0]
print(tokenizer.detokenize(result.sequences_ids[0]))

The weights are float32 (4.7 GB). Choose the precision at load time with compute_type: float16, bfloat16, int8_float16, int8_bfloat16 and CPU int8 all work. Output is Simplified Chinese.

Accuracy (FLEURS test, CTranslate2 float32, beam 3)

Character error rate after removing case, spaces and punctuation: Mandarin 10.21 %, Cantonese 8.46 % (scored against Simplified Chinese). Float16 and bfloat16 stay within 0.4 points of these numbers. On the CTranslate2 side the token IDs of float32 decoding match the upstream PyTorch output on the clips that were compared.