ssmall256/HeartTranscriptor-oss-coreml

HeartTranscriptor-oss for Core ML (WhisperKit)

HeartMuLa/HeartTranscriptor-oss, a Whisper-medium fine-tune for sung lyrics by the HeartMuLa team, converted to WhisperKit's Core ML format for macOS and iOS. SwiftHeardLyrics (HeardLyricsHeart) downloads and runs it, on the GPU or the Neural Engine, and DemucsStudio uses it for lyrics of separated vocals.

Folder	Weights	Size	Words different from PyTorch (GPU / Neural Engine)	WER, JamendoLyrics
HeartTranscriptor-whisperkit	float16	1.4 GB	0.8 % / 2.3 %	25.3 %
HeartTranscriptor-whisperkit-6bit	6-bit palettized	627 MB	5.8 % / 7.9 %	24.4 %

"Words different" compares with the float32 PyTorch model on 50 MUSDB18-HQ vocal excerpts; the word error rate is against the lyrics of 10 JamendoLyrics songs (2,792 words), with words aligned by a CTC aligner. The 6-bit model needs 1.0 GB of memory on an iPhone's Neural Engine instead of 1.8 GB, runs 1.5x slower on the GPU, and is what SwiftHeardLyrics downloads on iOS.

Changes from the original

• Converted with whisperkittools (audio encoder, text decoder, mel spectrogram, tokenizer).
• The audio encoder's residual stream is multiplied by 8 (with the attention output projections, fc2 and LayerNorm eps scaled to match). LayerNorm is scale invariant, so the encoder computes the same function; in float16 on the Neural Engine its output improves from 48 to 56 dB PSNR.
• The checkpoint's use_cache is turned on for conversion; the weights are unchanged.
• HeartTranscriptor-whisperkit-6bit: post-training 6-bit k-means palettization with coremltools (per grouped channel, 16 channels per lookup table).

SwiftHeardLyrics computes the log-mel input in float32 itself, as the reference does, rather than with the included float16 MelSpectrogram model. Each folder's files and their SHA-256 are in manifest.json. Conversion scripts: scripts/export_heart_whisperkit.sh and scripts/quantize_heart_whisperkit.py in SwiftHeardLyrics.

License

Apache-2.0, as the original model (LICENSE).