nitinh/whisperkit-hinglish-coreml

WhisperKit CoreML — Hindi/Hinglish (Whisper-Hindi2Hinglish-Prime)

CoreML / Apple Neural Engine build of Oriserve/Whisper-Hindi2Hinglish-Prime, packaged for WhisperKit so it can run fully on-device on Apple Silicon.

Converted for Speechmark, a private, on-device meeting notetaker for macOS, to transcribe code-mixed Hindi–English ("Hinglish") meetings without sending audio to a server.

Attribution & license

This is a derivative work of Oriserve/Whisper-Hindi2Hinglish-Prime, which is licensed under the Apache License 2.0. This conversion is released under the same license. All model weights and their speech-recognition capability are the work of Oriserve; the only changes here are format conversion and inference-graph packaging (see below). The underlying architecture derives from OpenAI's Whisper large-v3.

Usage

import WhisperKit

let config = WhisperKitConfig(
    model: "Oriserve_Whisper-Hindi2Hinglish-Prime_889MB",
    modelRepo: "nitinh/whisperkit-hinglish-coreml"
)
let pipe = try await WhisperKit(config)

// The model emits romanized Hinglish under the English token.
let options = DecodingOptions(task: .transcribe, language: "en")
let result = try await pipe.transcribe(audioPath: path, decodeOptions: options)

What was changed during conversion

1. Added alignment_heads to generation_config.json. The upstream repo omits this field. WhisperKit's TextDecoder expects a fourth output (alignment_heads_weights), so conversion fails with ValueError: Number of outputs provided, 4, do not match the number of outputs detected in the model, 3. The values were copied from openai/whisper-large-v3, whose architecture is identical to this fine-tune (32 encoder/decoder layers, 20 attention heads, d_model 1280, vocab 51866 — verified field by field).

2. Converted to CoreML with whisperkittools (whisperkit-generate-model), producing MelSpectrogram.mlmodelc, AudioEncoder.mlmodelc, and TextDecoder.mlmodelc.

3. Quantized to 4-bit mixed-bit palettization (group size 64), reducing the bundle from 2.9 GB to 889 MB. Conversion fidelity: torch2coreml PSNR 54.5 (decoder) / 60.9 (encoder); all conversion tests passed.

No weights were retrained or pruned; the only numerical change is the quantization in step 3.

Notes

• Outputs romanized Hinglish (Latin script), not Devanagari — pass language: "en", matching the upstream model's documented behaviour.
• Intended for batch transcription. Accuracy on code-mixed speech is substantially better than base Whisper large-v3; see the upstream model card for the author's benchmarks.