whisperkit-hinglish-large-v3-coreml

A float16 Core ML conversion of Trelis/whisper-hinglish-preview packaged for WhisperKit on Apple Silicon (macOS 14+ / iOS 17+).

It ships compiled .mlmodelc bundles, ready for on-device inference through WhisperKit's Swift API. For Python inference, use the upstream repo linked above.

Output script: Devanagari for Hindi, Latin for English, switching mid-sentence as the speaker does.

Usage

import WhisperKit

let pipe = try await WhisperKit(
    WhisperKitConfig(
        model: "Trelis_whisper-hinglish-preview",
        modelRepo: "vbhar/whisperkit-hinglish-large-v3-coreml"
    )
)
let result = try await pipe.transcribe(audioPath: "meeting.wav")

Or download the variant folder directly and point WhisperKit at a local path:

hf download vbhar/whisperkit-hinglish-large-v3-coreml \
  --include "Trelis_whisper-hinglish-preview/*" \
  --local-dir ./models

Layout

Trelis_whisper-hinglish-preview/
├── MelSpectrogram.mlmodelc/
├── AudioEncoder.mlmodelc/     # ~1.2 GB weights
├── TextDecoder.mlmodelc/      # ~1.7 GB weights
├── config.json
└── generation_config.json

No quantized variants and no decoder prefill data are included — float16 only.

Attribution and modifications

Lineage:

This repo	vbhar/whisperkit-hinglish-large-v3-coreml — Core ML float16 conversion
Direct source	Trelis/whisper-hinglish-preview — Apache-2.0
Its base	ARTPARK-IISc/whisper-large-v3-vaani-hindi
Upstream architecture	openai/whisper-large-v3

All credit for the model itself belongs to Trelis and, upstream of them, to ARTPARK-IISc and OpenAI. Licensed under the Apache License, Version 2.0; see https://www.apache.org/licenses/LICENSE-2.0.

Statement of modifications (Apache-2.0 §4(b)): the weights were converted from float32 to float16 and compiled to Core ML for the Apple Neural Engine. They are otherwise unmodified — no fine-tuning, no distillation, no pruning, no re-training. One structural change was required to make the model usable at all: the output embedding was resized from 51867 to 51866 tokens, dropping the trailing <|mixedcode|> token row. See below for why this is necessary and what it costs.

Reproducing this conversion

Converted with argmaxinc/whisperkittools under Python 3.11 (3.14 has no working coremltools/torch wheels as of this writing):

python3.11 -m venv .venv && source .venv/bin/activate
pip install git+https://github.com/argmaxinc/whisperkittools.git

whisperkit-generate-model \
  --model-version Trelis/whisper-hinglish-preview \
  --output-dir out \
  --repo-path-suffix fp16

float16, no quantized variants, no decoder prefill data. Apply the vocab fix below before the decoder is traced.

Conversion gotcha: the 51867th token silently breaks the tokenizer

Trelis/whisper-hinglish-preview has vocab_size: 51867 — one token more than whisper-large-v3's 51866. The extra entry is a <|mixedcode|> control token.

That single extra row is enough to produce a model that loads, runs, and emits garbage, with no error anywhere:

• WhisperKit selects its tokenizer solely from the width of the decoder's logits tensor.
• 51866 → it loads the openai/whisper-large-v3 tokenizer.
• Anything else → it silently falls back to openai/whisper-base.
• whisper-base is a 51865-token vocabulary with different special-token ids. Every <|startoftranscript|>, language, task and timestamp id is shifted, so decoding produces plausible-looking but wrong text — or control-token soup. Nothing throws.

Fix: shrink the vocabulary back to 51866 before conversion.

from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("Trelis/whisper-hinglish-preview")
model.resize_token_embeddings(51866)   # drops the trailing <|mixedcode|> row

proj_out is weight-tied to the input embedding, so this one call resizes both the embedding matrix and the decoder's output projection; the resulting logits tensor is [1, 1, 51866] and WhisperKit picks the correct large-v3 tokenizer. The cost is that <|mixedcode|> becomes unaddressable — in practice it is not needed, as the model code-switches from context anyway.

The converted decoder in this repo is verified to emit logits: [1, 1, 51866], so consumers of this repo do not need to do anything — the fix matters only if you re-convert from the Trelis source yourself.

A note on quantization

Float16 was chosen deliberately. During this work, a quantized Core ML build of these weights was observed to emit the literal string nan on real code-switched Hindi/English speech, while passing cleanly on short English clips such as WhisperKit's bundled jfk.wav.

The asymmetry is the point worth noting: quantization error is input-dependent, so a short smoke test — the normal way to sanity-check a conversion — can pass on a build that fails on the audio you actually care about. If you evaluate a quantized Whisper conversion, test it on your own code-switched audio at realistic length, not on a short clip.

No quantized variant is published here, because none was validated on that kind of input.

Limitations

• ~2.9 GB on disk. Full large-v3 parameter count at float16.
• First load is slow. The Neural Engine compiles the model once on first use: roughly 115–190 s on an M-series Mac. Subsequent warm loads are ~1–3 s. Cache the compiled artifacts and show progress; do not block a user on the cold path silently.
• Apple Silicon only. Requires macOS 14+ / iOS 17+ and an ANE.
• <|mixedcode|> is gone by construction (see above). Code-switching still works; the token just cannot be forced.
• Script behaviour is not configurable. This model writes Hindi in Devanagari and English in Latin. If you want Hindi romanized into Latin script, this is the wrong model — use vbhar/whisperkit-hindi2hinglish-prime-coreml instead. Picking the wrong one of the two is the most likely mistake with this pair.
• Marked preview upstream; inherits the biases and domain limits of the Vaani-Hindi base.

Verified usage

Tested live end-to-end in a real macOS menu-bar meeting-notes app (mic + system audio, streaming transcription) on natural Hindi/English code-switched speech.

Hindi is returned in Devanagari and English in Latin script, mixed inline within the same sentence.