litert-community/whisper-acft

Whisper ACFT — short-window TFLite (LiteRT), tiny/base/small × multilingual/.en

TFLite (LiteRT) exports of the FUTO audio-context fine-tuned (ACFT) Whisper checkpoints: fixed 5 s / 10 s / 30 s window graphs split into encode and decode signatures, matching the graph interface of litert-community/whisper-tiny and litert-community/whisper-base. Six models in one repo: tiny, base, small (multilingual) and tiny.en, base.en, small.en (English-only), each in three window lengths.

No retraining — the weights are FUTO's Apache-2.0 ACFT checkpoints (futo-org/whisper-acft method and training; HF checkpoints futo-org/acft-whisper-{tiny,base,small}{,.en}). This repo only converts them: transformers TFWhisperForConditionalGeneration → two-signature graph → converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT). Conversion by the LiteRT-LM-Unity project.

Why short windows

Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders are trained on the full 30 s window, so shrinking it puts them out of distribution (repetition loops: stock base at a 5 s window measured CER 10.4 on a sentence clip in our evaluation); the ACFT checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this.

Repo layout

One folder per model; filenames encode model and window. All files are dynamic-range quantized (int8 weights, fp32 activations).

Folder	Source checkpoint	Files (5s / 10s / 30s)	Size
tiny/	futo-org/acft-whisper-tiny	acft_whisper_tiny_{5s,10s,30s}_drq.tflite	57–58 MB
base/	futo-org/acft-whisper-base	acft_whisper_base_{5s,10s,30s}_drq.tflite	97–99 MB
small/	futo-org/acft-whisper-small	acft_whisper_small_{5s,10s,30s}_drq.tflite	273–276 MB
tiny.en/	futo-org/acft-whisper-tiny.en	acft_whisper_tiny.en_{5s,10s,30s}_drq.tflite	57–58 MB
base.en/	futo-org/acft-whisper-base.en	acft_whisper_base.en_{5s,10s,30s}_drq.tflite	97–99 MB
small.en/	futo-org/acft-whisper-small.en	acft_whisper_small.en_{5s,10s,30s}_drq.tflite	273–276 MB

Window ↔ encode input shape: 5 s = [1, 80, 500], 10 s = [1, 80, 1000], 30 s = [1, 80, 3000].

Integration notes

• Fixed short-window graphs: the encode signature input is [1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames.
• The decode signature input order is (mask, audio, tokens) — opposite of the stock litert-community exports — bind decode tensors by shape/name rather than position. The decoder is a fixed-length full re-run per step (no KV cache), matching the tiny/base graph interface.
• Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the encode signature on one code path.

Token ids — multilingual vs .en

• Multilingual (tiny/, base/, small/): 80 mel bins / vocab 51865 — classic Whisper token layout, same as litert-community tiny/base. Decode prompt e.g. [<|startoftranscript|>=50258, <|lang|>, <|transcribe|>=50359, <|notimestamps|>=50363]. Tokenizer: tokenizer.json from the matching openai/whisper-* repo.
• English-only (tiny.en/, base.en/, small.en/): 80 mel bins / vocab 51864 — English-only layout: <|startoftranscript|> = 50257, <|endoftext|> = 50256, <|notimestamps|> = 50362; there are no language/task tokens — the decode prompt is [50257, 50362]. Tokenizer: tokenizer.json from the matching openai/whisper-*.en repo.

Validation

Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed). Multilingual clips: Korean sentence + English sentence + short Korean voice commands; .en clips: English sentence pair.

tiny

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	2025년 3월 5일 전술평가 결과 보고 (3.98 s)	2015년 3월호일 전술 평가 결과 보고	0.118	0.015	0.36 (12)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather in Seoul is cloudy.	0.000	0.013	0.26 (8)
5s	소리 키워줘 (1.32 s)	소리 키워줘	0.000	0.014	0.14 (4)
5s	음량 증가 (1.15 s)	능량 증가	0.250	0.015	0.23 (6)
10s	2025년 3월 5일 전술평가 결과 보고	2015년 3월호일 전술 평가 결과보고	0.118	0.046	0.43 (13)
10s	The current weather in Seoul is cloudy	The current weather and soil is cloudy.	0.125	0.034	0.28 (8)
10s	소리 키워줘	소리 키워줘	0.000	0.040	0.15 (4)
10s	음량 증가	능냥 증가	0.500	0.040	0.24 (6)
30s	2025년 3월 5일 전술평가 결과 보고	2015년 3월호일 전술 평가 결과보고	0.118	0.214	0.67 (13)
30s	The current weather in Seoul is cloudy	The current weather and soul is cloudy.	0.094	0.210	0.45 (8)
30s	소리 키워줘	소리 키워줘	0.000	0.210	0.25 (4)
30s	음량 증가	는 양증가	0.500	0.206	0.31 (5)

The short windows track the checkpoint's own 30 s behavior (same clips right/wrong at every window; residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window.

tiny.en

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	Tactical Evaluation Results Report - March 5, 2025 (4.87 s)	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.016	0.29 (10)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather and soul is cloudy.	0.094	0.014	0.28 (8)
10s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.041	0.34 (10)
10s	The current weather in Seoul is cloudy	The current weather and soul is cloudy.	0.094	0.038	0.28 (8)
30s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report. March 5, 2025.	0.000	0.206	0.59 (10)
30s	The current weather in Seoul is cloudy	The current weather and soul is cloudy.	0.094	0.202	0.44 (8)

Transcript-identical to the 30 s window on both clips (including the same "and soul" mishearing — a tiny-capacity trait, present at 30 s too) at ~13x encoder speedup for the 5 s window.

base

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	2025년 3월 5일 전술평가 결과 보고 (3.98 s)	2025년 3월 5일 전술 평가 결과 보고	0.000	0.039	0.62 (12)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather and soul is cloudy.	0.094	0.045	0.44 (8)
5s	소리 키워줘 (1.32 s)	소리 키워줘	0.000	0.031	0.23 (4)
5s	음량 증가 (1.15 s)	음량 증가	0.000	0.036	0.27 (5)
10s	2025년 3월 5일 전술평가 결과 보고	2025년 3월 5일 전술 평가 결과 보고	0.000	0.092	0.72 (12)
10s	The current weather in Seoul is cloudy	The current weather and soul is cloudy.	0.094	0.080	0.53 (8)
10s	소리 키워줘	소리 키워줘	0.000	0.086	0.30 (4)
10s	음량 증가	음량 증가	0.000	0.090	0.33 (5)
30s	2025년 3월 5일 전술평가 결과 보고	2025년 3월 5일 전술 평가 결과 보고	0.000	0.422	1.20 (12)
30s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	0.428	0.85 (8)
30s	소리 키워줘	소리 키워줘	0.000	0.436	0.47 (4)
30s	음량 증가	음량 증가	0.000	0.449	0.60 (5)

~11x encoder speedup for the 5 s window; base is the recommended minimum for non-English voice commands.

base.en

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	Tactical Evaluation Results Report - March 5, 2025 (4.87 s)	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.035	0.52 (10)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather and soul is cloudy.	0.094	0.046	0.48 (8)
10s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.080	0.62 (10)
10s	The current weather in Seoul is cloudy	The current weather and soul is cloudy.	0.094	0.087	0.52 (8)
30s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.449	1.05 (10)
30s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	0.449	0.88 (8)

small

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	2025년 3월 5일 전술평가 결과 보고 (3.98 s)	2025년 3월 5일 전술평가 결과 보고	0.000	0.145	1.77 (13)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather in Seoul is cloudy.	0.000	0.145	1.17 (8)
5s	소리 키워줘 (1.32 s)	소리 키워줘	0.000	0.155	0.64 (4)
5s	음량 증가 (1.15 s)	음량 증가	0.000	0.128	0.82 (5)
10s	2025년 3월 5일 전술평가 결과 보고	2025년 3월 5일 전술평가 결과 보고	0.000	0.311	2.27 (13)
10s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	0.350	1.50 (8)
10s	소리 키워줘	소리 키워줘	0.000	0.339	0.83 (4)
10s	음량 증가	음량 증가	0.000	0.332	1.03 (5)
30s	2025년 3월 5일 전술평가 결과 보고	2025년 3월 5일 전술평가 결과 보고	0.000	1.456	4.19 (13)
30s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	1.446	2.73 (8)
30s	소리 키워줘	소리 키워줘.	0.000	1.545	1.74 (5)
30s	음량 증가	음량 증가	0.000	1.495	1.85 (5)

12/12 exact across all three windows — the 5 s window is character-identical to the 30 s window at ~10x encoder speedup.

small.en

Window	Clip	Transcript	CER	Encode s	Decode s (steps)
5s	Tactical Evaluation Results Report - March 5, 2025 (4.87 s)	Tactical Evaluation Results Report, March 5, 2025	0.000	0.138	1.30 (9)
5s	The current weather in Seoul is cloudy (2.9 s)	The current weather in Seoul is cloudy.	0.000	0.144	1.16 (8)
10s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report, March 5, 2025.	0.000	0.312	1.74 (10)
10s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	0.321	1.48 (8)
30s	Tactical Evaluation Results Report - March 5, 2025	Tactical Evaluation Results Report, March 5, 2025.	0.000	1.490	3.30 (10)
30s	The current weather in Seoul is cloudy	The current weather in Seoul is cloudy.	0.000	1.516	2.56 (8)

All exact. In the same evaluation pipeline on whisper-base, the ACFT 5 s graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup (f32) / ~20x (drq); stock checkpoints at a 5 s window (no ACFT) fall into a repetition loop on sentence-length clips (CER 10.4). Short-window graphs are device-validated on Android arm64 (Snapdragon 865, 0.5–0.8 s end-to-end per command clip for tiny/base 5 s) in the producing project.

Caveats

• The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; use the 30 s graph for dictation.
• Decoder is a full-sequence re-run per step (no KV cache) to match the litert-community tiny/base interface — a KV-cached runtime will be substantially faster per token.
• tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.
• Language forcing recommended for short clips on the multilingual models (e.g. <|ko|> / <|en|>).

Credits and license

• ACFT method and checkpoints: FUTO (futo-org/acft-whisper-*, Apache-2.0, derived from the MIT openai/whisper-* checkpoints). This repo contains TFLite conversions of those checkpoints, no retraining.
• Conversion and validation: LiteRT-LM-Unity.
• License: Apache-2.0, matching the source ACFT checkpoints.

Related: Korean-fine-tuned ACFT short-window models (tiny/base/medium/large-v3-turbo) live at leuconoe/whisper-acft-ko.