happyman11/TacRFC-whisper-large-v3

Whisper Large-v3 — Fine-tuned for Tactical Radio ASR (Hindi-English Code-Switched)

This is openai/whisper-large-v3 fine-tuned with LoRA (then merged into the base weights) to transcribe code-switched Hindi-English tactical radio communications over VHF/UHF push-to-talk channels, decoded to Latin/Roman-script text. It is one of eight architectures (FastConformer, SeamlessM4T v2 Large, IndicConformer, Whisper base/small/medium/large-v3, and wav2vec2-XLSR-53) benchmarked on the TacRFC dataset.

This repository contains the fully merged model (base weights + LoRA delta already folded in). It loads and runs like any standard WhisperForConditionalGeneration checkpoint; no PEFT/adapter loading code is required.

Updated September 2026. This repository now holds a retrained checkpoint. The model was fine-tuned again on chunked audio (each recording split into shorter segments paired with their transcript segments), with early stopping and checkpoint selection on validation loss, and is now scored on a held-out test split with a zero-shot baseline for comparison. The numbers in earlier versions of this card came from a validation split and are not comparable. The previous weights are still available in this repo's commit history.

Model Details

• Base model: openai/whisper-large-v3 (1.55B parameters)
• Architecture: Whisper encoder-decoder Transformer — 32 encoder layers, 32 decoder layers, d_model=1280, 20 attention heads, 128 mel bins
• Fine-tuning method: LoRA, rank=16, alpha=32, dropout=0.05, applied to the q_proj, k_proj, v_proj, out_proj attention projections in both encoder and decoder, then merged (merge_and_unload()) into the base weights before export
• Decode language forced to: en (English task token), so output stays Latin/Roman script even for the romanized-Hindi portions of speech, matching how the training labels are written
• Task: transcribe (speech-to-text, not translation)
• Weights: fp16 model.safetensors

Intended Uses

• Research and benchmarking of ASR robustness on noisy, code-switched, tactical radio speech.
• A starting checkpoint for further fine-tuning on related radio-communications or code-switched Hindi-English speech tasks.

Out of scope: general-purpose English transcription. Use the stock openai/whisper-large-v3 checkpoint for that; this fine-tune is specialized for the tactical-radio domain.

Training Data

This model was fine-tuned on TacRFC — "TacRFC: A Hindi-English Code-Mixed Tactical Radio Frequency Communication Corpus for Low-Resource Domain ASR" (internally referred to during data collection as merged-data-augmented-radio-bg). Apart from the base checkpoint's original pretraining, it was not trained on any public speech dataset.

• Domain: simulated/exercise military tactical voice traffic over VHF and UHF push-to-talk radio nets (e.g. "Delta one, report to DS that the net is through, over").
• Language: code-switched Hindi–English, transcribed entirely in Latin/Roman script (romanized Hindi + English). The training labels contain no Devanagari.
• Source corpus: 804 transcribed recordings matched to audio (out of 805 ground-truth rows), augmented with realistic radio background noise/static to reflect real net conditions.
• Chunking (this run): each recording was segmented into shorter chunks, each paired with the matching part of its transcript (e.g. record660_chunk_3). The chunks were split 80/10/10 with seed 42: about 1,100 training chunks, about 138 validation chunks, and a 138-chunk test set drawn from 117 different recordings.
• Split caveat: this run made its own chunk-level split. It did not use the recording-level 80/10/10 split and 5-fold assignment published with the dataset. Other chunks of a test recording can therefore appear in training, so the test scores may be optimistic for completely unseen recordings or speakers.
• Annotation: every recording is labeled with a channel type (VHF/UHF; fraction VHF ≈ 0.92, UHF ≈ 0.08); a speaker count is available for a fraction ≈ 0.89 of recordings and an English translation for ≈ 0.92. The source spreadsheet also defines a 30-category tactical operation taxonomy (Ambush, Contact with enemy, Casevac/medevac, etc.), but those columns are not populated in the released data.
• Availability: released as a private Hugging Face dataset alongside this model. It was collected/simulated for tactical-communications ASR research and is not derived from real intercepted or classified traffic.

Training Procedure

• Framework: Hugging Face transformers Seq2SeqTrainer, LoRA via peft
• Optimizer / schedule: AdamW (adamw_torch_fused), peak learning rate 2e-5, linear decay after 50 warmup steps, no weight decay, gradient clipping at 1.0
• Batch size: 1 per device × gradient accumulation 8 (effective 8)
• Precision: fp16 mixed precision (gradient checkpointing on)
• Epochs: up to 200, evaluated every epoch, with early stopping (patience 10) on validation loss. Training stopped after epoch 26; the best checkpoint by validation loss (epoch 16) was restored (load_best_model_at_end) before merging and export.
• Seed: 42
• Max transcript length: 448 tokens (Whisper's generation cap). Longer transcripts were excluded from training rather than truncated.
• Test-time decoding: greedy, language="en", task="transcribe", repetition_penalty=1.3, max_new_tokens=256

Evaluation Results

Held-out test split

137 scored test chunks (138 in the split; one chunk has an empty reference and is skipped). WER uses this project's standard normalization (lowercase, punctuation removed, whitespace collapsed); CER is computed on the raw text. All error rates on this card are fractions between 0 and 1 (lower is better); values above 1 mean more errors than reference words or characters.

Metric	Zero-shot baseline	Fine-tuned (this model)
Word Error Rate (WER)	0.7553	0.5779
Character Error Rate (CER)	0.6630	0.4847

The zero-shot baseline is the unmodified openai/whisper-large-v3 checkpoint, decoded with the same settings on the same chunks.

Best model in this benchmark round. Fine-tuning lowers test WER by about 0.18 (0.7553 → 0.5779) and CER by about 0.18 (0.6630 → 0.4847). The zero-shot model already starts ahead of the other baselines, and it keeps the lead after fine-tuning.

Validation WER was still falling when early stopping (on validation loss) ended the run at epoch 26: it reached 0.6043 at the final epoch, against 0.6567 at the selected epoch 16. Selecting on WER, or training longer, may give a somewhat better checkpoint.

Comparison with the other retrained models (same 138-chunk test set)

Model	WER zero-shot → fine-tuned	CER zero-shot → fine-tuned
Whisper Large-v3 (this model)	0.7553 → 0.5779	0.6630 → 0.4847
FastConformer Hybrid (RNNT+CTC)	0.9643 → 0.6174	0.9302 → 0.5636
Whisper Medium	0.8417 → 0.6263	0.7543 → 0.4945
Whisper Small	0.8761 → 0.6532	0.7522 → 0.5230
wav2vec2-XLSR-53	0.9717 → 0.8799	0.7068 → 0.5769

Whisper Base, SeamlessM4T v2 Large and IndicConformer were not retrained in this round. Their published numbers come from different evaluation splits, so they are not included in this table.

Validation split (best checkpoint)

Metric	Value
Word Error Rate (WER)	0.6567
Character Error Rate (CER)	0.5447
BERTScore F1	0.7707
Validation loss	1.5225
Selected checkpoint	epoch 16 (of 26 trained)

These values were logged during training. They use the training loop's own decoding and scoring, not the normalized test-time evaluation above, so compare models on the test numbers.

How to Use

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "happyman11/TacRFC-whisper-large-v3"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

processor = WhisperProcessor.from_pretrained(model_id, language="en", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=dtype).to(device)
model.eval()

audio, _ = librosa.load("your_radio_clip.wav", sr=16000, mono=True)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device, dtype)

with torch.no_grad():
    ids = model.generate(
        inputs,
        language="en",
        task="transcribe",
        repetition_penalty=1.3,   # helps avoid single-token loops on repetitive radio phrasing
        max_new_tokens=256,
    )
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

The model was trained and evaluated on short chunks. For long recordings, split the audio into segments of up to 30 seconds (Whisper's input window) before transcribing.

Limitations and Ethical Considerations

• Research artifact, not a production system. Fine-tuned on a very small corpus (804 recordings, about 1,100 training chunks) to benchmark how different architectures adapt to noisy, code-switched tactical radio speech. Absolute error rates remain high; treat outputs as a strong hint at content, not verbatim transcripts.
• Narrow domain. Performance is expected to degrade sharply outside VHF/UHF radio traffic, tactical/military phraseology, and Hindi-English code-switching in Latin script. It is not tuned for conversational, broadcast, or clean studio speech.
• Dual-use / sensitive domain. This model transcribes simulated military tactical radio communications. It is released for defense-communications ASR research and benchmarking purposes. Do not use it to process real operational, classified, or intercepted communications, and do not deploy it in any surveillance or targeting system without appropriate legal authorization and human oversight.
• No PII/OPSEC review guarantee. While the underlying dataset is a research corpus rather than real intercepted traffic, no formal OPSEC/PII audit of model outputs has been performed. Review outputs before publishing them verbatim in downstream applications.
• Small, chunk-level test set. 137 scored chunks is a small sample, and the chunk-level split lets other parts of a test recording appear in training (see Training Data). Treat the error rates as noisy estimates.

Citation

If you use this model, please cite the corresponding paper.

@misc{tacrfc_whisper_large_v3,
  title  = {TacRFC: A Hindi-English Code-Mixed Tactical Radio Frequency Communication Corpus for Low-Resource Domain ASR},
  author = {[Your name / lab], IIIT-Delhi},
  year   = {2026},
  note   = {Fine-tuned Whisper Large-v3 checkpoint, TacRFC corpus}
}

(Replace the author field above with the correct attribution before publishing.)