prasadvittaldev/svanita-0.6b

Svanita — speech-to-text for Indian languages, on a CPU

Svanita is a speech-to-text model family for South and Southeast Asian languages, starting with India. It is built for how people actually speak: mixing English words into their own language, over app audio and phone lines, with the transcript produced fast enough on an ordinary CPU for a live voice agent.

This release (v0.1) covers Telugu. Telugu + Hindi is in training; more Indian languages follow on the same recipe.

Svanita transcribes the local language and the English mixed into it in one transcript — native script for the native words, Latin script for the English ones:

OK home loan కావాలి అంటే మీకు ఎంత కావాలి

It runs faster than real time on an ordinary desktop CPU, with no GPU: a 3-second reply is transcribed in about 0.28 seconds.

svanita_demo.mp4 — how it runs fast on a CPU, then five clips from speakers it never heard in training, transcribed live.

Built by Prasad Vittaldev. Follow on LinkedIn for new languages and voices.

Why Svanita exists

Conversation across India is code-switched. In Telugu: "documents submit చేయాల్సి ఉంటుంది bank వచ్చి మీరు". Existing open Telugu recognizers write those English words in Telugu script (డాక్యుమెంట్స్ సబ్మిట్). That is accurate as phonetics, but a downstream LLM, search index or form-filler reads it as an unknown Telugu word. Svanita writes them as English, which is what a voice agent actually needs.

The second goal was CPU real time. Svanita is built on NVIDIA's Parakeet TDT, whose architecture is unusually cheap to run on a CPU (see How it is fast on a CPU).

Model

Architecture	FastConformer encoder (24 layers) + TDT (token-and-duration transducer) decoder
Parameters	~620M (encoder 609M, decoder + joint ~10M)
Languages in this release	Telugu, with code-switched English
Base model	nvidia/parakeet-tdt-0.6b-v3
Vocabulary	new 2,049-piece tokenizer covering Telugu script and Latin (the base model's vocabulary has no Telugu)
Input	16 kHz mono audio (wideband or telephone quality)
Output	Telugu script, with English words in Latin script; lowercase except acronyms (OK, OTP); no punctuation
Licence	CC-BY-4.0

Usage

Tested with transformers 5.12.1 and torch 2.11.

With transformers (simplest)

import torch, soundfile as sf
from transformers import ParakeetForTDT, ParakeetProcessor

repo = "prasadvittaldev/svanita-0.6b"
processor = ParakeetProcessor.from_pretrained(repo)
model = ParakeetForTDT.from_pretrained(repo).eval()

audio, sr = sf.read("sample_3.wav", dtype="float32")        # 16 kHz mono
inputs = processor(audio, sampling_rate=16000)
with torch.inference_mode():
    out = model.generate(**inputs)
print(processor.batch_decode(out.sequences, skip_special_tokens=True)[0])
# మళ్ళీ దానికి కూడా documents submit చేయాల్సి ఉంటుంది bank వచ్చి మీరు

Fast CPU transcription

transcribe_cpu.py is a single-file greedy TDT decoder over the model's own modules — the loop the speed figures below were measured with:

pip install transformers torch soundfile
python transcribe_cpu.py sample_1.wav --threads 8
# OK home loan కావాలి అంటే మీకు ఎంత కావాలి
# [3.6 s of audio transcribed in 295 ms on CPU, 8 threads]

Three sample clips (sample_1.wav, sample_3.wav, sample_5.wav) are included; all three speakers are from the held-out test set.

Evaluation

All scores are on speakers never seen in training. The same normalization is applied to every system (Unicode NFC, zero-width characters removed, punctuation stripped, Latin lowercased).

Two WER figures are reported, because they answer different questions:

• WER (either script) counts an English word as correct whether the system wrote it in Latin (order) or Telugu script (ఆర్డర్). This is the fairest measure of recognition across systems.
• WER (Latin required) counts it as correct only in Latin. This measures what Svanita is designed for: text a downstream system can read as English.

IndicVoices Telugu, conversational / extempore speech — 1,499 clips, 2.34 h, 153 speakers:

system	WER (either script)	WER (Latin required)	CER	English kept in Latin
Svanita	36.5	36.9	16.4	76%
IndicConformer 600M, RNN-T	27.4	39.1	22.1	0%
IndicConformer 600M, CTC	29.0	40.3	22.1	0%
Whisper large-v2, Telugu fine-tune	44.0	52.3	32.0	0%

Other slices

slice	Svanita WER / CER	IndicConformer RNN-T	Whisper-Telugu large-v2
IndicVoices, 8 kHz telephone copy	38.8 / 18.2	31.1 / 23.5	45.5 / 33.0
IndicVoices, code-mixed clips only (800)	37.6 / 19.2 (Latin required: 38.2)	29.8 / 33.5 (Latin required: 49.2)	49.0 / 44.1 (Latin required: 62.7)
Kathbath, read speech (2,378 clips, 20 speakers)	36.3 / 8.9	20.9 / 3.1	23.8 / 4.2
Kathbath, 8 kHz telephone copy	40.2 / 11.3	23.6 / 4.2	26.8 / 5.9

Baselines: ai4bharat/indic-conformer-600m-multilingual and vasista22/whisper-telugu-large-v2, run on the same audio with the same scoring.

Reading the results honestly: IndicConformer recognizes more words correctly, clearly so on read speech, and is the stronger recognizer overall. Svanita is ahead on character accuracy, on keeping English as English (it wins every Latin-required comparison, by 11 points on code-mixed speech), and against Whisper-Telugu on conversational speech. Short turns are the hardest case for every system: on clips of 3 seconds or less, Svanita's WER is 47.2% (IndicConformer RNN-T: 42.1%).

How it is fast on a CPU

1. It only processes the audio it is given. The encoder downsamples 8×, so one second of speech is just 13 frames and a 3-second reply is 38. Whisper pads every input to a fixed 30-second window (1,500 frames) regardless of length.
2. The big network runs once. The 609M-parameter encoder reads the whole clip in a single pass; only the ~10M-parameter decoder and joint network loop while the text is written. About 98% of the model runs once per utterance.
3. It jumps over frames. A token-and-duration transducer predicts, at each step, a word-piece and how many frames to skip (0–4), so pauses and long sounds are crossed in one hop instead of frame by frame.

Measured latency — AMD Ryzen 5 8600G (6 cores / 12 threads), fp32, no GPU, transcribe_cpu.py, median of 10 runs:

turn length	4 threads	8 threads
1 s	227 ms	220 ms
3 s	293 ms	275 ms
5 s	393 ms	396 ms

That is 10–13× faster than real time on 3–5 second turns. For comparison, Whisper-Telugu large-v2 reached 4× real time on a GPU in our evaluation.

Keep it in full precision. In our tests, PyTorch dynamic int8 quantization made it about twice as fast but raised WER from 38.6% to 69.3% on the dev set, and an ONNX export was fast but not numerically faithful (48.3% WER). The fp32 model above is what we recommend and what the numbers describe.

Training

Data — 451 hours of Telugu, 241,291 clips, 1,691 speakers:

• ai4bharat/IndicVoices (Telugu): conversational, extempore and read speech from many districts. Its annotations mark English words spoken inside Telugu (టైం [time]); Svanita's targets use the Latin form, so 46% of training clips contain Latin-script English. Transcriber annotations ([noise], [uhh], [stammers], …) are removed.
• ai4bharat/Kathbath (Telugu): read speech.
• Held-out evaluation speakers were removed from training entirely (76,692 training clips from speakers who also appear in the IndicVoices validation split were dropped).
• About 40% of training clips were degraded on the fly to telephone quality (300–3,400 Hz, 8 kHz, μ-law/8-bit quantization, 15–30 dB noise), so one model serves app audio and phone calls. SpecAugment was also applied.

Recipe

1. Vocabulary swap. The base model's tokenizer has no Telugu. A 2,049-piece Unigram tokenizer was trained on the Telugu+Latin transcripts, and only the two vocabulary-shaped tensors were re-initialized (decoder embedding and joint output head, ~2.6M parameters). The pretrained duration outputs were kept; the 609M encoder and the decoder LSTM were kept.
2. Stage A — encoder frozen, decoder and joint trained: 17,680 steps, learning rate 1e-3, 8-bit AdamW.
3. Stage B — full fine-tune from Stage A's best checkpoint: 35,359 steps (4 epochs), learning rate 3e-4 with the encoder at 3e-5, warmup 2,000, cosine decay, bf16 autocast, gradient checkpointing, gradient clipping at 1.0. This checkpoint is Stage B step 30,000, selected by dev WER on held-out speakers (dev WER 38.6% wideband, 42.5% at 8 kHz).

Trained on a single NVIDIA RTX 5060 Ti (16 GB).

Limitations

• Accuracy is well short of the best open Telugu recognizer. About one word in three is wrong on conversational speech, and nearly one in two on very short turns. It is suitable for intent-level understanding by a downstream LLM, not for text a person will read verbatim.
• Read speech is a weakness (36.3% WER on Kathbath versus 20.9% for IndicConformer).
• Pure English sentences are not supported well. English was learned only as words inside Telugu speech; on all-English clips WER is high and some English words, especially Indian names, come out in Telugu script.
• English words fused to Telugu suffixes (డేస్‍లో, "days-lo") are usually written in Telugu script, because the training annotations do not split them.
• No punctuation or casing apart from acronyms.
• Speaker coverage is the IndicVoices/Kathbath population; accents, age groups and recording conditions outside it are untested.

Licence and attribution

Released under CC-BY-4.0. It is derived from nvidia/parakeet-tdt-0.6b-v3 (CC-BY-4.0) and trained on IndicVoices and Kathbath (both CC-BY-4.0, AI4Bharat). If you use Svanita, please credit those works as well.

Roadmap

release	languages	status
v0.1 (this checkpoint)	Telugu + code-switched English	released
v0.2	Telugu + Hindi + code-switched English, one model	in training
later	more Indian languages, then South and Southeast Asia	planned

The recipe (vocabulary swap on a CPU-fast transducer, English kept in Latin script, telephone-quality augmentation) is language-agnostic; each new language needs transcribed speech, not new architecture. Follow Prasad Vittaldev on LinkedIn for new releases.