ka-fastconformer: Georgian speech recognition for conversation and phone calls (FastConformer, 115M)

A Georgian speech-to-text model. It is NVIDIA's stt_ka_fastconformer_hybrid_large_pc fine-tuned on 429 hours of Georgian speech, most of it unscripted speech from YouTube. It was trained on simulated phone lines, VoIP codecs and background noise, so that it holds up on calls, and it writes punctuation.

The base model was trained on read speech (Common Voice and FLEURS). On spontaneous speech from YouTube channels that were not used in training, this model gets 0.254 WER, against 0.456 for the base model.

• Size and decoders: 115M parameters. A FastConformer encoder feeds two decoders: Transducer (the default, more accurate) and CTC (faster).
• Output: Georgian letters, spaces and the marks . , ?. Numbers are written as words; there are no digits or Latin letters.
• Input: 16 kHz mono audio. Resample phone audio (8 kHz) to 16 kHz first. It was trained on clips of 1–25 s; cut longer audio into windows of up to 30 s (see below).

Results

All test sets are held out: their channels or speakers never appear in training. Before scoring, every transcript was normalized the same way: no punctuation, numbers as words. WER is the share of words wrong and CER the share of letters wrong; lower is better.

test set	clips	WER / CER, Transducer	WER / CER, CTC decoder
YouTube, 7 channels, creator-made subtitles	1,952 (7.8 h)	0.254 / 0.111	0.295 / 0.115
Conversational YouTube, 10 channels (references: YouTube's automatic captions)	2,905 (6.4 h)	0.217 / 0.080	0.261 / 0.085
Public institutions (references: YouTube's automatic captions)	3,000 (8.0 h)	0.122 / 0.037	0.155 / 0.042
FLEURS Georgian, test	774 (2.3 h)	0.117 / 0.027	0.144 / 0.033
Common Voice Georgian, test (see the note)	3,000 (4.9 h)	0.044 / 0.009	0.058 / 0.012

• Public institutions: 44 videos from 7 channels: the National Bank of Georgia, the Ministries of Infrastructure and of Justice, and channels of the Georgian Public Broadcaster.
• Automatic-caption references are YouTube's own speech recognition, not human transcripts, so they contain errors of their own. Read those two rows as approximate.
• FLEURS: sentences that contain digits are left out, because the copy we used had their numbers spelled out wrongly.
• Common Voice: don't read this as a fair score. The base model was trained on Common Voice 17, and Common Voice reshuffles its splits between releases, so many of these recordings were probably in its training data.
• The base model on the same clips: YouTube 0.456, FLEURS 0.124, Common Voice 0.039.

Phone calls (simulated)

The same test clips, degraded the same way for every run: the condition of each clip is seeded by its id.

condition	YouTube, creator subtitles	Conversational YouTube	FLEURS
clean	0.254	0.217	0.117
phone line: G.711 mu-law, 8 kHz, 300–3400 Hz	0.292	0.248	0.142
VoIP: Opus at 8 kbps	0.316	0.272	0.178
phone line + 6 background voices at 10 dB SNR	0.458	0.398	0.258

WER of the Transducer decoder.

Punctuation

Scored on clips whose reference has marks: the transcript and the reference are aligned word by word, and a mark counts as right when the same mark follows the same word. Precision is the share of written marks that are right; recall, the share of the reference's marks that the model wrote.

test set	clips	. F1	, F1	any mark in the right place: precision / recall / F1
Public institutions	2,053	0.55	0.56	0.89 / 0.49 / 0.63
Conversational YouTube	1,114	0.46	0.46	0.86 / 0.36 / 0.51
YouTube, creator subtitles	1,281	0.32	0.34	0.66 / 0.25 / 0.36
Common Voice (read sentences)	2,990	0.99	0.83	0.96 / 0.94 / 0.95

The model is precise but sparse: most marks it writes are right, but it writes only a quarter to a half of the marks in the references. Subtitles punctuate inconsistently, so part of the low recall is the references. It almost never writes ?.

Speed

On one NVIDIA A100 PCIe 40 GB with batches of 32 clips, the Transducer decoder runs at about 670x real time and the CTC decoder at about 2,900x. On an RTX 6000 Ada, a single 30 s window takes about 264 ms with the Transducer and 40 ms with CTC (measured with the same network).

How to use

# pip install "nemo_toolkit[asr]"
import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.from_pretrained("lukab/ka-fastconformer")
print(model.transcribe(["audio.wav"])[0].text)  # 16 kHz mono

model.change_decoding_strategy(decoder_type="ctc")  # optional: the faster CTC decoder

To transcribe a long recording, cut it into windows of up to 30 s:

import soundfile as sf

audio, sr = sf.read("long.wav", dtype="float32")  # 16 kHz mono
win = 30 * sr
chunks = [audio[i:i + win] for i in range(0, len(audio), win)]
text = " ".join(h.text for h in model.transcribe(chunks, batch_size=16))

Fixed cuts can split a word at a window edge. Cutting at pauses avoids that.

Training

Two fine-tuning stages, both of all weights. The tokenizer (SentencePiece, 1,024 tokens, with . , ?) is the base model's. The settings of both stages are in training_config.yaml.

• Stage 1: 356.8 h of transcribed speech with plain labels (no punctuation), for 3 epochs.

  • Data: YouTube videos with creator-made subtitles, Common Voice and FLEURS.
  • Augmentation: SpecAugment and speed perturbation.

• Stage 2: 428.8 h for 2 epochs from the stage-1 weights. This stage adds conversation, phone audio and punctuation.

  source	hours
  YouTube: conversational channels, YouTube's automatic captions	154.6
  YouTube: creator-made subtitles, human speakers (at most 8 h per channel)	143.2
  YouTube: narration channels not verified as human	27.6
  YouTube: AI-narrated channels (capped)	4.5
  TV phone-in shows, YouTube's automatic captions	1.75
  Common Voice Georgian (train splits)	93.5
  FLEURS Georgian (train)	3.7

  • Labels keep the original subtitle punctuation (. , ?); numbers are spelled out.
  • Automatic captions were cut at their word timings into clips of up to 20 s, with a cut at every change of speaker.
    • A clip was kept only if the stage-1 model's transcript agreed with its caption (CER at most 0.4).
    • A channel was dropped if its median CER was above 0.35: 6 of 131 channels.
    • 94% of the hours were kept.
  • Phone-call augmentation, one channel per clip:
    • 50% clean;
    • 20% phone line (G.711 mu- or A-law at 8 kHz, 300–3400 Hz);
    • 15% low-bitrate VoIP (Opus 6–12 kbps);
    • 15% good VoIP (Opus 16–32 kbps).
  • Before the channel:
    • background on 35% of the clips at 5–20 dB SNR: 40% of it babble of 3–6 other training clips, the rest MUSAN noise;
    • room echo on 10% (RT60 0.2–0.7 s);
    • packet loss on 20% of the VoIP clips (1–3% of 20 ms frames).
  • Speed perturbation (0.9 / 1.0 / 1.1) and SpecAugment as in stage 1.

• Recipe (NeMo), stage 2:

  • 11,150 updates of 32 clips.
  • Optimizer: AdamW (betas 0.9 and 0.98, weight decay 1e-3).
  • Learning rate: peak 1e-4, 5% warmup, cosine decay to 1e-6.
  • Gradient clipping at 1.0 and bf16 mixed precision.
  • Hybrid loss: Transducer + 0.3 × CTC.

• Checkpoint: chosen by WER (on words, without punctuation) on 150 YouTube dev clips from held-out channels; the last update was the best.

• Splits are by channel: no dev or test channel appears in training.

• Hardware: stage 1 on one RTX 6000 Ada 48 GB (1.8 h); stage 2 on one A100 PCIe 40 GB (1.6 h).

Limitations

• Real calls are only partly tested. The call results above come from simulated phone lines and codecs on recorded speech. Background voices at 10 dB still nearly double the errors.
• Spontaneous speech is still hard, especially fast or overlapping speech.
• Punctuation follows subtitle habits: periods and commas are mostly right where written, but many are left out, and question marks are rare.
• No digits or Latin letters in the output.
• Offline, not streaming: the model needs the whole window before it writes text. Live captions need chunked decoding or a streaming model.
• Narrow domain: the YouTube data covers a limited set of channels and topics. Far-field and meeting audio are untested.

License and credits

• License: CC-BY-4.0, the license of the base model.
• Base model: NVIDIA's stt_ka_fastconformer_hybrid_large_pc, trained on about 163 h of Common Voice 17 and FLEURS.
• Toolkit: NVIDIA NeMo.
• FastConformer: Rekesh et al., 2023, arXiv:2305.05084.
• Training data:
  • Common Voice (CC0) and FLEURS (CC-BY-4.0) via the Peacockery/georgian-asr-corpus-v0 collection (CC-BY-4.0).
  • Georgian YouTube videos with creator-made subtitles or YouTube's automatic captions, used for research. The audio is not redistributed.
• Noise for augmentation: MUSAN (Snyder, Chen and Povey, 2015, arXiv:1510.08484, CC BY 4.0).