From Conversational Speech to Readable Text: Post-Processing Noisy Transcripts in a Low-Resource Setting (2025)

Abstract: We present ongoing research on automatic post-processing approaches to enhance the readability of noisy speech transcripts in low-resource languages, with a focus on conversational speech in Latvian. We compare transformer-based sequence-labeling models and large language models (LLMs) for the standard punctuation and capitalization restoration task, while also considering automatic correction of mispronounced words and disfluency, and partial inverse text normalization. Our results show that very small LLMs (approx. 2B parameters), fine-tuned on a modest text corpus, can achieve near state-of-the-art performance, rivaling orders of magnitude larger LLMs. Additionally, we demonstrate that a fine-tuned Whisper model, leveraging acoustic cues, outperforms text-only systems on challenging conversational data, even for a low-resource language. Error analysis reveals recurring pitfalls in sentence boundary determination and disfluency handling, emphasizing the importance of consistent annotation and domain adaptation for robust post-processing. Our findings highlight the feasibility of developing efficient post-processing solutions that significantly refine ASR output in low-resource settings, while opening new possibilities for editing and formatting speech transcripts beyond mere restoration of punctuation and capitalization.

## introduction automatic punctuation and capitalization restoration has been widely studied as a post-processing step for automatic speech recognition (asr) systems, aiming to improve transcript readability and facilitating downstream nlp tasks such as machine translation, named entity recognition, etc. early methods leveraged statistical approaches, such as n-gram language modeling and prosodic cues @xcite , as well as sequence labeling techniques like conditional random fields (crfs) @xcite and maximum entropy models. with the advent of deep learning, recurrent neural networks (rnns) and long short-term memory (lstm) models proved to be more efficient in modeling sequential dependencies @xcite . bidirectional rnns and transformer-based architectures further enhanced accuracy by using richer contextual representations @xcite . recent work has demonstrated that transformerbased models outperform previous neural approaches. bert-based models, such as roberta and electra, have achieved state-of-the-art results on punctuation restoration by leveraging largescale pretraining @xcite . other studies have explored multilingual transformer models such as xlm-roberta @xcite to address punctuation restoration across multiple languages. end-to-end asr models, such as openai's whisper @xcite , directly generate transcriptions with punctuation and capitalization. whisper is trained on large-scale weakly supervised data, allowing it to outperform conventional asr models that require separate punctuation restoration modules. recent advances in large-scale auto-regressive large language models (llms), such as @xcite , have introduced new paradigms for punctuation restoration. unlike conventional sequence labeling approaches, gpt-style models perform text infilling and editing, enabling them to restore punctuation in a generative manner. recent developments in open-source multilingual llms have led to the creation of smaller models that effectively support low-resource languages @xcite . developing robust punctuation restoration models relies on sufficiently large and representative annotated corpora. europarl @xcite and ted-lium @xcite have been widely used, but they often lack domain-specific noise typical of real-world asr. @xcite demonstrated that domain-adaptive fine-tuning with ngram similarity-based data sampling can improve model robustness. data augmentation methods that simulate asr errors have also been shown to yield significant performance gains @xcite . for latvian (approx. 1.5m native speakers), relevant work on punctuation restoration includes @xcite , which focus on bidirectional models and sequence labeling for punctuation and capitalization. one publicly available resource is a proprietary on-line service that allows users to correct the punctuation and formatting of a text, where the underlying model, likely an encoder-decoder, is trained on academic texts foot_0 . another publicly available resource is an open-source punctuation model based on xlm-roberta foot_1 @xcite , trained on europarl data. the best available end-to-end latvian asr models that include text formatting are whisper-large-v3 and whisper-large-v3-lv, the latter being fine-tuned on the dataset described in section 2 as well as on the common voice 19.0 dataset foot_2 @xcite . our contributions in this study are as follows: ## dataset we use the late-media corpus foot_3 @xcite , which comprises approximately 70 hours of conversational latvian speech from broadcast recordings, sourced from public media. the data includes both spontaneous and prepared speech (but not read speech) from more than 250 speakers, offering a diverse range of speaking styles and topics. transcriptions are provided in standard latvian orthography, with additional punctuation and grammar rules applied. when necessary, annotations in square brackets capture non-standard pronunciation (e.g., "lasām [lasam]") and foreign words (e.g., "rail [reil] baltica [boltik]"). the corpus also documents the reading of numbers, accounting for syntactic agreement in context (e.g., nominative vs. dative forms). this rich annotation scheme ensures that spontaneous variations -such as word repetitions, truncated words, and different realizations of abbreviations -are properly represented. to simplify the punctuation restoration task, we unify several less frequent or inconsistently annotated marks by replacing them with periods. specifically, we map exclamation marks, ellipses, and em dashes to periods. we also ignore seldom-used marks such as colons and semicolons, which tend to be subjectively annotated. these steps reduce annotation noise and help stabilize model performance in subsequent training and evaluation. the dataset statistics, including the distribution of punctuation, capitalization types, average sentence length, and correction annotations, are presented in ## experimental setup in our experiments, we address the following key research questions: we evaluate two distinct scenarios: formatting asr-generated transcripts and formatting manually transcribed reference text. this setup allows us to assess how models handle noisy asr outputs and whether they can refine reference transcripts without unnecessary modifications. the evaluation of asr-generated transcripts is conducted on the outputs of whisper-large-v3-lv, currently the strongest open-source latvian asr model. we use publicly available salieckomatus.lv and xlm-roberta @xcite as baselines. additionally, we evaluate the performance of whisperlarge-v3 and whisper-large-v3-lv. performance is measured using f1-score (f1) for punctuation restoration and capitalization. to ensure that models do not introduce unnecessary modifications, we also compute the word error rate (wer) on the normalized formatted transcript. a heuristic fuzzy alignment method is used to align incorrectly recognized words and words that differ in spoken and written forms, such as number expressions, acronyms, and abbreviations. for llms, we employ the following taskspecific prompt: for fine-tuned models, we use a shorter prompt, observing no noticeable drop in performance: models are fine-tuned exclusively on the training split, without incorporating any external data. the fine-tuning uses a linearly decreasing learning rate of 2e-5, a warm-up ratio of 0.1, a batch size of 32, and runs for 3 epochs. ## results the results of our experiments are presented in fine-tuned (ft) models show significant gains in consistency, with gpt-4o ft achieving the highest overall performance (f1 scores of 81.5 for punctuation and 84.4 for capitalization). notably, smaller fine-tuned models (e.g., gemma-2b, eurollm-1.7b) perform at levels comparable to gpt-4o mini, suggesting that the model size alone does not dictate effectiveness for this task. smaller generative models like gpt-4o mini can introduce more pronounced substitutions (e.g., "bračka" (brother) → "brāk , a" (defect)) as well as occasionally produce non-existent words (e.g., "paliec" (stay) → "palik , " (∅), "filmēs" (will shoot) → "filmes" (∅)). overall, fine-tuning reduces unintended text changes by an order of magnitude for all model sizes. while fine-tuned models in the two billion parameter range rarely alter transcripts, the errors they produce typically manifest as ungrammatical forms rather than semantic substitutions. the whisper model fine-tuned for latvian (i.e., whisper-large-v3-lv) achieves wer of 12.7, significantly outperforming the base whisper large-v3 model while maintaining strong punctuation and capitalization scores. models generally perform better on reference transcripts than on asr outputs, which is expected since asr-generated text contains recognition errors that interfere with punctuation and capitalization. similarly, fine-tuned llms outperform their non-fine-tuned counterparts when applied to asr outputs. we manually annotated 100 samples to analyze errors made by the various models. in the cases of mismatched predictions, we categorized errors as follows: for 21 of the cases, we had to listen to the audio to apply a correct markup, highlighting the importance of audio features, for example, "labi. sakratīts ir." ('well. shaken [it] is.') vs. "labi sakratīts ir." @xcite is.'). this also explains the better question mark performance for whisper-large-v3-lv without any extra processing. we further evaluated model performance in correcting mispronounced words: by using annotated mispronunciations, number expressions, and generally acceptable written forms annotated in the dataset. approximately 50% of these cases were correctly replaced by llms, suggesting a potential for these models to learn error correction and inverse text normalization tasks for the low-resource latvian from relatively small datasets. however, further investigation is needed, since the current test set is too small for a reliable evaluation. we have also evaluated a broader set of punctuation marks. however, because of their low frequency beyond commas, periods, and question marks, these results can currently only be considered preliminary and are not yet reliable. moreover, their usage in conversational asr transcripts is often subjective, justified by increased inter-annotator disagreement. ## conclusion and further work end-to-end asr systems, such as whisper finetuned for latvian (using a relatively small amount of data), already provide reasonably well-formatted transcripts for general-domain speech by leveraging acoustic features that are unavailable in textonly approaches. however, even without acoustic cues, formatting performance can be improved with llms using a prompt-based approach. further task-specific fine-tuning yields the best and most stable results, and it is feasible even with smaller llms in the 2b parameter range on a small dataset. larger models often provide higher accuracy but come with increased computational costs and deployment complexity. audio features (pauses, intonation) remain a crucial signal for punctuation restoration. sentence boundaries in speech are often ambiguous, with multiple valid interpretations, and better annotation guidelines could improve consistency. one major challenge in training asr models for latvian and other low-resource languages is the lack of datasets that include both conversational speech and formatted transcriptions. llms enable transcript transformations such as inverse text normalization and error correction by leveraging their built-in language knowledge, even when trained on relatively small datasets. thus, fine-tuned llms can expedite the addition of such formatting to existing orthographically transcribed datasets, for instance, the late-conversational speech corpus @xcite which comprises 35 hours of informal conversations in latvian -this is currently a work in progress, to be followed by human verification and evaluation.