Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLM s (2025)

Abstract: This paper presents an approach for generating high-quality, same-language subtitles for Estonian TV content. We finetune the Whisper model on human-generated Estonian subtitles and enhance it with iterative pseudo-labeling and large language model (LLM) based post-editing. Our experiments demonstrate notable subtitle quality improvement through pseudo-labeling with an unlabeled dataset. We find that applying LLM-based editing at test time enhances subtitle accuracy, while its use during training does not yield further gains. This approach holds promise for creating subtitle quality close to human standard and could be extended to real-time applications.

## introduction same-language subtitles for video material, like tv talk shows, investigative pieces, and educational content serve as a valuable resource for the deaf and hard-of-hearing community, non-native speakers and native speakers alike. for instance, recent studies @xcite have revealed that 50% of americans and 85% of the netflix users overall frequently watch tv and streaming video content with subtitles. studies show that subtitles can enhance understanding and memory retention. a lot of viewers choose to enjoy their content quietly at home, keeping subtitles on to avoid disturbing their roommates or family. subtitles differ from verbatim (word-by-word) transcripts in many aspects. subtitles represent typically a condensed version of the speech, designed to convey the essential meaning without capturing every word. they may omit filler words, repetitions, and non-verbal sounds, and may rewrite phrases, focusing on clarity and readability for viewers. since subtitles are displayed on-screen during playback, they are formatted to fit within a limited time frame and limited line length, ensuring they are easy to read while the viewer is watching. this paper outlines the development of an accurate offline same-language subtitle generation model for estonian tv content. using existing human-created subtitles, we fine-tune whisper @xcite and explore further improvements with semi-supervised learning and llm-based post-editing techniques. our findings demonstrate that whisper can be trained to closely replicate human subtitling style, creating well-segmented and often rephrased subtitles. additionally, we find that iterative pseudo-labeling of a large unlabeled dataset improves subtitle quality across all metrics. while a state-of-the-art commercial llm (openai gpt-4o foot_0 ) can enhance subtitle quality during test time, it's use at training time to improve pseudo-labeled subtitles through post-editing is not effective. ## related work both iterative pseudo-labeling and llm-based post-editing have been an active area of research in the context of verbatim automatic speech recognition (asr). pseudo-labeling based semisupervised learning in asr has been studied since at least @xcite and has been later investigated in several works, e.g. by @xcite ; @xcite . to the best of our knowledge, ma et al. (2023) was the first to show the potential of zero-shot and few-shot llm-based asr error correction. this approach has been later extended to take into account uncertainty estimation of asr outputs @xcite and retrieval-augmented generation for correcting speech recognition entity name errors @xcite . @xcite showed that llm-based error correction and data filtering can be also used for refining the pseudo-label transcripts during semisupervised learning. this work is similar to ours, however, it is applied in the context of a codeswitched mandarin-english asr task. ## method our method for developing an automated subtitle generation system involves several steps: training with supervised data, using iterative pseudolabeling, and applying llm-based error correction. we start by training the whisper large-v3 model @xcite on a supervised dataset. this dataset consists of audio recordings paired with their subtitles. next, we use an unsupervised dataset to perform two iterations of pseudo-labeling. in this step, we generate pseudo-labels using the last trained model and combine them with the original supervised dataset, followed by training a new model on this data. we also apply llm-based post-editing of the generated subtitles, by instructing the llm to fix the mistakes in the subtitles and giving it a segment of generated subtitle file. we experiment with applying this llm-based post-editing in two distinct phases: at test time (i.e., to generated subtitles of the test data) and during training time (i.e., to generated subtitles of the unsupervised dataset). ## conclusion in this work, we presented an approach to automated subtitle generation, leveraging the multilingual whisper model, semi-supervised learning, and llm-based post-editing. by utilizing supervised and unsupervised datasets, we demonstrated that iterative pseudo-labeling can indeed improve the quality of subtitles. our results show that applying an llm during test time has a more significant impact on the results across all the key metrics than during training time. future work will focus on adapting our approach to real-time scenarios.