Automatic Closed Captioning for E stonian Live Broadcasts (2023)

Abstract: This paper describes a speech recognition based closed captioning system for Estonian language, primarily intended for the hard-of-hearing community. The system automatically identifies Estonian speech segments, converts speech to text using Kaldi-based TDNN-F models, and applies punctuation insertion and inverse text normalization. The word error rate of the system is 8.5% for television news programs and 13.4% for talk shows. The system is used by the Estonian Public Television for captioning live native language broadcasts and by the Estonian Parliament for captioning its live video feeds. Qualitative evaluation with the target audience showed that while the existence of closed captioning is crucial, the most important aspects that need to be improved are the ASR quality and better synchronization of the captions with the audio.

## introduction deaf and hard of hearing (dhh) individuals face significant barriers when it comes to accessing live television broadcasts. without closed captioning, they are unable to fully understand and engage with the content being presented. an automatic closed captioning system for live tv broadcasts would help to address this issue and provide dhh individuals with greater access to the same information and entertainment as their hearing counterparts. closed captioning is not only beneficial for dhh individuals, but also for those who may have difficulty hearing the audio on their television due to background noise or other factors. until the beginning of 2022, estonian public television (etv) provided dhh-focused subtitles for some pre-recorded native language programmes, but not for live programmes. from march 2022, captions generated using automatic speech recognition (asr) technology were added to the majority of live native-language programmes, such as news and talk shows. the same technology is used to provide closed captions to the live streams of the estonian parliament sessions (see figure 1 ). this paper describes the system used to generate the subtitles. we provide information on the architecture of the system, its different components, their training data and performance. we also summarise the results of a qualitative evaluation of the live captioning system carried out with the target audience, and discuss how the system could be improved. the reported system is free and available under open-source license foot_0 . ## previous work real-time captioning systems based on speech recognition have been in use for several decades. initially, such systems relied on so-called respeakers -trained professionals who repeat what they hear in the live broadcast in a clear and ar-ticulate manner @xcite @xcite . this allows supervised speaker adaptation of asr acoustic models to be used, resulting in very accurate output. in some use cases, re-speakers also simplify and rephrase the original speech, instantly check and correct the resulting captions, and insert punctuation symbols. in some captioning systems, asr is applied directly to the speech in the live programme, but a human editor is used to correct the asr errors @xcite . however, training respeakers and real-time editors is a long and expensive process. in addition, several re-speakers and/or editors are usually required, as one person cannot usually work continuously for more than two hours without a break. as the quality of asr systems has improved rapidly in recent years, there are more and more cases where an asr system is used directly to produce subtitles without any post-processing. for example, asr-based captions in multiple languages are available in online meeting platforms such as zoom, skype and teams. moreover, youtube offers captioning for live streams, albeit exclusively in english at the time of writing. streaming asr for estonian is available through several commercial vendors; however, recent evaluations have demonstrated that the asr quality provided by these services falls short compared to the models developed at tallinn university of technology @xcite . ## integrations at the time of writing this paper, the closed captioning system is used by the estonian public television (etv) and by the estonian parliament. in etv, the captioning system runs continuously, but the captions are actually delivered only to specific native-language programs (see figure 2 ). the system outputs captions on a word-byword basis to special caption transmission software that formats the words into caption lines and blocks. due to the approximately 5-second delay in the video signal caused by the encoding process, the captions and video are roughly synchronized, but the synchronization is currently not exact: captions tend to be delayed at the beginning of a speaker's turn and arrive relatively faster at the end of a turn. closed captions are transmitted on a dedicated dhh digital closed captioning channel and are not displayed by default. end users can enable closed ## conclusion and future work this paper described an asr-based realtime closed captioning system for estonian broadcasts. the system consists of several open-source components and is currently used for providing captions to estonian public television native language broadcasts and for captioning the live streams of the estonian parliament. qualitative evaluation with the hard-of-hearing focus group showed that providing captions to live tv broadcasts is of high importance to this community. the study emphasized that it is urgent to further improve the asr quality of the closed captions and to improve the synchronization between audio and captions. we are currently working on several aspects of the system that would address some problems highlighted in the qualitative evaluation. first, we are preparing to migrate to end-to-end streaming transducer asr models that would provide improved accuracy with relatively low latency. we are also experimenting with integrating the decoding of punctuation symbols to the main asr model, since currently the separate punctuation symbol insertion model is a source of around two second latency in subtitle presentation. also, we have already implemented modifications to the system that would allow exact synchronization between the audio and displayed captions.