S lide AVSR : A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition (2024)

Abstract: Audio-visual speech recognition (AVSR) is a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. In AVSR, considerable efforts have been directed at datasets for facial features such as lip-readings, while they often fall short in evaluating the image comprehension capabilities in broader contexts. In this paper, we construct SlideAVSR, an AVSR dataset using scientific paper explanation videos. SlideAVSR provides a new benchmark where models transcribe speech utterances with texts on the slides on the presentation recordings. As technical terminologies that are frequent in paper explanations are notoriously challenging to transcribe without reference texts, our SlideAVSR dataset spotlights a new aspect of AVSR problems. As a simple yet effective baseline, we propose DocWhisper, an AVSR model that can refer to textual information from slides, and confirm its effectiveness on SlideAVSR.

## introduction research on multimodal models capable of handling multiple types of data, such as language, images, videos, and audio simultaneously, has garnered significant attention. an example is audiovisual speech recognition (avsr), a multimodal extension of automatic speech recognition (asr), using video as a complement to audio. most previous studies in avsr have been conducted with the aim of improving accuracy on lip reading datasets @xcite . while models built in these studies @xcite @xcite demonstrate high performance on lip reading data, their applicability to other types of videos remains limited. in this paper, we aim to evaluate the image comprehension capabilities of avsr models across a broader spectrum of visual contents than facial features. to achieve this, we construct slideavsr, an avsr dataset that contains various technical terms that are notoriously challenging to transcribe without referring to textual information on slides. specifically, we collect scientific paper explanation videos from youtube, apply data refinement procedures with several custom filters, and perform data partitioning considering the speakers' accents. furthermore, we propose docwhisper, a simple yet effective avsr baseline that can efficiently refer to the content of slides using optical character recognition (ocr). in experiments utilizing slideavsr, docwhisper demonstrated a performance improvement of up to 14.3% compared to whisper @xcite , which relies solely on audio input. additionally, to address the long-tail problem in ocr results, we introduce fq ranker, which calculates word ranks based on the frequency of word occurrences, and we evaluate its effectiveness integrated with docwhisper. ## related work compared to the efforts that have been made on lip reading datasets @xcite @xcite , avsr datasets in other types of videos remain scarce. visspeech @xcite is constructed from a subset of the instructional video dataset howto100m @xcite where the visual stream and speech audio are semantically related. the audiovisual diarization benchmark in the ego4d challenge @xcite consists of 585 egocentric video clips. in terms of utilizing textual information extracted from videos, slidespeech @xcite builds an avsr dataset on online conference videos enriched with slides. however, the filtering process of slidespeech relies heavily on human annotators. in this work, we exclude videos and utterances that do not correspond to slides by utilizing a vision language model. this approach reduces annotation costs and enhances the purity and quality of our dataset within the slide domain. in the context of extending whisper to an avsr model, @xcite employed clip @xcite to transform the input visual stream into word sequences, which were then utilized as prompts for whisper. they reported that this approach enhances the zero-shot performance on vis-speech. in this study, we employ ocr to create prompts and implement fine-tuning to improve performance rather than using zero-shot prompting. ## slideavsr: dataset construction in this study, we construct slideavsr, an avsr dataset based on scientific paper explanation videos incorporating various technical terms, making accurate transcription difficult without referring to the slides. based on jtubespeech @xcite , a framework for building audio corpora from youtube videos, we implement several custom filters to target videos, thereby applying highprecision data refinement. this section describes the construction flow of slideavsr. figure 1 illustrates the flow. ## conclusion and future work we constructed an avsr dataset, slideavsr, by utilizing paper explanation videos. we proposed docwhisper, which leverages ocr to refer to slide content. we verified the effectiveness of docwhisper on slideavsr and conducted a detailed analysis. additionally, we introduced fq ranker, which calculates word ranks based on word frequency, and evaluated its performance on docwhisper. in the future, we plan to continually refine ocrbased methods and aim to construct an end-to-end avsr model that is not dependent on ocr. furthermore, we intend to build a benchmark that allows a comprehensive evaluation of the image comprehension capabilities of avsr models by incorporating diverse types of videos, such as sports commentary, gaming commentary, cooking videos, and more. ultimately, we aim to construct a foundation model for avsr that exhibits high performance across diverse video inputs.