M ^3 AV : A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset (2024)

Abstract: Publishing open-source academic video recordings is an emergent and prevalent approach to sharing knowledge online. Such videos carry rich multimodal information including speech, the facial and body movements of the speakers, as well as the texts and pictures in the slides and possibly even the papers. Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations. In this paper, we propose a novel multimodal, multigenre, and multipurpose audio-visual academic lecture dataset (M ^3 AV), which has almost 367 hours of videos from five sources covering computer science, mathematics, and medical and biology topics. With high-quality human annotations of the slide text and spoken words, in particular high-valued name entities, the dataset can be used for multiple audio-visual recognition and understanding tasks. Evaluations performed on contextual speech recognition, speech synthesis, and slide and script generation tasks demonstrate that the diversity of M ^3 AV makes it a challenging dataset.

## introduction the rapid progress of technology has brought numerous academic presentations and talks available on the web with open access @xcite @xcite . these resources are particularly helpful to researchers as they contain rich specialised knowledge in auditory and visual modalities. with the development of neural-based ai systems, it is desired that ai could comprehend and process these multimodal information resources to assist scientists in accelerating their research. abilities, including transcribing and generating speech for a presentation with the aid of slides, as well as presentation slide or script generation, are particularly useful for researchers to conduct investigations and prepare presentations. as a natural data source containing multimodal content and knowledge, academic lectures have been explored to construct a series of datasets. some datasets focus on the evaluation of the model's ability to recognize multimodal content @xcite , others focus on promoting models to understand multimodal knowledge in academic videos @xcite @xcite @xcite . however, none of them has yet simultaneously paid attention to the model's ability to recognize multimodal content and understand rich academic knowledge, which is partly due to their respective lack of adequate manual labelling. the homogeneity of holding two characteristics is critical to the implementation of end-to-end academic recognition and understanding systems. to fill the gap, we release the multimodal, multigenre, and multipurpose audio-visual academic lecture dataset (m 3 av) as shown in figure 1 . the dataset contains selected academic lectures and presentation videos from multiple fields, including computer science, biomedical science, and mathematics. each video is endowed with highly qualified speech transcriptions and ocr labels for both printed and handwritten characters, including complex mathematical formulae. in addition, academic papers for some videos are also provided to supplement knowledge more extensively. together with the dataset, three benchmark tasks are also proposed that reflect the perception and understanding of the multimodal information in those videos. this includes automatic speech recognition (asr) with a particular focus on contextual asr (casr), spontaneous text-to-speech (tts) synthesis, and slide and script generation (ssg). representative baseline models are used to bench- ## related work the datasets based on academic lectures mainly fall into two categories: measuring the model's ability to recognize multimodal content and evaluating its capacity to capture academic knowledge. in ## conclusion we release the multimodal, multigenre, and multipurpose audio-visual dataset with academic lectures (m 3 av) covering a range of academic fields. this dataset contains manually annotated speech transcriptions, slide text, and additional extracted papers, providing a basis for evaluating ai models for recognizing multimodal content and understanding academic knowledge. we detail the cre-ation pipeline and conduct various analyses of the dataset. furthermore, we build benchmarks and conduct experiments around the dataset. we find there is still large room for existing models to improve perceptions and understanding of academic lecture videos.