Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR (2024)

Abstract: Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration.

## introduction recent years have witnessed remarkable progress in multimodal large language models (mllms), with gpt-4o achieving impressive performance on tasks involving the integration of information from multiple data modalities such as text, images, speech and video @xcite . however, many real-world applications, particularly in the educational and scientific domains, present significant challenges that push the boundaries of current multimodal capabilities @xcite . one such challenging task is multimodal automatic speech recognition (asr) from knowledgeintensive presentation videos, which involves not only transcribing spoken content but also understanding and integrating visual information from images or videos @xcite . presentation videos, often found in scientific conferences or educational contexts, contain dense, knowledge-rich content spanning various topics and concepts. effective recognition and comprehension of such content are crucial for applications such as automatic transcript generation @xcite , simultaneous translation @xcite in conference venue, lecture summarization @xcite and accessibility tools for researchers and learners @xcite . asr on such videos presents several challenges, including keeping pace with the rapid evolution of technical domain knowledge and handling the variability in speaker accents, proficiency in english, and speech fluency @xcite . mitigating these challenges requires asr models to effectively extract and incorporate information from the speech and presentation slides in the video recordings (see figure 1 ). in this paper, we investigate whether the current multimodal systems can successfully generate transcripts for presentation videos. we propose the task of transcribing scientific conference presentation recordings, namely multimodal scientific asr (ms-asr). unlike conventional multimodal asr tasks that focus on motion-rich scenarios, ms-asr focuses on knowledge-intensive content, requiring accurate transcription of domain-specific terminology using visual information. standard evaluation metrics like word error rate (wer) fail to accurately assess these specialized terms, leading to biased performance evaluations. to address this, we propose a more nuanced evaluation protocol: severity-aware wer (swer) for the task, which calibrates wer based on the severity of asr errors, aligning system performance more closely with human evaluation. solving the proposed ms-asr task requires proficiency in several areas, including speech recognition, visual information extraction and integrating them to post-edit the transcript. to this end, we propose a zero-shot inference framework named scientific vision augmented asr (scivasr) that leverages mllms to collaboratively produce contextually accurate transcripts. we further explore the potential of the current sota mllm, such as gpt-4o, for end-to-end post-editing the transcript. experimental results demonstrate that mllms can effectively improve the quality of the transcript compared to the speech-only asr baseline, where visual information plays a key role for all mllms including gpt-4o to obtain the improvements. we found that speech-only asr suffers from high term errors (298) and 43.04% critical errors, adding visual information not only reduces 152 term errors but also 28% critical errors (section 6). human evaluation confirmed that remaining errors are negligible to affect the understanding on transcripts. the contributions of this paper are threefold. ## scientific vision-augmented asr due to the nature of the long, knowledge-rich, and un-segmented videos, to effectively tackle the task of transcribing presentation videos, a multimodal asr must excel at four subtasks: (i) speech recognition to accurately transcribe the speech audio, (ii) video segmentation to identify slide boundaries, (iii) visual information extraction from each slide, and (iv) integrating the visual information to postedit the initial speech recognition transcript. in this section, we present our approach to address these subtasks, illustrated in figure 2 . ## severity-aware evaluation protocol when evaluating the transcripts, the annotators focus more on the correctness of keywords, terminologies, and numbers, which could impact the meaning of the speech, and care less about minor errors in casing, tense, spelling and grammar. this inspires us to propose the severity-aware evaluation protocol, imitating human evaluations. ## re-evaluating scivasr in this section, we re-evaluate our proposed framework using the fine-grained metrics. ## conclusion this paper introduces the ms-asr task, addressing the challenge of transcribing knowledge-intensive scientific presentations by integrating visual and speech modalities. we identified the inefficacy of wer and proposed the swer metric to better evaluate asr performance by considering the content type and error severity. through our inference framework -scivasr, we demonstrate that state-of-the-art mllms, particularly gpt-4o can significantly enhance transcript accuracy by leveraging visual information, improving by 45% over speech-only baselines. the experiments underscore the importance of multimodal integration for accurate asr in complex, real-world scenarios. our future works focus on expanding benchmark datasets and refining evaluation protocols to further enhance the applicability of the task.