VPAI _ L ab at M ed V id QA 2022: A Two-Stage Cross-modal Fusion Method for Medical Instructional Video Classification (2022)

Abstract: This paper introduces the approach of VPAI_Lab team’s experiments on BioNLP 2022 shared task 1 Medical Video Classification (MedVidCL). Given an input video, the MedVidCL task aims to correctly classify it into one of three following categories: Medical Instructional, Medical Non-instructional, and Non-medical. Inspired by its dataset construction process, we divide the classification process into two stages. The first stage is to classify videos into medical videos and non-medical videos. In the second stage, for those samples classified as medical videos, we further classify them into instructional videos and non-instructional videos. In addition, we also propose the cross-modal fusion method to solve the video classification, such as fusing the text features (question and subtitles) from the pre-training language models and visual features from image frames. Specifically, we use textual information to concatenate and query the visual information for obtaining better feature representation. Extensive experiments show that the proposed method significantly outperforms the official baseline method by 15.4% in the F1 score, which shows its effectiveness. Finally, the online results show that our method ranks the Top-1 on the online unseen test set. All the experimental codes are open-sourced at https://github.com/Lireanstar/MedVidCL .

## introduction one of the key goals of artificial intelligence (ai) is to develop a multimodal system that uses natural language queries to facilitate communication with the visual world (i.e., images, videos) @xcite . in recent years, the gap between language and visual understanding has narrowed @xcite due to the development of pre-trained models @xcite and the introduction of large-scale language-vision datasets ( @xcite . improvements have been made in numerous vision-and-language tasks, such as visual classification @xcite , video question answering @xcite ) and natural language video localization @xcite @xcite . ## proposed approach in this section, we will elaborate on the proposed approach for the medical video classification (med-vidcl) track. as the pre-training language method can enhance the performance of semantic representation queried by the text subtitles @xcite , we design the two-stages cross-modal fusion method, which is described in turn as follows. ## results and discussions in this section, we introduce the experimental results of the monomodal in language and the multimodal in language-version where the further discussions and official results are also presented. ## conclusion this paper introduces our approach to solving the medical video classification (medvidcl) task in bionlp of the acl2022. specifically, we propose the two-stage method with cross-modal fusion using the pre-trained language model. we report the performance of our model compared on the test set in monomodal and multimodal settings. the experimental results show that our method obtains the best performance on the seen test set and unseen official test set, which proves that our method is effective. also, experimental results show that language understanding is better than multimodal video understanding. in the future, we will further study how to design a more efficient structure to jointly learn the representation in visual language for better multimodal video understanding.