D o SSIER at M ed V id QA 2022: Text-based Approaches to Medical Video Answer Localization Problem (2022)

Abstract: This paper describes our contribution to the Answer Localization track of the MedVidQA 2022 Shared Task. We propose two answer localization approaches that use only textual information extracted from the video. In particular, our approaches exploit the text extracted from the video’s transcripts along with the text displayed in the video’s frames to create a set of features. Having created a set of features that represents a video’s textual information, we employ four different models to measure the similarity between a video’s segment and a corresponding question. Then, we employ two different methods to obtain the start and end times of the identified answer. One of them is based on a random forest regressor, whereas the other one uses an unsupervised peak detection model to detect the answer’s start time. Our findings suggest that for this task, leveraging only text-related features (transmitted either verbally or visually) and using a small amount of training data, lead to significant improvements over the benchmark Video Span Localization model that is based on deep neural networks.

## introduction nowadays, the number of users that turn to the web to satisfy their health-related information needs has grown significantly. however, providing the user with textual answers is insufficient for some particular information needs because, occasionally, these answers are hard to interpret correctly. therefore, it could be useful if the answers are accompanied by a visual aid, i.e., a part of a video (video segment), that presents the answer. such information needs are the scope of this work, where we focus on identifying those video segments that contain the answer to health-related user questions. these user questions are written in natural language and the corresponding answers are part of an instruc- † equal contribution. while the majority of the proposed works in the literature rely on deep neural models to allocate the relevant video segments to the answer @xcite , we explore another alternative. specifically, we aim to study the impact of using only textual features to find the answers in the video, which also implies reducing the requirements for the amount of training data. as input features we use the information transmitted verbally by the presenter in the form of video transcript and also extract the text embedded in the video frames. our main contributions are as follows: the remainder of the paper is organized as follows: section 2 describes in detail the studied task and the related works. section 3 presents our methodology and assumptions. in section 4, we present the experimental setup, our baseline and our submissions. finally, section 5 presents the obtained results, followed by the conclusions drawn from our participation. ## task description & related works this work studies the task of video segment identification for medical videos, introduced as a shared task in @xcite . in particular, given a medical or health-related question written in natural language, the system must provide the user with the video segment that contains the answer. the task focuses on instructional medical videos. a characteristic of these videos is that they deliver the key information to the user both visually and verbally @xcite . in visual question answering, identifying relevant video segments given a user's questions in a natural language is a task that requires processing of both textual and visual signals. as reported by @xcite , a system designed to tackle this problem consists of three components, namely, feature extraction, feature fusion and answer prediction. previously published studies exploit standard embedding models to obtain text features @xcite , and cnn based models to extract image features @xcite . @xcite introduced etm-trans which is a deep transfer learning approach that also addresses the issue of feature fusion. in the field of visual question answering, as reported in @xcite the majority of the proposed techniques employ pre-trained models for image and language encoders. another finding reported in @xcite , is related to the fact that only a small portion of the proposed approaches investigate their generality and interpretability. the introduction of large-scale multimodal datasets covering both language and vision enabled the development of efficient deep neural network techniques that bridge the gap between language, and visual understanding @xcite @xcite . while the majority of the proposed methods in the literature are based on deep neural models, our approach leverages only the textual information that can be extracted from a video without the need for extensive training. it estimates the relevance of each video segment to a given question, and ultimately it returns the starting time and duration of the answer. ## methodology our methodology exploits the characteristics of the videos in the current task. specifically, we extract a video's transcripts. the transcripts contain the text that one can hear during the video, its start time and its duration, and correspond to a specific video segment. moreover, we enrich this information by adding the text presented in video segments (video frames), for instance text that contains the topic, the steps of an exercise, among others. then, given a question, we estimate distinct similarity scores for every video segment using four different models that will be described in section 3.2. at this point, two distinct approaches can be followed to identify the answer's starting time and duration. the first one employs a multi-output regression model that inputs the similarity scores for every video segment and outputs the starting time and duration of the answer. for the second approach we set the starting time equal to the starting time of the segment that has the highest similarity score, obtained by aggregating the similarity scores obtained by four models, and hard-set the answer's duration based on the training data. the following sections present the hypothesis and assumptions behind each step of our methodology. ## experiment setup and submissions this section describes the dataset used to train and evaluate our approaches and our methodology for extracting text from videos. in addition, we provide the details of our submissions. our code is publicly available 3 . ## evaluation and results in this section we present the results on both the evaluation and test datasets. ## conclusion this work investigates two different approaches for detecting answer timestamps from medical instructional videos in the context of the medvidqa 2022 mval shared task (task 2). our approaches rely only on the text extracted from the videos, either as transcripts or as the text displayed in the video's frames. after extracting the text corresponding to every video segment, we estimate its similarity to the question using four different models. we employ two different strategies to map the questiontext similarity to the answer timestamp, i.e. multioutput regression model based on random forest and a peak detection model. our best performing peak detection model achieves 40.52 @xmath0.3 on medvidqa 2022 shared task and outperforms the vsl benchmark model on the medvidqa test dataset. we also show a positive impact of using multiple video-totext conversion methods on the overall quality of models. our feature extraction methods could easily extend the set of features used by end-to-end deep learning models. further analysis is needed to assess other ways of processing the text-question similarity importance for obtaining more accurate predictions.