A ssist SR : Task-oriented Video Segment Retrieval for Personal AI Assistant (2022)

Abstract: It is still a pipe dream that personal AI assistants on the phone and AR glasses can assist our daily life in addressing our questions like “how to adjust the date for this watch?” and “how to set its heating duration? (while pointing at an oven)”. The queries used in conventional tasks (i.e. Video Question Answering, Video Retrieval, Moment Localization) are often factoid and based on pure text. In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR). Each of our questions is an image-box-text query that focuses on affordance of items in our daily life and expects relevant answer segments to be retrieved from a corpus of instructional video-transcript segments. To support the study of this TQVSR task, we construct a new dataset called AssistSR. We design novel guidelines to create high-quality samples. This dataset contains 3.2k multimodal questions on 1.6k video segments from instructional videos on diverse daily-used items. To address TQVSR, we develop a simple yet effective model called Dual Multimodal Encoders (DME) that significantly outperforms several baseline methods while still having large room for improvement in the future. Moreover, we present detailed ablation analyses. Code and data are available at https://github.com/StanLei52/TQVSR.

## introduction as shown in fig. 1a , a user is looking at his watch and wondering "how to adjust the date for this watch". it would be great if our phone or glasses can be powered by an intelligent agent, i.e. ai assistant, which perceives exactly what the user sees and can find out the answer in the instructional video of this watch, either provided by the watch's seller or posted on youtube. however, such a video is usually quite long and thus time-consuming for users to watch, let alone there are a lot of online tutorial † corresponding author. ## method we introduce dual multimodal encoders (dme) for tqvsr. fig. 7 gives an overview of our model. formally, we define the inputs into the model as: a candidate video segment associated with transcripts and a user question composed of an image with or without referring regions, and a textual query. model architecture. dme is composed of two multimodal encoders based on the transformer network @xcite . in tqvsr, both user question and a context video segment are multimodal, i.e. both of them are composed of vision input and language input, requiring a model for learning joint contextualized representations. inspired by the recent success of the vision and language model @xcite @xcite , we follow @xcite to use the self-attention mechanism within the transformer to implicitly align elements of the input text and regions in the question and model the contextual information of the input video segment. input representation. we extract visual features for video appearance, query image and region with resnet-50 backbone @xcite with weights from grid-feat @xcite , which is trained on visual genome @xcite . we extract contextualized text features using a 6layer pretrained distilbert @xcite . both visual features and textual features are projected into the same dimension, and added with token type embedding and positional embedding. we concatenate the text feature and visual feature to [cls] create a single sequence embedding and a learnable [cls] token (devlin et al., 2019) is concatenated to the beginning of the input feature, which is used to produce the final output representation of each transformer-based encoder. training and inference. during training, one relevant segment is randomly sampled to be paired with a question. for training, we use loss function as in @xcite for segment retrieval setting, where matching question-segment pairs in a batch are treated as positives, and all other pairwise combinations in the same batch are treated as negatives. we maximize the score between positive pairs and minimize the score between negative pairs. at inference time, the dme model requires only the dot product between the multimodal question embedding and candidate video segment embeddings. this retrieval inference is of trivial cost since questions and video segments are indexable and therefore it is scalable to large scale retrieval. more details are in appendix. ## experiment 6.1 baseline to explore the performance of other video-related tasks on tqvsr, we evaluate the following methods and modify the original implementation to fit in our inputs if necessary. more details could be found in appendix. ## conclusion and future work this paper presents a new task, task-oriented question-driven video segment retrieval (tqvsr), requiring intelligent systems to address the given task-oriented question by retrieving relevant segments from a video corpus. to support the evaluation of such a task, we construct a dataset, assistsr, which contains 3.2k questions on 1.6k video segments from instructional videos on diverse daily-used items. further, we provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for tqvsr. the experimental results show that tqvsr is still a formidable challenge for all current methods. we hope this benchmark can help drive the vision and language models to be deployed in real-world applications and boost the development of personal ai assistants.