m TVR : Multilingual Moment Retrieval in Videos (2021)

Abstract: We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21.8K TV show video clips. The dataset is collected by extending the popular TVR dataset (in English) with paired Chinese queries and subtitles. Compared to existing moment retrieval datasets, mTVR is multilingual, larger, and comes with diverse annotations. We further propose mXML, a multilingual moment retrieval model that learns and operates on data from both languages, via encoder parameter sharing and language neighborhood constraints. We demonstrate the effectiveness of mXML on the newly collected mTVR dataset, where mXML outperforms strong monolingual baselines while using fewer parameters. In addition, we also provide detailed dataset analyses and model ablations. Data and code are publicly available at https://github.com/jayleicn/mTVRetrieval

## introduction the number of videos available online is growing at an unprecedented speed. recent work @xcite introduced the video corpus moment retrieval (vcmr) task: given a natural language query, a system needs to retrieve a short moment from a large video corpus. figure 1 shows a vcmr example. compared to the standard text-to-video retrieval task @xcite , it allows more fine-grained momentlevel retrieval, as it requires the system to not only retrieve the most relevant videos, but also localize the most relevant moments inside these videos. various datasets @xcite @xcite have been proposed or adapted for the task. however, they are all created for a single language (english), though the application could be useful for users speaking other languages as well. besides, it is also unclear whether the progress and findings in one language generalizes to another language @xcite . while there are multiple existing multilingual image datasets @xcite @xcite @xcite , the availability of multilingual video datasets @xcite is still limited. therefore, we introduce mtvr, a large-scale, multilingual moment retrieval dataset, with 218k human-annotated natural language queries in two languages, english and chinese. mtvr extends the tvr @xcite dataset by collecting paired chinese queries and chinese subtitle text (see figure 1 ). we choose tvr over other moment retrieval datasets @xcite @xcite because tvr is the largest moment retrieval dataset, and also has the advantage of having dialogues (in the form of subtitle text) as additional context for retrieval, in contrast to pure video context in the other datasets. we further propose mxml, a compact, multilingual model that learns jointly from both english and chinese data for moment retrieval. specifically, on top of the state-of-the-art monolingual moment retrieval model xml @xcite , we enforce encoder parameter sharing @xcite where the queries and subtitles from the two languages are encoded using shared encoders. we also incorporate a language neighborhood constraint @xcite to the output query and subtitle embeddings. it encourages sentences of the same meaning in different languages to lie close to each other in the embedding space. compared to separately trained monolingual models, mxml substantially reduces the total model size while improving retrieval performance (over monolingual models) as we show in section 4. detailed dataset analyses and model ablations are provided. ## dataset the tvr @xcite dataset contains 108,965 high-quality english queries from 21,793 videos from 6 long-running tv shows (provided by tvqa @xcite ). the videos are associated with english dialogues in the form of subtitle text. mtvr extends this dataset with translated dialogues and queries in chinese to support multilingual multimodal research. ## method our multilingual moment retrieval model mxml is built on top of the cross-model moment localization (xml) @xcite ) model, which performs efficient video-level retrieval at its shallow layers and accurate moment-level retrieval at its deep layers. to adapt the monolingual xml model into the multilingual setting in mtvr and improve its efficiency and effectiveness, we apply encoder parameter sharing and neighborhood constraints @xcite which encourages the model to better utilize multilingual data to improve monolingual task performance while maintaining smaller model size. query and context representations. we represent videos using resnet-152 @xcite and i3d @xcite features extracted every 1.5 seconds. we extract language features using pre-trained, then finetuned (on our queries and subtitles) roberta-base @xcite , for english @xcite and chinese (cui et al., 2020), respectively. for queries, we use token-level features. for subtitles, we maxquery (zh) video encoder query encoder sub (en) sub (zh) subtitle encoder neighborhood constraint figure 2: illustration of mxml's encoding process. compared to monolingual models, mxml learns from the two languages simultaneously, and allows them to benefit each other via encoder parameter sharing and neighborhood constraints. we show the detailed encoding process of the model in the appendix (figure 3 ). encoders and parameter sharing. we follow @xcite to use self-encoder as our main component for query and context encoding. a self-encoder consists of a self-attention @xcite layer, a linear layer, and a residual @xcite connection followed by layer normalization @xcite . we use a self-encoder followed by a modular attention @xcite to encode each query into two modularized query vectors q v lang , q s @xmath0, zh}) for video and subtitle retrieval, respectively. for videos, we apply two self-encoders instead of a self-encoder and a cross-encoder as in xml, because we found this modification simplifies the implementation while maintains the performance. we use the outputs from the first and the second self-encoder h v vr,lang , h v mr,@xmath1. similarly, we have h s vr,lang , h s mr,@xmath2. all the self-encoders are shared across languages, e.g., we use the same self-encoder to encode both english and chinese queries, as illustrated in figure 2 . this parameter sharing strategy greatly reduces the model size while maintaining or even improving model performance, as we show in section 4. ## experiments and results we evaluate our proposed mxml model on the newly collected mtvr dataset, and compare it with several existing monolingual baselines. we also provide ablation studies evaluating our model design and the importance of each input modality (videos and subtitles). data splits and evaluation metrics. we follow tvr @xcite to split the data into 80% train, 10% val, 5% test-public and 5% test-private. we report average recall a natural language query, the goal of video corpus moment retrieval is to retrieve relevant moments from a large video corpus. the methods for this task can be grouped into two categories, (i) proposal based approaches (mcn @xcite and cal @xcite ), where they perform video retrieval on the pre-segmented moments from the videos; (ii) retrieval+re-ranking methods (mee @xcite +mcn, mee+cal, mee+excl @xcite and xml @xcite )), where one approach is first used to retrieve a set of videos, then another approach is used to re-rank the moments inside these retrieved videos to get the final moments. our method mxml also belongs to the retrieval+re-ranking category. across all metrics and both languages, we notice retrieval+re-ranking approaches achieve better performance than proposal based approaches, indicating that retrieval+reranking is potentially better suited for the vcmr task. meanwhile, our mxml outperforms the strong baseline xml significantly foot_2 while using few parameters. xml is a monolingual model, where a separate model is trained for each language. in contrast, mxml is multilingual, trained on both languages simultaneously, with parameter sharing and language neighborhood constraints to encourage multilingual learning. mxml prediction examples are provided in the appendix. ## conclusion in this work, we collect mtvr, a new large-scale, multilingual moment retrieval dataset. it contains 218k queries in english and in chinese from 21.8k video clips from 6 tv shows. we also propose a multilingual moment retrieval model mxml as a strong baseline for the mtvr dataset. we show in experiments that mxml outperforms monolingual models while using fewer parameters.