Cross-Lingual Cross-Modal Consolidation for Effective Multilingual Video Corpus Moment Retrieval (2022)

Abstract: Existing multilingual video corpus moment retrieval (mVCMR) methods are mainly based on a two-stream structure. The visual stream utilizes the visual content in the video to estimate the query-visual similarity, and the subtitle stream exploits the query-subtitle similarity. The final query-video similarity ensembles similarities from two streams. In our work, we pro- pose a simple and effective strategy termed as Cross-lingual Cross-modal Consolidation (C3 ) to improve mVCMR accuracy. We adopt the ensemble similarity as the teacher to guide the training of each stream, leading to a more powerful ensemble similarity. Meanwhile, we use the teacher for a specific language to guide the student for another language to exploit the complementary knowledge across languages. Ex- tensive experiments on mTVR dataset demonstrate the effectiveness of our C3 method.

## introduction video corpus moment retrieval (vcmr) task has been proposed in @xcite , which aims to retrieve a short moment from a large video corpus given a natural language query. recently, @xcite introduces a multilingual video corpus moment retrieval (mvcmr) task. compared with vcmr, mvcmr supports queries in multiple languages. it is more useful in practice, especially in international applications. to facilitate the research in mvcmr, @xcite builds a large-scale mtvr dataset, where queries are in two languages. apart from the video's visual content, the video's textual subtitles are provided as auxiliary information to help the query-to-video retrieval. @xcite proposes an mxml model to generate the query-video similarity for video retrieval and the query-clip similarity for moment localization with a two-stream structure. the subtitle stream captures the similarity between the text query and the video's textual subtitles. in parallel, the visual stream describes the similarity between the text query and the video's visual content. the final text-video similarity is a summation of the similarities from two streams. as the final similarity is obtained from summing up similarities of two streams, straightforwardly, it is reasonable to hypothesize that improving the effectiveness of the similarity from each stream is beneficial to enhancing the performance of the final similarity. meanwhile, since the final similarity fuses the information from two modalities, it is also reasonable to hypothesize that the final similarity is more reliable than the similarity from each stream. based on the above two hypotheses, we propose a simple approach to improve the performance of two-stream architecture for the mvcmr task. to be specific, we use the final similarity fusing two modalities as the teacher and the similarity from each stream as the student. we train the student through the guidance of the teacher's knowledge. meanwhile, to exploit the natural compensation across languages, we devise a student in one language and the teacher in another language. we term our method as cross-lingual cross-modal con-solidation (c 3 ), as visualized in figure 1 . comprehensive experimental results on the mtvr dataset demonstrate the effectiveness of our c 3 method. ## limitations and potential risks although our c 3 has achieved substantial improvement based on mxml on the mtvr dataset, we find that there exists some hyper-parameters (e.g., the τ vr , τ loc ) to tune in c 3 , which may be timeconsuming. besides, we develop the c 3 strategy to improve the performance of mvcmr task, and we have not seen the potential ricks in our paper. ## conclusion in our work, for the multilingual video corpus moment retrieval (mvcmr), we introduce a simple and effective cross-lingual and cross-modal consolidation (i.e., c 3 ) strategy. it enhances the reliability of the similarity score from a single modality through the knowledge distillation from the similarity score with access to the multi-modal information. meanwhile, it exploits the complementary information across languages by cross-lingual knowledge distillation for both video retrieval and moment localization. extensive experimental results demonstrate the effectiveness of our method.