Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection (2024)

Abstract: We propose Lighthouse, a user-friendly library for reproducible video moment retrieval and highlight detection (MR-HD). Although researchers proposed various MR-HD approaches, the research community holds two main issues. The first is a lack of comprehensive and reproducible experiments across various methods, datasets, and video-text features.This is because no unified training and evaluation codebase covers multiple settings. The second is user-unfriendly design. Because previous works use different libraries, researchers set up individual environments. In addition, most works release only the training codes, requiring users to implement the whole inference process of MR-HD. Lighthouse addresses these issues by implementing a unified reproducible codebase that includes six models, three features, and five datasets. In addition, it provides an inference API and web demo to make these methods easily accessible for researchers and developers. Our experiments demonstrate that Lighthouse generally reproduces the reported scores in the reference papers. The code is available at https://github.com/line/lighthouse.

## introduction with the rapid advance of digital platforms, videos become ubiquitous and popular on the web. although they offer rich, informative, and entertaining content, watching entire videos can be timeconsuming. hence, there is a high demand for multimodal tools that enable users to quickly find specific moments within videos and browse through highlights in the moments from natural language queries. the former is called moment retrieval (mr) and the latter is called highlight detection @xcite . given a video and a language query, mr retrieves relevant moments (start and end timestamps), and hd detects highlighted frames within these moments by calculating saliency scores reprelighthouse (proposed library) moment retrieval (mr) -14.000000000000000 -12.000000000000000 -10.000000000000000 -8.000000000000000 -6.000000000000000 -4.000000000000000 -2.000000000000000 0.000000000000000 highlight detection (hd) saliency score most salient frame ✔: reproducible mr-hd 6 mr-hd methods, 3 features, 5 datasets ✔: user-friendly design easy to setup, inference api, web demo relevant moments figure 1: overview of mr-hd and lighthouse. given a video and query, the model predicts relevant moments for mr and saliency scores for hd. lighthouse achieves reproducible mr-hd by supporting multiple settings. in addition, it aims at a user-friendly design with an easyto-setup environment, inference api, and web demo. although mr and hd share common characteristics, such as learning the similarity between input queries and video frames, they were separately treated due to the lack of annotations supporting both tasks @xcite . to address this, @xcite proposed the qvhighlights dataset comprising videos, language queries, and moment/highlight annotations, enabling researchers to tackle both tasks simultaneously. we refer to this unified task of mr and hd as mr-hd to distinguish it from the individual tasks of mr and hd. based on this dataset, various approaches have been proposed to perform mr-hd. note that most methods are applicable for single tasks of either mr or hd as well as mr-hd. despite the rapid development of mr-hd, the research community holds two issues. the first is a lack of comprehensive and reproducible experiments across various methods, datasets, and fea- mr-hd mr hd qvhighlights activitynet captions charades-sta tacos tvsum features api? web demo? moment detr (lei et al., 2021) ✓ c+s qd-detr (moon et al., 2023b) ✓ ✓ c+s eatr (jang et al., 2023) ✓ ✓ ✓ c+s tr-detr (sun et al., 2023) ✓ ✓ c+s uvcom (xiao et al., 2024) ✓ ✓ c+s cg-detr (moon et al., 2023a) ✓ ✓ ✓ ✓ c+s+v+g lighthouse (ours) ✓ ✓ ✓ ✓ ✓ c+s+r+g ✓ ✓ the second is user-unfriendly design. because previous works use different libraries for their method, mr-hd researchers should set up individual environments. in addition, most previous works release only training codes, requiring users to implement the whole inference process of mr-hd and apply it to their videos. this includes frame extraction from videos, video-text feature extraction, and forwarding them into the trained model. implementing all of these steps accurately is challenging for developers who are interested in mr-hd but lack expertise in video-text processing. our goal is to address these issues and foster the mr-hd research community. to this end, we propose lighthouse, a user-friendly library for reproducible mr-hd. lighthouse unifies training and evaluation codes to support six recent mr-hd methods, three features, and five datasets for mr-hd, mr, and hd, resolving the reproducibility issue. while this results in 90 possible configurations (6 methods × 3 features × 5 datasets), the configuration files are written in yaml format, allowing researchers to easily reproduce experiments by specifying the necessary file. our experiments demonstrate that lighthouse mostly reproduces the original experiments in the referenced six papers. in addition, to resolve the user-unfriendliness, lighthouse provides an inference api and web demo. the inference api covers the entire mr-hd process and provides users with easy-to-use code for mr-hd. the web demo, built upon the api, enables users to confirm the results visually. the codes are under the apache 2.0 license. ## highlights of lighthouse reproducible training and evaluation: lighthouse enables researchers to reproduce the training process with a single python command by specifying the configuration files, where hyperparameters are written in yaml format (figure 2 ). the lighthouse users can easily test different hyper-parameters by modifying these files. we release all of the files used or generated during experiments, including video-text features, trained weights, and logs during the training. therefore, to reproduce the experiments, researchers can obtain the same results by downloading the necessary files and running a single python evaluation command with the trained weights. ## architecture of lighthouse figure 4 shows an overview of lighthouse architecture, consisting of four components: datasets, video-text feature extractor, models, and evaluation metrics. ## experiments we perform experiments on mr-hd, mr, and hd tasks individually. we used 1 nvidia a100 gpu (48gb) for all experiments. the hyperparameters used in this paper are the same as in the reference papers. ## conclusion in this paper, we proposed lighthouse, a userfriendly library for reproducible mr-hd. it supports six methods, five datasets, and three features. lighthouse includes the inference api and web demo, enabling users to try mr-hd methods eas-