MERLIN : Multimodal Embedding Refinement via LLM -based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline (2024)

Abstract: The rapid expansion of multimedia content has made accurately retrieving relevant videos from large collections increasingly challenging. Recent advancements in text-video retrieval have focused on cross-modal interactions, large-scale foundation model training, and probabilistic modeling, yet often neglect the crucial user perspective, leading to discrepancies between user queries and the content retrieved. To address this, we introduce MERLIN (Multimodal Embedding Refinement via LLM-based Iterative Navigation), a novel, training-free pipeline that leverages Large Language Models (LLMs) for iterative feedback learning. MERLIN refines query embeddings from a user perspective, enhancing alignment between queries and video content through a dynamic question answering process. Experimental results on datasets like MSR-VTT, MSVD, and ActivityNet demonstrate that MERLIN substantially improves Recall@1, outperforming existing systems and confirming the benefits of integrating LLMs into multimodal retrieval systems for more responsive and context-aware multimedia retrieval.

## introduction multimedia content has recently grown rapidly in both quantity and quality, making the task of finding relevant videos from vast collections increasingly challenging. while recent studies on textvideo retrieval have primarily focused on crossmodal interaction @xcite @xcite , largescale foundation model training @xcite @xcite and probabilistic modeling @xcite @xcite , there remains a notable lack of consideration for the discrepancy in text-video retrieval. for instance, as illustrated in figure 1 : an illustration of the discrepancy between the video caption which could be treated as a user query and the video from msr-vtt dataset. blue indicates the details that can be observed statically within the video frame, while red reflects the information that can be obtained temporally across multiple frames. figure 1 , the video caption "a baby playing with a cat's tail" fails to fully capture the additional context of a playful interaction between the baby and the cat. in real-world scenarios, such discrepancies often arise because users tend to submit succinct queries that do not capture the full context of the videos related to their search intent. consequently, this mismatch can lead to unsatisfactory retrieval performance. moreover, neglecting the user perspective makes users refine their natural language query multiple times to fully reflect their search intent. this degrades the quality of user experience and makes it difficult to understand the search intent, leading to a discrepancy between user queries and the information within the retrieved videos. to address this issue, we introduce merlin (multimodal embedding refinement via llmbased iterative navigation), a novel training-free and iterative feedback learning pipeline that leverages the power of large language models (llms) to augment queries based on the user perspective, thereby mitigating the aforementioned discrepancies and significantly improving the textvideo retrieval performance. inspired by human problem-solving and cognitive feedback mechanisms @xcite , we employ an interactive and iterative feedback learning @xcite @xcite @xcite @xcite @xcite consisting of a question answering process that iteratively refines query embeddings for text-video retrieval. moreover, to our best knowledge, merlin presents the first implementation of a retrieval-rerank pipeline in the domain of text-video retrieval, establishing a novel framework that prioritizes user intention and interaction in refining search results. the primary strength of merlin lies in its capability to iteratively adapt and refine query embeddings without necessitating the costly re-training of pre-trained models. as shown in figure 2 , when a user submits a query, merlin generates questions based on the metadata of the retrieved video candidates and presents these questions to the user. by gathering additional information from the user's responses, merlin refines the embeddings to improve retrieval accuracy, thereby helping users find "video in mind" 2 . experimental results on benchmark datasets, including msr-vtt, msvd, and activitynet, demonstrate the superiority of the retrieval performance (e.g. r@k) by showing significant improvement. specifically, merlin boosts text-video retrieval performance (r@1) of google multimodal embedding from 44.00 to 78.00 on msr-vtt, from 52.39 to 77.61 on msvd and from 56.58 to 68.44 on activitynet. the key contributions of our paper are as follows: (1) introduction of merlin, a novel llmbased framework for multimodal embedding refinement that addresses discrepancies between user queries and video content by integrating user perspectives. (2) implementation of an iterative, costeffective method for refining query embeddings using llms, significantly reducing computational demands while improving retrieval accuracy. (3) presentation of the first retrieval-rerank pipeline in text-video retrieval, enhancing interactivity and context-awareness within multimodal systems. (4) experimental results shows that merlin substantially improves r@1 on msr-vtt, msvd 2 "video in mind" refers to the specific video users are looking for or have in mind during the search process. and activitynet, thereby demonstrating notable enhancements in zero-shot text-video retrieval. ## case study the main objective of merlin is to improve the ranking of failure cases where the target video is not among the top-ranked candidates. at the same time, it is important to keep the success case to stay in the top-ranked candidates while merlin proceeds to chat with the user. retrieving the target video among the top-ranked candidates indicates that merlin consistently reflects user intention during the conversation. to qualitatively verify that merlin performs its tasks according to the aforementioned objectives, we reviewed several case studies. we focused on how merlin brings the rank of failure cases. case study for activitynet as shown in figure 4 , the initial ranking of the target video was 224 using a paired query from the dataset. however as merlin augmented the query using the user's response, the rank boosted to 36 → 14 → 4 → 1 as the round proceeded. during the conversation, merlin was able to understand that the user was looking for a video about christmas themes, featuring two people, and involving gift wrapping. it managed to rank the target video on top with the augmented information. case study for msvd as shown in figure 5 , the initial ranking of the target video was 154 using a paired query from the dataset. however as merlin augmented the query using the user's response, the rank boosted to 14 → 1 → 1 → 1 as the round proceeded. during the conversation, merlin was able to understand that the user was looking for a video about the nba all-star game, broadcasted on tnt and the scoreboard telling 74:75. it managed to rank the target video on top with the augmented information at an early round and managed to keep the top rank during multiple rounds. case study for msr-vtt as shown in figure 6 , the initial ranking of the target video was 361 using a paired query from the dataset. however as merlin augmented the query using the user's response, the rank boosted to 197 → 14 → 1 → 1 as the round proceeded. during the conversation, merlin was able to understand the detailed features and gestures of humans featured on "video in mind". it managed to rank the target video on top with the augmented information at an early round and managed to keep the top rank during multiple rounds. ## conclusion in conclusion, the merlin framework addresses a critical gap in the field of text-video retrieval by integrating the often-overlooked user perspective into the retrieval process. this integration is achieved through a novel, training-free pipeline that utilizes llms for iterative feedback learning, allowing for the dynamic refinement of query embeddings based on user interactions. merlin not only aligns more closely with user intent but also enhances the overall search experience by reducing discrepancies between user queries and retrieved video content. the implementation of merlin shows a signifi-cant advancement in multimedia retrieval, introducing the first retrieval-rerank pipeline in this domain. by incorporating iterative feedback mechanisms inspired by human cognitive processes, merlin facilitates a more aligned and context-aware approach to text-video retrieval. our experimental results demonstrate the effectiveness of this approach, with substantial improvements in retrieval performance observed across msr-vtt, msvd, and activitynet datasets.