EDIS : Entity-Driven Image Search over Multimodal Web Content (2023)

Abstract: Making image retrieval methods practical for real-world search applications requires significant progress in dataset scales, entity comprehension, and multimodal information fusion. In this work, we introduce Entity-Driven Image Search (EDIS), a challenging dataset for cross-modal image search in the news domain. EDIS consists of 1 million web images from actual search engine results and curated datasets, with each image paired with a textual description. Unlike datasets that assume a small set of single-modality candidates, EDIS reflects real-world web image search scenarios by including a million multimodal image-text pairs as candidates. EDIS encourages the development of retrieval models that simultaneously address cross-modal information fusion and matching. To achieve accurate ranking results, a model must: 1) understand named entities and events from text queries, 2) ground entities onto images or text descriptions, and 3) effectively fuse textual and visual representations. Our experimental results show that EDIS challenges state-of-the-art methods with dense entities and the large-scale candidate set. The ablation study also proves that fusing textual features with visual features is critical in improving retrieval results.

## introduction image search, also known as text-to-image retrieval, is to retrieve matching images from a candidate set given a text query. despite the advancements in large-scale vision-and-language models @xcite @xcite , accurately retrieving images from a large web-scale corpus remains a challenging problem. there remain several critical issues: 1) lack of large-scale datasets: existing image retrieval datasets typically contain 30k-100k images query: gop presidential candidate ron paul makes a point to mitt romney and rick santorum at the new hampshire republican debate. retrieval model 1m image-text pairs rank 1 rank 2 rank 3 rank 4 ## entity-driven image search (edis) we select queries and candidates from humanwritten news articles and scraped web pages with different stages of filtering. then we employ human annotators to label relevance scores. fig. 2 illustrates the overall dataset collection pipeline. ## multi-modal retrieval method given a text query q, a model should be able to encode both images i n and headlines h n to match the query encoding. therefore, the model should include a multi-modal candidate encoder f c and a query encoder f q . within f c , there is a branch for image input f i and a branch for headline f h . we formalize the matching process between a query q m and a candidate c @xmath0, h n ) as: ## conclusion training and evaluating large-scale image retrieval datasets is an inevitable step toward real image search applications. to mitigate the gap between existing datasets and real-world image search challenges, we propose a large-scale dataset edis with a novel retrieval setting and one million candidates. edis queries and candidates are collected from the news domain describing abundant entities and events. edis candidates are image-headline pairs since realistic image search utilizes the surrounding text of an image to facilitate accurate searching results. as a primary step towards handling multimodal candidates in edis, we review two primary fusion approaches and propose a feature-level fusion method to utilize the information from both images and headlines effectively. our experimental results show ample space for improvement on edis. future work should consider more principled solutions involving knowledge graphs, entity the work was funded by the national science foundation award #2048122. the writers' opinions and conclusions in this publication are their own and should not be construed as representing the sponsors' official policy, expressed or inferred.