Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings (2025)

Abstract: This work stems from an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within encoded semantics, resulting in failed retrieval even in simple cases. To examine such behaviors, we first introduce a new evaluation dataset, CapRetrieval, in which passages are image captions and queries are phrases targeting entity or event concepts in diverse forms. Zero-shot evaluation suggests that encoders often struggle with these fine-grained matching, regardless of training sources or model size. Aiming for enhancement, we proceed to finetune encoders with our proposed data generation strategies, enabling a small 0.1B encoder to outperform the state-of-the-art 7B model. Within this process, we further uncover the granularity dilemma, a challenge for embeddings to capture fine-grained salience while aligning with overall semantics. Our dataset, code and models in this work are publicly released at https://github.com/lxucs/CapRetrieval.

## introduction dense passage retrieval (dpr) is a crucial component in searching, characterized by the classic dual-encoder paradigm @xcite : queries and candidate passages are independently encoded to embeddings that capture their semantics, then top candidates are retrieved based on their embedding similarity. for large language models (llms) especially, dpr serves a pivotal role in retrieval augmented generation (rag) @xcite . the training of such encoders has thereby become an active direction in representation learning @xcite . however, despite their growing capabilities in encoding complex queries and documents, embedding matching can still fail on simple cases, and yet unable to fully supersede the conventional lexicalbased methods such as bm25 @xcite . consequently, retrieval systems in practice usually take a hybrid approach for optimized performance @xcite @xcite . in this work, we probe into such limitation of dpr encoders foot_0 , and highlight that even for simple passages, state-of-the-art encoders still often lack a fine-grained view of concepts and the integration of world knowledge, resulting in failed retrieval on entities or events. to remedy this issue, we delve into data generation strategies towards enhanced encoder training, and we identify a granularity dilemma within this process, posing a challenge for general training data composition. illustrated by to facilitate the analysis of such behaviors and potential improvements, we first construct a new evaluation dataset in both chinese and english, tailored to a practical image search scenario, where the candidate passages are image captions, and queries are short phrases of entities or events reflected in captions. the dataset overall comprises seemingly straightforward queries and captions, dubbed capretrieval, and features two unique aspects. first, it naturally requires fine-grained semantic encoding of passages, inquiring concepts of entities or events reflected in the captions. second, distinguished from prior caption-related datasets, e.g. ecom/video retrieval @xcite , whose labels are collected from real-world user clicks and thus may contain many false negatives, capretrieval provides explicit annotations for each query-passage pair, enabling more reliable and indepth analysis by near 1.3 million pair labels. upon our zero-shot evaluation on capretrieval using open-source encoders of various sizes, all of them are revealed certain flaws performing these fine-grained matching, underscoring the current limitations of embeddings. we then proceed to finetune the encoder with our proposed data generation strategies in sec. 4, where we leverage llms for automatic keyword generation as training data to enforce the semantic encoding of precise salience. empirical results show that our finetuned 0.1b encoder outperforms all baselines on capretrieval, surpassing 7b encoders, validating that the data generation strategies are effective strengthening a more subtle semantic matching. nevertheless, we also identify a dilemma upon further analysis: with the introduced keywords added into training, the model improves on granular saliency and concepts, but may lose grasp on the overall critical semantics in a bigger picture, resulting in degradation on certain scenarios. we describe it as the granularity dilemma to handle both coarse and fine granularity (sec. 4), showing that it still remains a challenge to obtain general embeddings of a full semantic view. ## dataset: capretrieval our dataset capretrieval is constructed for typical retrieval evaluation towards a practical image search scenario, composed of three following parts. passages a pool of image captions are prepared as candidate passages, derived from three steps. (1) a set of images is collected from personal phone albums for diverse coverage of different types, including daily photographs, web pictures, screenshots of various apps or articles. (2) each image is transcribed into a short caption by prompting gpt-4o. (3) each caption is manually reviewed and anonymized to ensure privacy compliance. queries user queries are short phrases to search relevant images, collected by two rounds. the first round of queries are brainstormed by researchers of this work, being usual entities or events that align with real-world usage scenarios. for the second round, a combination of queries in the first round is performed, then manually revised to form more complex queries. among these queries, the easiest records min max avg passages 3024 8 75 30.90 queries 404 1 16 3.78 labels following the convention of typical retrieval datasets in mteb, the label is a relevance score of [0, 1, 2] for a pair of query and passage, signaling no/weak/strong relevance. unlike prior webcollected datasets with large-scale passage pools, our controlled setting enables exhaustive annotations for each query-passage pair, which minimizes false negatives for a more reliable evaluation. note that the annotation and the entire retrieval process only concern textual captions, without involving vision features. the accuracy of image captioning thus does not affect the evaluation. statistics capretrieval was originally constructed in chinese. we further employ gpt-4o to translate all queries and passages into english to provide the corresponding english version, of which the translation quality is verified by human experts. the remaining sections in this paper reports on the original capretrieval without further specification. ## zero-shot evaluation to examine the performance of off-the-shelf encoders on capretrieval, we evaluate five popular open-source encoder series for chinese on huggingface, offering various model sizes: ## encoder training the results of zero-shot evaluation on capretrieval calls for more expressive embeddings that capture fine-grained concepts and world knowledge integration. we proceed with further examination by finetuning encoders with training data strategies. ## conclusion we focus on the embedding granularity that stems from the observation, where text encoders can fail to recognize entities or events of even simple cases. a new evaluation set is introduced to probe the limitation of embeddings, and zero-shot experiments reveal the room for improvement on these fine-grained matching. we further investigate data generation strategies for encoder training, and identify the granularity dilemma that calls for future efforts towards more expressive embeddings.