M ulti C on IR : Towards Multi-Condition Information Retrieval (2025)

Abstract: Multi-condition information retrieval (IR) presents a significant, yet underexplored challenge for existing systems. This paper introduces MultiConIR, the first benchmark specifically designed to evaluate retrieval and reranking models under nuanced multi-condition query scenarios across five diverse domains. We systematically assess model capabilities through three critical tasks: complexity robustness, relevance monotonicity, and query format sensitivity. Our extensive experiments on 15 models reveal a critical vulnerability: most retrievers and rerankers exhibit severe performance degradation as query complexity increases. Key deficiencies include widespread failure to maintain relevance monotonicity, and high sensitivity to query style and condition placement. The superior performance GPT-4o reveals the performance gap between IR systems and advanced LLM for handling sophisticated natural language queries. Furthermore, this work delves into the factors contributing to reranker performance deterioration and examines how condition positioning within queries affects similarity assessment, providing crucial insights for advancing IR systems towards complex search scenarios.

## introduction information retrieval (ir) is critical for helping users find relevant information across various domains. traditionally, ir systems retrieve documents by matching queries based on lexical similarity, such as bm25 @xcite , or semantic similarity using dense vector representations @xcite . though highly effective for queries with straightforward query-document * equal contribution † corresponding author relationships @xcite , they often fail to fully capture nuanced intent as user needs become more complex @xcite @xcite . a significant challenge arises when users specify multiple requirements simultaneously, as illustrated in fig. 1 . whether searching for a movie with specific attributes or selecting a product that meets various criteria, multi-condition search has become an integral part of modern informationseeking behavior. traditional ir systems handle such scenarios using structured filtering, such as sql-based queries that retrieve information from backend databases based on predefined conditions. however, this approach is inherently rigid and limited, as it relies on explicitly defined attributes and lacks the flexibility to accommodate evolving or diverse user preferences. as a result, it struggles to support nuanced or semantic-level queries that go beyond structured data filtering. existing benchmarks do not offer a structured framework for evaluating multi-condition retrieval along these dimensions. to address this gap, we introduce multiconir-a benchmark designed to comprehensively evaluate multi-condition retrieval systems. through systematic experiments on 15 state-of-the-art models (including dense retrievers, cross-encoders, and llm-based agents), we uncover several critical insights: by quantifying these gaps, our work reveals key deficiencies in the ability of current ir systems to understand multi-condition intent, laying the groundwork for advancing ir toward human-like reasoning in complex search scenarios. benchmarks in complex retrieval tasks existing datasets for information retrieval, such as ms marco @xcite , natural questions @xcite , and mteb @xcite , primarily focus on queries sourced from search engines. the relationships between queries and documents are typically simple and direct @xcite . recent studies have expanded retrieval benchmarks to address more complex scenarios. birco @xcite introduces a benchmark designed to evaluate information retrieval tasks with complex objectives. instruction-based datasets @xcite @xcite , for instance, evaluate the instruction-following capabilities of retrieval models by embedding explicit instructions within queries to better represent users' retrieval intents. furthermore, some works have assessed retrieval models' abilities to handle logical reasoning tasks, including boolean logic @xcite , negation @xcite , and multi-hop reasoning @xcite . these efforts mark significant progress in increasing query complexity. however, while research in the generative modeling domain has explored the ability of llms to handle multi-constraint instructions @xcite @xcite , studies on retrieval models in multi-condition scenarios remain sparse. ## multiconir we introduce multiconir, a benchmark designed to evaluate the capacity of retrieval models to process multi-condition queries. formally, given a query q k composed of k conditions @xmath0, c 2 , . . . , c k } with @xmath1, . . . , 10}, we construct a structured retrieval setup consisting of: this controlled design enables a principled evaluation of multi-condition retrieval along three fundamental axes: (1) complexity robustness: the model's retrieval effectiveness as k increases, measured by its ability to distinguish d + from d k-1 ; (2) relevance monotonicity: the extent to which the retrieval model enforces a strict ordering such that s(q k , d j ) > s(q k , d j+1 ) for all j, ensuring that documents satisfying more conditions are ranked higher; and (3) format invariance: the stability of retrieval performance under transformations of query representation, quantified by discrepancies in ranking outcomes across query formats. ## experiments we evaluate 15 representative retrieval and reranking models from diverse architectures and varying model sizes, including sparse retrieval model: bm25 @xcite ; bertbased retrieval models: gte-large-en-v1.5 @xcite and jina-embeddings-v3 @xcite ; llm-based retrieval models: nv-embed-v2 @xcite , bge-enicl @xcite , gte-qwen2-7b-instruct @xcite , gte-qwen2-1.5b-instruct @xcite , e5-mistral-7b-instruct @xcite , gritlm-7b @xcite , @xcite ; pointwise reranking models: bge-reranker-v2-m3 @xcite , bge-reranker-v2-gemma @xcite , followir-7b @xcite ; finetuned list-wise reranker: rankzephyr @xcite ; advanced llm for zero-shot ranking: gpt-4o @xcite . details of each model are provided in appendix a. ## conclusion in this work, we introduced multiconir, a novel benchmark designed to rigorously evaluate information retrieval models in realistic multi-condition scenarios, a critical area where existing evaluation frameworks are lacking. through three specifically designed tasks-complexity robustness, relevance monotonicity, and query format sensitivity-conducted across five diverse domains. experiments revealed that existing models struggle with multi-condition retrieval, with their performance degrading as the number of conditions increases; rerankers excel for single-condition queries but fail in multi-condition scenarios. notably, rerankers are more sensitive to complex inputs. gpt-4o outperforms specialised ir systems, exposing a performance gap in handling complex information needs. our findings highlight an urgent need for new modeling approaches and training paradigms specifically tailored for robust multi-condition understanding. multiconir serves as a valuable resource to drive this research, benchmark progress, and ultimately propel information retrieval systems towards a more sophisticated, human-like comprehension of complex information needs.