Measuring Retrieval Complexity in Question Answering Systems (2024)

Abstract: In this paper, we investigate which questions are challenging for retrieval-based Question Answering (QA). We (i) propose retrieval complexity (RC), a novel metric conditioned on the completeness of retrieved documents, which measures the difficulty of answering questions, and (ii) propose an unsupervised pipeline to measure RC given an arbitrary retrieval system.Our proposed pipeline measures RC more accurately than alternative estimators, including LLMs, on six challenging QA benchmarks. Further investigation reveals that RC scores strongly correlate with both QA performance and expert judgment across five of the six studied benchmarks, indicating that RC is an effective measure of question difficulty.Subsequent categorization of high-RC questions shows that they span a broad set of question shapes, including multi-hop, compositional, and temporal QA, indicating that RC scores can categorize a new subset of complex questions. Our system can also have a major impact on retrieval-based systems by helping to identify more challenging questions on existing datasets.

## introduction retrieval-based approaches are a dominant paradigm in qa. they constitute a basic building for many qa pipelines, as they can gather accurate information from external sources that are useful for answering questions. notable examples include the retrieve and rerank system @xcite and the recent retrieval augmented generation (rag) @xcite . while retrieval systems effectively address questions targeting common knowledge, they can struggle with addressing novel, domain-specific questions that require figure 1 : while both questions are complex using the definition in prior works, the "low rc" question on the left is easy for rag systems, while the "high rc" question on the right is far more difficult because the probability of finding an existing document comparing lions and freezers is low. documents retrieved by commercial search engines illustrate this challenge. rc correlates with the probability that the retrieved documents contain sufficient information to generate a reference answer. fresh, contextually accurate information. generative models, on the other hand, bring the ability to generate responses to such challenging questions. for instance, bing-gpt uses retrieval to answer questions about real-time events beyond the llms' training data cutoff @xcite . however, retrieval augmentation can fail for challenging questions. to address this problem, the qa community developed benchmarks to measure the performance of qa approaches on these types of questions. for example, hotpotqa @xcite , com-plexwebquestions @xcite , and musique @xcite challenge systems by proposing questions with "complex" shapes, such as multi-hop questions, requiring multiple steps of reasoning, e.g., "what is the gdp of the country with the tallest bridge?". while it is as-sumed that these question shapes are equally difficult for rag and other qa systems, subsequent works indicate that this is not the whole picture of qa complexity. for example, @xcite show that nearly 61% of hotpotqa's multi-hop questions actually have "single-hop answer solutions" -i.e., a single document that contains the correct answer -that a simple rag system can use to answer correctly. more recent rag-oriented benchmarks such as fresh qa @xcite further illustrate that the complexity of the answer evidence is relevant in identifying difficult questions since the evidence can become false if not recent enough. in this paper, we introduce a novel metric named retrieval complexity and an unsupervised pipeline, named reference-based question complexity pipeline (rrcp) (section 4) to recognize it. specifically, given a query question and one or more reference answers, rrcp employs an arbitrary retriever to procure pertinent documents. subsequently, utilizing a novel reference-based evaluator named geneval, rrcp scores the question based on the retrieved documents and the supplied reference answers. conceptually, rc approximates the degree to which the evidence required to answer the question is spread across documents in the retrieval batch. the intuition behind rc is that the greater the fragmentation of required information within a retrieval batch, the harder it is for a rag system to answer the question correctly. for example, fig. 1 shows two questions with highly similar question semantics but significantly different retrieval quality (e.g., relevance and comprehensiveness of the information) to illustrate what rc measures. we compare rrcp with supervised and unsupervised models, demonstrating its superior accuracy in classifying rc questions. our unsupervised approach enables applicability to new qa benchmarks, acting as a valuable resource for the qa community. rc is intimately linked to a specific retrieval corpus, capturing (i) index coverage, (ii) retrieval method quality, and (iii) varying semantics of questions into a single metric. to support this claim, we compute rrcp scores for six qa benchmarks using a state-of-the-art search engine (bing) and reveal a strong correlation between rc scores and qa system performance. in addition, we show that our human expert assessment of question difficulty against retrieved evidence also correlates with rrcp scores. finally, we present a qualitative analysis of 200 high-rc questions drawn from all benchmarks. these findings indicate a connection between high rc and diverse question types, such as multi-hop (23%), comparative (10%), temporal (15%), superlative (3%), and aggregate (16%). in summary, our paper (i) motivates the need to estimate the answering difficulty of questions given the availability of evidence, (ii) defines rc, a new metric that measures the difficulty of questions conditioned on a retrieval batch, (iii) proposes rrcp, an unsupervised pipeline to measure retrieval complexity, and (iv) empirically demonstrates that our pipeline is superior at classifying high rc questions and (v) shows that higher rc scores correlate with lower qa performance. ## related work in this section, we introduce the related work on the main topics of our paper: question complexity, llm-based qa, retrieval for qa, and automatic evaluation. ## retrieval complexity of questions a retrieval system provides users with relevant and accurate information in response to their queries @xcite . recently, retrieval augmentation -that is, conditioning generation on retrieved evidence -has been shown to enable generative models to address more complex, multi-faceted questions @xcite . we conceptualize rc, a specific notion of question complexity, which can also be used to evaluate the complexity of questions used in prior retrieval-based qa systems, e.g., rag models. intuitively, a question can be considered complex for retrieval augmentation if it cannot be answered by extracting a snippet from a retrieved document. this dimension of complexity can occur for multiple reasons, such as if the answer requires multiple reasoning steps or a composition of supporting facts spread across retrieved content @xcite . effectively, retrieval complexity occurs when multiple pieces of information must be combined in a complex fashion to generate the correct answer, such as reasoning, comparison, or composition over multiple facts. multiple factors can increase the retrieval complexity, including index coverage, retrieval quality, and question syntax, as each plays a role in determining whether a document containing a snippet that answers the question is retrieved. for example, answering the question "are lions bigger than tigers?" (fig. 1 ) requires a comparison between two related entities. while this question could be considered complex, the close relation of the lions and tigers increases the probability that a single snippet talks about the size of both animals. however, the probability above is an order of magnitudes lower for a similar question, such as "are lions bigger than freezers?". ## modeling retrieval complexity to model and recognize the retrieval complexity (section 3), we designed an unsupervised reference-figure 2 : overview of our rrcp for measuring the complexity of questions. a retrieval system retrieves documents for the input question. these are examined by a reference-based evaluation system, which provides the probability of correctness scores. these approximate the answerability and completeness of retrieved information, i.e., an estimation of rc. ## experiments in this section, we focus on studying retrieval complexity (rc) and our rrcp framework for its detection. first, in sections 5.1 and 5.2 we describe the datasets we considered and the setup of rrcp. then, we validate rc and rrcp by conducting both quantitative and qualitative analyses. specifically, we first compare rrcp against a strong prompted llm unsupervised baseline directly on a set of 4 selected datasets in, targeting the specific complexity class for each of them (section 5.3). secondly, in section 5.4, we evaluate the correlation between the rc classification (complex/not complex) with llm answer capability. this allows us to understand some possible limitations of our notion of complexity. in the third quantitative experiment (section 5.5), we measure the answerability of the questions identified as complex by our pipeline using a state-of-the-art search engine, such as bing. finally, we perform a qualitative analysis to inspect the limitations and the generalizability of our approach (section 5.7). ## retrieval complexity applications in section 5, we show that rrcp is a valuable tool which allows for the recognition of rc with high accuracy. this could be difficult to achieve with a reference-free approach. although the need of references could limit the applicability of rrcp in real-world scenarios, it serves as a crucial step to-wards understanding the potential applications and future directions of rc in the field of qa systems. indeed, rc is a valuable diagnostic tool that can be applied to various settings, including question routing, optimizing document usage in relevance and ranking (rag) systems, filtering qa datasets, and identifying interesting questions according to some complexity notion. the motivating concepts of rc are potentially applicable to many settings, as they provide insights into the underlying complexity of questions and can help reveal challenging questions in existing academic benchmarks. ## conclusion in this paper, we introduced a novel concept of question complexity, measuring the difficulty of finding accurate answers from multiple sources. by combining a top-tier retrieval system with an effective automatic evaluation system, our approach (rrcp) demonstrated high accuracy in handling complex questions that go beyond single-source responses. we performed an extensive experimentation, conducting both quantitative and qualitative analyses on various datasets, proving the robustness of our pipeline in managing and recognizing complex question types, including multi-hop questions, time-based queries, and comparative inquiries. the results consistently showed that our approach outperformed state-of-the-art supervised transformer models and large language models in these challenging scenarios. additionally, our benchmark evaluations underscored the superiority of our method, highlighting its capability to deliver more precise answers where traditional models struggled. the empirical evidence confirmed that our pipeline is particularly effective in classifying high retrieval complexity (rc) questions and that higher rc scores were correlated with lower qa performance, underscoring the validity of our rc metric. as future work, we plan to reduce the reference dependency of the retrieval complexity pipeline (rrcp) in our evaluation system by incorporating large language models (llms) and parametric knowledge. this enhancement aims to increase the number of application scenarios for our framework, broadening its usability and effectiveness in diverse contexts. by leveraging these advanced models, we anticipate further improvements in the efficiency and accuracy of our system, making it even more versatile in handling a wide range of complex queries. while our study provides valuable insights about retrieval complexity and our framework proved to be a useful tool to determine whether a question fits with our notion of complexity, it is important to acknowledge several limitations that may impact our approach: corpus dependency: as mentioned in section 4 and section 5, the performance of our pipeline is limited by the goodness of the corpora considered to build the index. for example, this limitation is evident when thinking about a time-based question asking about recent events not covered by our index. we partially mitigate this problem by working on the completeness constraints as specified in section 4, but we consider this limitation as an open problem, planning to address it in future work. reference-based approach: the core of our framework is based on a reference-based evaluation system. although we can rely on high accuracy and human-like performance in terms of correctness evaluation, on the other hand, we are limited by the need for gold answers to get a good answerability prediction, limiting the scope of rrcp on labeled data. however, despite this limitation, we believe that larger reference-free approaches based on llms could soon replace the automatic evaluation component. quality of the retrieval system: our methodology accuracy scales with the quality and accuracy of the retrieval system. thresholds: determining t ans and t com is a trivial task that can be done with few examples. however, their values are critical to determine the rc with high accuracy. as mentioned in section 5 and section 7, we plan to address these limitations in future work. drop: a reading comprehension benchmark requiring discrete reasoning over paragraphs. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2368-2378, minneapolis, minnesota. association for computational linguistics. yixing fan, jiafeng guo, yanyan lan, jun xu, chengxiang zhai, and xueqi cheng. 2018. modeling diverse relevance patterns in ad-hoc retrieval. in the 41st international acm sigir conference on research & development in information retrieval, sigir '18, page 375-384, new york, ny, usa. association for computing machinery. matteo gabburo, siddhant garg, rik koncel kedziorski, and alessandro moschitti. 2023. square: automatic question answering evaluation using multiple positive and negative references. matteo gabburo, rik koncel-kedziorski, siddhant garg, luca soldaini, and alessandro moschitti. 2022. knowledge transfer from answer ranking to answer generation. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 9481-9495, abu dhabi, united arab emirates. association for computational linguistics. siddhant garg, thuy vu, and alessandro moschitti. 2020. tanda: transfer and adapt pre-trained transformer models for answer sentence selection. proceedings of the aaai conference on artificial intelligence, 34(05):7780-7788. mor geva, daniel khashabi, elad segal, tushar khot, dan roth, and jonathan berant. 2021. did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. transactions of the association for computational linguistics (tacl). vishal gupta, manoj chinnakotla, and manish shrivastava. 2018. retrieve and re-rank: a simple and effective ir approach to simple question answering over knowledge graphs. in proceedings of the first workshop on fact extraction and verification (fever), pages 22-27, brussels, belgium. association for computational linguistics. pengcheng he, jianfeng gao, and weizhu chen. 2021. debertav3: improving deberta using electra-style pretraining with gradient-disentangled embedding sharing. chao-chun hsu, eric lind, luca soldaini, and alessandro moschitti. 2021. answer generation for retrievalbased question answering systems. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 4276-4282, online. association for computational linguistics.