Streamlining Conformal Information Retrieval via Score Refinement (2024)

Abstract: Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.

## introduction information retrieval (ir) methods lie at the heart of numerous modern applications, ranging from search engines and recommendation systems to question-answering platforms and decision support tools. these methods facilitate the identification and extraction of relevant information from vast collections of data, enabling users to access the knowledge they seek efficiently and effectively. a popular example of ir is retrieval augmented generation (rag), a technique for reducing hallucinations in large language models (llms) by grounding their responses on factual information retrieved from external sources. while ir methods have been widely adopted, they traditionally lack statistical guarantees on the relevance of retrieved information. this limitation can lead to uncertainty regarding the reliability and correctness of the retrieved information. conformal prediction @xcite is an uncertainty quantification framework that can be used with * equal contribution. in this work, we address this limitation by introducing a novel score refinement method that employs a simple yet effective monotone transformation, inspired by ranking measures, to adjust the scores of any given information retrieval system. by applying standard conformal prediction methods to these refined scores, we deliver significantly smaller retrieved sets while preserving their statistical guarantees, striking a crucial balance between efficiency and accuracy. an illustration of the proposed pipeline is shown in figure 1 . we validate the effectiveness of our method through experiments on three of beir @xcite benchmark datasets, demonstrating its ability to outperform competing approaches in producing compact sets that contain the relevant information. ## background to lay the groundwork for our work, we present a simplified description of the operation of information retrieval systems and how conformal inference figure 1 : retrieval pipeline. the query is first embedded using a semantic embedder, and then the top n candidates are retrieved from a vector store. crucially, their corresponding scores then undergo a refinement transformation before being passed through a conformal prediction method that outputs an adaptive set of documents. ## conclusion we addressed the challenge of large prediction sets in conformal prediction for ir by introducing a novel score refinement method. our experiments on the beir benchmark demonstrated its effectiveness in generating compact, statistically reliable prediction sets, enabling the deployment of conformal prediction in real-world ir systems without sacrificing performance.