Embedding-Free RAG (2025)

Abstract: Retrieval-Augmented Generation (RAG) is the current state-of-the-art method for mitigating the shortcomings of large language models (LLMs) by incorporating external knowledge sources to provide more relevant and accurate responses to user queries. However building performant RAG systems for real use-cases typically requires heavy investment from NLP experts, such as fine-tuning embedding models for specialized domains, experimenting with text chunking strategies and other niche hyperparameter tunings. We propose Embedding-Free RAG, a model-agnostic approach that enables the deployment of a one-size-fits-all RAG pipeline for user-provided grounding documents. Unlike traditional RAG, which relies on embedding models for information retrieval, Embedding-Free RAG leverages the generalized reasoning abilities of LLMs in a novel algorithmic framework during the retrieval stage. Extensive experiments demonstrate that Embedding-Free RAG outperforms existing state-of-the-art methods, achieving up to 4.6x higher F1 scores and up to 2x better question answering accuracy across a wide range of challenging domains.

## introduction large language models (llms) have made remarkable strides since the release of chatgpt in late 2022 @xcite . despite advancements like multimodality @xcite , large context windows @xcite , and improved reasoning @xcite , the newest generation of llms still face challenges first noted during their rise to popularity. issues like factual hallucination @xcite , knowledge staleness @xcite , limited access to proprietary data @xcite , and poor answer attribution @xcite continue to affect even the most advanced models. what information does the company store about me? fiverr privacy policy llm reference quotation generation @xcite grounding document sentence index mapping using fuzzy matching ## embedding-free rag we formally define the "chat with a document" problem and introduce embedding-free rag. in section 2.2, we detail the generalized embedding-free rag framework. section 2.4 discusses some optimizations and algorithm variants, including parallelization in the retrieval stage (section 2.4.1), use of task-specific llms (section 2.4.2) and leveraging rich metadata from structured documents (section 2.4.3). ## empirical evaluations our empirical evaluations focus on domains which prove challenging for modern rag systems, specifically question answering on legal and financial documents. we use the legalbench-rag @xcite and financebench @xcite benchmarks. the queries for both benchmarks directly reference the document to be used, narrowing the scope of rag from a corpus-wide search to effectively the "chat with a document" problem. unless otherwise specified, our evalu-ations use the parallelized variant of embedding-free rag described in section 2.4.1, with gemini 1.5 flash as the llm for both quotation and final answer generation, subdocument size of 3000 words and chunk window size @xmath0. legalbench-rag. this benchmark prioritizes accurate retrieval by extracting concise, highly relevant text segments from legal documents for 4 datasets: privacy question answering (privacyqa) @xcite , contract understanding atticus dataset (cuad) @xcite , mergers and acquisitions understanding dataset (maud) @xcite and contract natural language inference (contractnli) @xcite . the datasets span a variety of areas in the legal space, with privacyqa focusing on privacy policies of consumer apps, cuad on private contracts, maud on merger and acquisition documents of public companies and contractnli focusing on nda-related documents. each of the nearly 7000 queries in the legalbench-rag benchmark is associated to one legal document, with ground-truth annotations created by domain-experts through a rigorous quality controlled process. financebench. this benchmark evaluates llms on financial question answering, covering 40 publicly traded us companies and public filings from 2015 to 2023. we used the open-source evaluation dataset that consists of 150 cases, designed for a fine-grained analysis of question answering capabilities across the following 3 categories: ## conclusion embedding-free rag advances rag by replacing traditional embedding models with an llmbased, model-agnostic approach. our one-size-fitsall pipeline removes the need for extensive hyperparameter tuning, cutting deployment costs and complexity. extensive empirical evaluations on challenging legal and financial datasets demonstrate embedding-free rag's superior performance on retrieval tasks question-answering accuracy. scalability is enabled through parallelization, supporting efficient processing of large documents. embedding-free rag offers a powerful and adaptable solution for real-time, user-driven question answering on user-provided documents, particularly in specialized domains where domain-specific knowledge is crucial.