Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALM s (2024)

Abstract: Retrieval-Augmented Large Language Models(RALMs) have made significant strides in enhancing the accuracy of generated responses. However, existing research often overlooks the data quality issues within retrieval results, often caused by inaccurate existing vector-distance-based retrieval methods. We propose to boost the precision of RALMs’ answers from a data quality perspective through the Context-Driven Index Trimming (CDIT) framework, where Context Matching Dependencies (CMDs) are employed as logical data quality rules to capture and regulate the consistency between retrieved contexts. Based on the semantic comprehension capabilities of Large Language Models (LLMs), CDIT can effectively identify and discard retrieval results that are inconsistent with the query context and further modify indexes in the database, thereby improving answer quality. Experiments demonstrate average improvement of 3.75% in accuracy on challenging open-domain question-answering tasks. Also, the flexibility of CDIT is verified through its compatibility with various language models and indexing methods, which offers a promising approach to bolster RALMs’ data quality and retrieval precision jointly.

## introduction retrieval-augmented large language models (ralms) have drawn extensive attention, as they effectively ameliorate hallucination @xcite , update the knowledge required for llms with minimal cost @xcite , and provide explanations for contents generated by llms @xcite . however, recent study has demonstrated that not all retrieved citations are useful for the generation result @xcite * these authors contribute equally to this work. figure 1 : improve data quality of database to enhance the accuracy of generated answers by ralms. @xcite , where retrieval may reduce the quality of generation. for example, if retrieval contains information conflicts, the generation quality may deteriorate, which leads to false answers to factual questions @xcite . in view of the challenges above, the nlp community mainly focus on enhancing the retrieval precision, e.g. arr @xcite , replug @xcite , and atlas @xcite , which learns to align the retriever outputs with the preferences of llms. however, these works pay little attention to the data quality of knowledge itself. more specifically, it is commonly assumed that data in the knowledge base (usually implemented as vectors in a vector database) is consistent, which is actually not the case in real-world applications. some sentences appear contextually similar yet are actually opposite in reality. for ex-ample, consider two sentences "he turned on the radio" and "he turned off the radio". it is obvious that the semantic meanings of actions in these two sentences are completely opposite. however, the distance between vector representations of the two tends to be small, as most text embedding models are trained to project sequences of tokens that frequently co-occur as neighboring vectors in the high dimensional semantic space @xcite . thus, existing vector-distance-based retrieval methods implemented in vector databases may treat these two sentences as "highly similar" knowledge, and provide such irrelevant or even conflicting sentences as referring knowledge to the down-stream language generator, which confuses the llms and deteriorates the quality of the generated answer @xcite . as shown in figure 1 , we approach the issue of retrieval quality from the perspective of data quality. specifically, inspired by matching dependencies(mds), a classical rule-based data quality management method in the database community @xcite , we propose context matching dependencies (cmds) that capture and regulate the consistency between the knowledge context and its vector representation. then we establish a context-driven index trimming (cdit) framework that mainly utilizes cmds and llm to improve the quality of ralms answers by trimming the indexes of vector database. the cdit framework starts with an initial retrieval by the retriever in ralms. then the preliminary retrieval results are sent to the cmds where an llm is employed to determine whether the retrieved knowledge conforms to the cmds constraints. if the retrieval satisfies the cmds, it will be passed to llms following conventional ralms. otherwise, the retrieval will be discarded, and the vector-search index related to this retrieval will be corrected such that future similar queries can avoid unrelated retrievals return by the vector database. we experimentally verify the effectiveness of cdit by open-domain question answering. in addition, we integrate cdit with different language generation models and index construction methods, demonstrating the flexibility of our framework. cdit surpasses the basic models with average accuracy improvements of 3.75% on vairous language models. it also boosts the model accuracy by 3.44%, 4.07%, and 3.75% over indexflatl2, indexhnswflat, and indexivfflat, respectively. among them, the highest performance improve-ment can reach up to 15.21%. our main contributions are as follows: ## related work retrieval improvements in rag. not all of the retrieved contexts benefit the final results @xcite . in order to improve the retrieval quality, previous work mainly focuses on fine-tuning the retriever to align with the language model. for example, replug @xcite freezes the parameters of language model llm and optimizes the retriever to adapt to the language model. atlas @xcite , by contrast, jointly trains the retriever and the language model. additionally, other work explores improving strategies before and after retrieval. specifically, document segmentation strategies @xcite and embedding models @xcite ) can be improved before retrieval. diversity ranker in haystack @xcite and lostinthemiddleranker @xcite , on the other hand, investigates document re-ranking after retrieval. different from previous methods, we approach the retrieval quality issue from the perspective of data quality management, where the vector index is trimmed based on data consistency captured by cmds and llms. ## context matching dependencies we first recall matching dependencies (mds) @xcite before introducing our methods. given a relational schema r consisting of a set of attributes attr(r), for each attribute @xmath0, dom(a) denotes the domain of a. consider an instance r of r and a tuple @xmath1, then for ∀@xmath2, t[a] ∈ dom(a), where t[a] represents the projection of t onto a. matching dependencies (mds) are defined to match the attributes of different tuples as follows. similar to relational database tuples that consist of multiple attributes, natural language sentences can be represented by their linguistic components such as subject, predicate and object @xcite . thus, inspired by mds for entity resolution in relational databases, we can determine whether two sentences are semantically similar based on the similarity between their corresponding linguistic components. let sub, pre and obj denote the subject, predicate and object of a sentence, respectively. we also define semantic id (sid) that denote the semantic meaning of a natural sentence. similar to "id" as the main key in relational databases, sid identifies a sentence in the high-dimensional semantic space. if sentences s 1 and s 2 have similar semantics, then their semantic ids are consistent, denoted as example 1 consider the following two sentences. inspired by mds designed for entity resolution in databases, in order to serve rag better, we the corresponding words in sentence ∼, ̸ ∼ consistency predicate ≈, ̸ ≈ similarity predicate ⇀ ↽ the matching operator example 2 consider the following cmd. ## conclusion our study presents a context-driven index trimming (cdit) framework, which enhances the accuracy of ralms by focusing on data quality within vector databases. experiments show an average 3.75% increase in accuracy, highlighting the robustness of cdit across models and indexing methods. while challenges in long text handling and reliance on llms are noted, the adaptability and potential of cdit suggest a bright future in nlp.