Reimagining Retrieval Augmented Language Models for Answering Queries (2023)

Abstract: We present a reality check on large language models and inspect the promise of retrieval-augmented language models in comparison. Such language models are semi-parametric, where models integrate model parameters and knowledge from external data sources to make their predictions, as opposed to the parametric nature of vanilla large language models. We give initial experimental findings that semi-parametric architectures can be enhanced with views, a query analyzer/planner, and provenance to make a significantly more powerful system for question answering in terms of accuracy and efficiency, and potentially for other NLP tasks.

## introduction as language models have grown larger @xcite , they have fared better and better on question answering tasks @xcite and have become the foundation of impressive demos like chat-gpt @xcite . models like gpt-3 @xcite and chat-gpt generate fluent, human-like text, which comes the potential for misuse as in high-stakes healthcare settings @xcite . large language models (llms) also come with several significant issues @xcite . llms are costly to train, deploy, and maintain, both financially and in terms of environmental impact @xcite . these models are also almost always the exclusive game of industrial companies with large budgets. perhaps most importantly, the ability of llms to make predictions is not commensurate with their ability to obtain insights about their predictions. such models can be prompted to generate false statements @xcite , often do so unprompted @xcite and when combined with its ability to easily fool humans, can lead to misuse @xcite . in recent years, we have seen the promise of retrieval-augmented language models partially addressing the aforementioned shortcomings @xcite @xcite @xcite . the architecture of such models is semi-parametric, where the model integrates model parameters and knowledge from external data sources to make its predictions. the first step of performing a task in these architectures is to retrieve relevant knowledge from the external sources, and then perform finer-grained reasoning. some of the benefits these architectures offer are that the external sources can be verified and updated easily, thereby reducing hallucinations @xcite and making it easy to incorporate new knowledge and correct existing knowledge without needing to retrain the entire model @xcite . models that follow semi-parametric architectures (spa) are typically smaller than llms and they have been shown to outperform llms on several nlp tasks such as open domain question answering (see in this paper we argue that building on the core ideas of spa, we can potentially construct much more powerful question answering systems that also provide access to multi-modal data such as image, video and tabular data. we describe post-text, a class of systems that extend spa in three important ways. first, posttext allows the external data to include views, a concept we borrow from database systems @xcite . a view is a function over a number of data sources, @xmath0, ..., d n ). in databases, sql queries are used to define tabular views. for example, v can be a table of records of minors that is derived from a table of person records by selecting only those with @xmath1. in general, however, views need not be tabular. when a view is materialized model #params outperformed llm's sizes tasks realm @xcite ) 330m 11b (t5) open-qa retro @xcite 7.5b 178b (jurassic-1), 280b (gopher) language modeling atlas @xcite 11b 175b (gpt-3), 540b (palm) multi-task nlu, open-qa rag @xcite 400m 11b (t5) open-qa fid @xcite 770m 11b (t5), 175b (gpt-3) open-qa second, posttext contains a question analyzer and planner module that decides on the best strategy to answer a question that may involve first answering multiple subquestions in sequence or in parallel. this module bears similarity to query optimization techniques in database systems but will go significantly beyond the techniques established in database systems since, there are multiple different ways to answer a natural language question, especially with the availability of multi-modal data. finally, posttext supports computing the provenance of answers to questions. the provenanceaware answer generator module can track the evidence (training data or external sources) that is used for the answers, even if views are used as intermediate results. we illustrate the power of posttext with examples in the next section and also the overview of its architecture. in the remaining sections, we describe the different components of posttext. ## overview of posttext example 1 consider a setting where we answer questions over data that includes images of dishes and text with restaurant reviews. we can create a view that aligns these two data sets so we can answer more complex queries readily. the view, the table in the middle of figure 1 , aligns dishes with relevant reviews and the corresponding restaurants. note that creating this view involves an intermediate step of identifying the name of the dish in an image. the view also stores the provenance links to the actual reviews from which the snippets were 1 we use queries and questions interchangeably. this view can be used to answer questions that would be more difficult without it. for example, if a person recalls a nice dish she had in the past but does not remember its name and is trying to figure out which restaurants serve the same dish and what are the reviews, she can pose the question, which includes both the question in text and an image of the dish. the answer states the name of the dish in question and lists restaurants with top reviews for that dish, along with images of the dish and snippets of those reviews and their provenance. ## question analyzer & planner the question analyzer and planner (qap) module examines the input question and generates a plan, i.e., a sequence of sub-questions whose answers can be combined to form an answer to the input question. for each subquestion in the plan, qap first checks whether external knowledge is needed. if not, the language model can be used to derive the answer. otherwise, the subquestion is passed to the knowledge retriever to discover and retrieve relevant data for the subquestion at hand. the results from the knowledge retriever and the plan are passed to pag (i.e., the rightmost green box in figure 2 ). it is still an open and challenging question to determine whether a language model can confidently answer a question @xcite . any solution to this problem will help improve the plan generator. an example plan from the qap module for our running example is as follows: (1) find the name of the dish x in the input image, (2) find restaurants that serve x, (3) find the top restaurant among the results from ( 2 ). this plan is viable because (a) there is an index associating embeddings of images with the name of the main entity of the image, (b) there exists a view as shown in figure 1 , which supports the search for restaurants that serve a particular dish. top answers can be derived by computing the scores of the reviews or approximating it based on the sentiment of the reviews and then ranking the results based on such scores. the information from ( 2 ) is passed to pag which will compute the answer along with its provenance. this plan is based on the heuristic to push selection conditions early before joining/combining different data sources if needed. the conditions in the question are "good version" and "this dish". in this case, no joins are required as the view already combines the required information in one place. hence, qap seeks to first find the name of the dish to narrow down the reviews restricted to this dish. alternatively, it could also retrieve all good reviews before conditioning on the name of the dish. yet another plan could be to match the image directly to the images of the view to find the top reviews. or, it may decide to directly retrieve only top reviews with images similar to the image in the question from the external data sources and condition the answer based on the name of the restaurant mentioned in the reviews. in all possible plans, the knowledge retriever is responsible for discovering and retrieving the relevant data for the qap plan. in addition to the logic that may be needed for decomposing the question into subquestions, a plan is also needed for composing the subanswers obtained to form an answer to the input question. the plan is shared with the pag module for deriving the associated provenance. a fundamental challenge in developing the qap module is how to derive candidate plans and decide what is the "best" plan for answering the question when there are different ways to obtain an answer. achieving this requires understanding how to compare amongst alternative plans for deriving an answer to the question. this problem bears similarity to query evaluation techniques for database systems (e.g., @xcite ). it will be interesting to investigate whether database query planning techniques and ideas can synergize with question understanding and planning techniques (e.g., @xcite @xcite to develop a comprehensive query planner. emerging work such as chain of thought reasoning @xcite , where a sequence of prompts are engineered to elicit better answers, react @xcite , where reasoning and action techniques are applied for deriving an answer, and more recently, work that generates a plan which can call lms for resolving subquestions @xcite are also relevant. these techniques so far are restricted to text and does not compare among different plans. another challenge in the context of nl questions is that while there is a single correct answer to an sql query over a database, there are po-tentially many different correct answers to a nl question @xcite @xcite . hence the space of possible plans to derive the "best" answer most efficiently is even more challenging in this case. we are advocating for a system that can reason and compare at least some viable strategies to arrive at a best plan for deriving a good answer efficiently. naturally, one can also train a lm to create a plan. our belief is that taking a more systematic route to planning can relief the need for the amount of training data required and will also aid provenance generation through its ability to describe the steps it took and the sources of data used in each step to generate an answer. as we shall explain in section 5, the cost and accuracy of knowledge retrievers can also play a role in determining what is a better strategy for computing a good answer. ## knowledge retriever the role of the knowledge retriever is to provide the information that the system lacks in order to fulfill the given task, typically at the inference time. more importantly, we envision that the knowledge retriever proposed in our framework has the ability to access knowledge stored in different sources and modalities, retrieve and integrate the relevant pieces of information, and present the output in a tabular data view. the structured output contains raw data items (e.g., text documents, images or videos) and and optionally different metadata, such as textual description of each data item. such structured output allows downstream (neural) models to consume the retrieved knowledge efficiently and also allows developers and users to validate the provenance conveniently. existing information retrieval models mostly focus on a single form of data. below we first describe briefly how knowledge retrieval is done for unstructured and structured data. we then discuss the technical challenges for building a unified knowledge retriever, as well as recent research efforts towards this direction. retrievers for unstructured data for unstructured data, such as a large collection of documents (i.e., text corpus) or images, knowledge retrieval is often reduced to a simple similarity search problem, where both queries and data in the knowledge source are represented as vectors in the same vector space @xcite . data points that are close to the query are considered as relevant and thus returned as the knowledge requested. traditional information retrieval methods, whether relying on sparse vector representations, such as tfidf @xcite and bm25 @xcite , or dense representations, such as lsa @xcite , dssm @xcite , dpr @xcite , are the canonical examples of this paradigm. notice that the vector space model is not restricted to text but is also applicable to problems in other modalities, such as image tagging @xcite and image retrieval @xcite . retrievers for structured data when the knowledge source is semi-structured (e.g., tables) or structured (e.g., databases), the query can be structured and allows the information need to be defined in a more precise way. because the data is typically stored in a highly optimized management system and sometimes only accessible through a set of predefined api calls, the key technical challenge in the knowledge retriever is to formulate the information need into a formal, structured query. to map natural language questions to structured queries, semantic parsing is the key technical component for building a knowledge retriever for structured data. some early works propose mapping the natural language questions to a generic meaning representation, which is later translated to the formal language used by the target knowledge base through ontology matching @xcite . others advocate that the meaning representation should be closely tight to the target formal language @xcite , such as sparql for triple stores. because of the success of deep learning, especially the large pre-trained language models, semantic parsing has mostly been reduced to a sequence generation problem (e.g., text-to-sql). for example, rasat @xcite and picard @xcite , which are generation models based on t5 @xcite , give state-of-the-art results on benchmarks like spider @xcite and cosql @xcite . towards a unified knowledge retriever as knowledge can exist in different forms, a unified knowledge retriever that can handle both structured and unstructured data in different modalities is more desirable. one possible solution for realizing a unified retriever is to leverage multiple single-source knowledge retrievers. when a query comes in, the qap module first decomposes it into several smaller sub-queries, where each sub-query can be answered using one component knowledge retriever. the results from multiple knowledge retrievers can be integrated and then returned as the final output. however, several technical difficulties, including how to accurately decompose the question and how to join the retrieved results often hinder the success of this approach. alternatively, unifying multiple sources of information in a standard representation, using text as a denominator representation, has been promoted recently @xcite . if all data items have a corresponding textual description, it is possible for the knowledge retriever to use only text-based retrieval techniques to find relevant data items once all input entities of non-textual modality have been mapped to their corresponding textual descriptions. such approach circumvents the complexity of managing multiple knowledge stores in different format. moreover, with the success of large multilingual and multi-modal language models (conneau and lample, 2019; @xcite , data of different structures or from different modalities can naturally share the same representation space. while unifying multiple sources of information through representation learning seems to be a promising direction, it should be noted that certain structured information may be lost in the process. for example, by flatting a knowledge graph to sequences of (subject, predicate, object) triples, the graph structure is then buried in the textual form. whether the information loss limits the retriever's ability to handle certain highly relational queries remains to be seen. ## provenance-aware answer generators 6.1 semi-parametric engine demonstrating the provenance of a qa model prediction should center on identifying the datawhether in training data, retrieval corpora, or input-that is most influential in causing the model to make a particular prediction. for example, given the question "who was the first u.s. president?", the system should return the correct answer "george washington" and references to training or retrieval corpora that are-to the model-causally linked to the answer. if the training or retrieval data included washington's wikipedia page, a typical human would expect for this to be included. however, the requirement we impose is causal and counterfactual: had the model not used that data, the pre-diction should change. if the prediction does not change, then from the causal perspective, there may be other data that is either more influential or duplicative (e.g., if whitehouse.gov is in the training data, it is duplicative). next, we describe common semi-parametric models and sketch how this casually-based answer provenance could be obtained and computational challenges to overcome. provided an input prompt and retrieved text, semi-parametric models like atlas @xcite or passing documents as prompts to gpt-3 @xcite are adept at generating freetext, short answers. likewise, parametric models with flexible input like gpt-3 can be combined with retrievers to achieve a similar goal; alternatively, transformer models can be retrofitted with layers so that passages can be integrated in embedding space @xcite . while retrievalaugmentation is no catch-all panacea to model hallucination, it does mitigate the problem @xcite . additionally, models' explanations can make it easier to know when to trust models and when not to @xcite . in the case of qa models that take question plus retrieved text as input, there are several options. first, the model could provide several alternative answers which provide insight into the distribution of model outputs, rather than just a point estimate. second, the model could provide a combination of feature-based explanations such as token saliency maps and the model's confidence in a correct answer @xcite . when combined, they can jointly influence the degree to which humans trust the model @xcite . however, to provide a complete account of model behavior, we must return to the training of model and the data used. in short, we endeavor to identify the combination of input, training data, and retrieved text that caused the model to produce the distribution of outputs (i.e., answer(s)). this is, of course, challenging due to scale of language model training data like c4 @xcite and the pile @xcite and that establishing causal-and therefore more faithful-explanations of model behavior is difficult. training data attribution is one promising idea in this direction-it uses gradient and embedding based methods to attribute inference behavior to training data @xcite . for example, influence functions @xcite and tracin @xcite ) link predictions to specific training examples, but are computationally expensive and are approximate rather than exact solutions. to firmly establish a causal connection, one could fully re-train the model without the identified training examples, but this is prohibitively expensive in practice. future development of efficient training data attribution, combined with methods like interpretations of input plus retrieved data, is a promising direction towards more complete explanations of model predictions. ## preliminary findings to test our hypothesis that views are valuable for answering queries, especially queries that involve counting or aggregation, we have implemented a first version of posttext foot_0 and compared it against some qa baselines. the current implementation of posttext assumes views over the underlying data are available in tabular format. the qap module simply routes the query to a view-based engine (vbe) or a retrieval-based engine (rbe) to answer the query. vbe picks the best view and translates the natural language query into an sqlite query against the view using openai's gpt-3.5-turbo/gpt-4 model. it then executes the sqlite query against the view to obtain a table result which is then translated into english as the final answer. vbe also analyzes the sqlite query to compute the provenance of the answers. at present, it does so by simply retrieving all tuples that contributed to every (nested) aggregated query that is a simple (select-fromwhere-groupby-having clause) and does not handle negations. an example of the vbe process is described in appendix b. rbe is implemented with langchain's retrievalqawithsources library. it first retrieves top-k documents that are relevant for the query and then conditions its answer based on the retrieval. the answer and the ids of the retrieved documents are returned. for our experiments, we use the 42 multihop queries over 3 synthetic personal timelines of different sizes from timelineqa's benchmark @xcite . the personal timelines model the daily activities (e.g., the trips made, things bought, people talked to) of a person over a period of time. we create a view around each type of activity (e.g., trips, shopping, daily_chats) for vbe. for further comparison, we also ran langchain's sql-databasechain (dbchain) to perform qa over the same vbe views. furthermore, we ran it over timelines loosely structured as a binary relation of (date,description) pairs (called dbchain (no views)). we compared the returned answers against the ground truth answers by grading them on a scale of 1-5, with a llm, where 5 means the returned answer has the same meaning as the ground truth answer (the grading scheme is described in the appendix c). our results are shown in tables 2 and 3 . across both tables, the results on dbchain vs. dbchain(no views) reveal that adding some structure (in this case adding views) is crucial for better performance. although the benchmark is a relatively small dataset, the scale of the timelines already reveals an impact on the accuracy across all qa systems. for dbchain, the drop in accuracy as the size increases because it sometimes relies on generating sql queries that return all relevant records and passing all the records to the language model to compute the aggregate. when the results returned are large, which tends to be the case for larger timelines, the token limit of the llm is often exceeded. vbe has a similar downward trend. it tends to generate queries that push the aggregates to the sql engine and hence, avoids the issue of exceeding the token limit of the language models for many cases encountered in dbchain. still, as the timeline gets larger, the result returned by the generated sql query tends to be bigger and when these results are passed to the verbalization component to compose an answer in english, this may sometimes exceed the token limit of the language model. we also found that on a handful of cases, it so happens that the sql query generated for l is invalid compared with those generated for the sparse dataset. the scores of rbe is relatively stable across all data densities. but overall, it tends to score lower compared with vbe and dbchain . this is because rbe relies on retrieving the top k documents from an index to condition the answers upon, regardless of the size of the timeline. however, these retrieved documents may not contain all the necessary information for answering the question in general. even though the grading scores may not reveal this, the answers tend to be "more wrong" for aggregate queries over a larger timeline. ## conclusion posttext enhances the core ideas of semiparametric architectures with views, a query analyzer & planner, and a provenance-aware answer generator. our initial results indicate that post-text is more effective on queries involving counting/aggregation when we provide structured views to facilitate computation. we plan to further develop and investigate posttext to automatically determine what views to construct, how does one generate plans and compare amongst plans, and how can one measure the quality of answers with provenance.