Reducing hallucination in structured outputs via Retrieval-Augmented Generation (2024)

Abstract: A current limitation of Generative AI (GenAI) is its propensity to hallucinate. While Large Language Models (LLM) have taken the world by storm, without eliminating or at least reducing hallucination, real-world GenAI systems will likely continue to face challenges in user adoption. In the process of deploying an enterprise application that produces workflows from natural language requirements, we devised a system leveraging Retrieval-Augmented Generation (RAG) to improve the quality of the structured output that represents such workflows. Thanks to our implementation of RAG, our proposed system significantly reduces hallucination and allows the generalization of our LLM to out-of-domain settings. In addition, we show that using a small, well-trained retriever can reduce the size of the accompanying LLM at no loss in performance, thereby making deployments of LLM-based systems less resource-intensive.

## introduction with the advent of large language models (llms), structured output tasks such as converting natural language to code or to sql have become commercially viable. a similar application is translating a natural language requirement to a workflow, a series of steps along with logic elements specifying their relationships. these workflows encapsulate processes that are executed automatically upon certain conditions, thereby increasing employee productivity. while enterprise systems offer such functionality to automate repetitive work and standardize processes, the barrier to entry is high, as building workflows requires specialized knowledge. generative ai (genai) can lower this barrier since novice users can specify in natural language what they want their workflows to execute. however, as with any genai application, using llms naively can produce untrustworthy outputs. such is the public concern for llms producing hallucinations that the cambridge dictionary chose hallucinate as its word of the year in 2023 @xcite . retrieval-augmented generation (rag) is a well-known method that can reduce hallucination and improve output quality, especially when generating the correct output requires access to external knowledge sources @xcite . in this work, we describe how, in the process of building a commercial application that converts natural language to workflows, we employ rag to improve the trustworthiness of the output by reducing hallucination. workflows are represented as json documents where each step is a json object. figure 1 shows an example of a text requirement and its associated json document. for simplicity, we include only the basic properties needed to identify a step along with properties indicating the relationship between steps. besides the workflow steps, there may also be a trigger step that determines when the workflow should start, and sometimes this trigger requires a database table name. hallucination in this task means generating properties such as steps or tables that do not exist. while fine-tuning a sufficiently large llm can produce reasonably good workflows, the model may hallucinate, particularly if the natural language input is out-of-distribution. as the nature of enterprise users requires them to customize their applications, in this case by adding their own type of workflow steps, a commercial genai application needs to minimize the out-of-distribution mismatch. while one could fine-tune the llm per enterprise, this may be prohibitively expensive due to the high infrastructure costs of fine-tuning llms. another consideration when deploying llms is their footprint, making it preferable to deploy the smallest llm that can perform the task. our contributions are the following: ## related work retrieval-augmented generation is a common approach to limit generation of false or outdated information in classical nlp tasks such as question answering and summarization @xcite @xcite . in the genai era, it refers to a process where relevant information from specific data sources is retrieved prior to generating text; the generation is then based on this retrieved information @xcite . our work differs from standard rag as we apply it to a structured output task. instead of retrieving facts, we retrieve json objects that could be part of the json output document. providing plausible json objects to the llm before generation increases the likelihood that the output json properties exist and that the generated json can be executed. a crucial ingredient of rag is the retriever since its output will be part of the llm input. compared to classical methods such as tf-idf or bm25 that use lexical information, dense retrieval has been shown to be more effective as it maps the semantics to a multidimensional space where both queries and documents are represented @xcite @xcite . these retrievers are often used in open-domain question answering systems @xcite , where both queries and documents are unstructured data and thus share the same semantic space. in our case, the queries are unstructured (natural language) and the documents (json objects) are structured. our retrieval training is similar to structure aware dense retrieval (santa), which proposes a training method to align the semantics between code and text @xcite . generating structured data falls within the realm of structured output tasks, which consist of generating a valid structured output from natural language, such as text-to-code, text-to-sql @xcite @xcite or if-then program synthesis @xcite @xcite . they are challenging as they not only require generating output that can be parsed, but also entities or field values that exist in a given lexicon; otherwise the resulting output cannot be interpreted or compiled. for simple database schemas or small lexicons, this extra information can be included in the prompt. however, in our task the available pool of steps that can be part of a workflow is potentially very large and customizable per deployment, thereby making in-context learning impractical. with the arrival of llms, these tasks have become more accessible. in particular, code llms enable developers to write code faster by providing instructions to the llm to generate code snippets @xcite @xcite . these models, trained on large datasets of source code @xcite , have acquired broad knowledge of many programming languages and have been shown to perform better at tasks that necessitate reasoning @xcite . since the json schema to represent workflows is domain-specific, we cannot use these models off-the-shelf. while fine-tuning them on a small dataset increases the quality of results, extra steps are required to reduce hallucination and support out-of-domain queries. lastly, an alternative and complementary technique to reduce hallucination with llms is guided generation using tools such as outlines @xcite . a sufficiently expressive contextfree grammar could ensure that the steps generated by the model exist, but it does not provide extra knowledge as to which steps the flow should include given the natural language query. ## methodology figure 2 depicts the high-level architecture of our rag system. during initialization, indices of steps and tables are created using the retriever. when a user submits a request, the retriever is called to suggest steps and tables. the suggestions are then appended to the user query to form the llm prompt. the llm is then called to generate the workflow in the json format via greedy decoding. to build our system, we first train a retriever encoder to align natural language with json objects. we then train an llm in a rag fashion by including the retriever's output in its prompt. ## experiments as the task we are interested in is part of a commercial enterprise system, we had to devise our own datasets as well as evaluation metrics. ## conclusion we propose an approach to deploy a retrieval-augmented llm to reduce hallucination and allow generalization in a structured output task. reducing hallucination is a sine qua non for users to adopt real-world genai systems. we show that rag allows deploying a system in limited-resource settings as a very small retriever can be coupled with a small llm. future work includes improving the synergy between the retriever and the llm, through joint training or a model architecture that allows them to work better together.