Retrieval-Augmented Chain-of-Thought in Semi-structured Domains (2023)

Abstract: Applying existing question answering (QA) systems to specialized domains like law and finance presents challenges that necessitate domain expertise. Although large language models (LLMs) have shown impressive language comprehension and in-context learning capabilities, their inability to handle very long inputs/contexts is well known. Tasks specific to these domains need significant background knowledge, leading to contexts that can often exceed the maximum length that existing LLMs can process. This study explores leveraging the semi-structured nature of legal and financial data to efficiently retrieve relevant context, enabling the use of LLMs for domain-specialized QA. The resulting system outperforms contemporary models and also provides useful explanations for the answers, encouraging the integration of LLMs into legal and financial NLP systems for future research.

## introduction building nlp systems for answering questions in the legal and financial domains could save time and resources, ensure compliance, and enhance the overall accuracy and effectiveness of legal and financial operations @xcite . applying qa systems to such domains poses unique challenges. these domains feature complex jargon, nuanced phrasing, and contextual dependencies that require specialized knowledge and expertise @xcite . a system tailored to these domains should be able to efficiently process and analyze large volumes of legal, financial, or regulatory documents, extracting relevant insights and answering targeted queries. large language models (llms) have shown impressive performance on several nlp tasks @xcite . @xcite show that llms trained on large amounts of data are able to obtain the necessary domain knowledge through incontext learning (icl) @xcite . how-ever, a major limitation of llms is the limit on the input size. there are many attempts to address this limitation @xcite @xcite and multiple transformer models are able to handle longer contexts @xcite @xcite . however, @xcite show that model performance on certain parts of the input decreases with input size. further, the cost and latency of llms increases with the input size. the context required for legal and financial questions is often large and may not fit within the token limit, requiring more efficient retrieval. financial and legal documents are often semi-structured. for example, figure 1 shows a section from the us internal revenue code. the text is organized into subsections, paragraphs and bullet points, which we leverage for better information retrieval. further, financial reports often contain quantities in tabular format. we exploit these structures in a prompting approach that incorporates retrieval to workaround the context token limit. we evaluate the proposed method on two datasets: finqa @xcite and sara @xcite . these datasets feature complex questions which require multiple steps of reasoning and arithmetic computations, which is challenging for language systems. we adopt chainof-thought (cot) prompting @xcite for generating the answers since it is well suited for performing reasoning in a step-by-step manner. a chain of thought is a coherent sequence of reasoning steps that lead to the correct answer in a stepby-step manner. providing examples of questionanswer pairs along with their cots prepended to the test question causes gpt-3 to likewise output a cot along with the answer for the test question, and improve its overall reasoning accuracy. cot prompting is especially useful for complex tasks which require multiple steps of reasoning over the given input. the results demonstrate that this simple and efficient approach outperforms state-of-the-art models in these domains. training llms on financial and legal data may not be feasible as they may contain sensitive information. the use of icl circumvents the problem and avoids expensive and tedious process of data collection and training. this makes the proposed approach a practical solution in scenarios where labeled data is limited or expensive to obtain. additionally, cot prompting offers the advantage of generating explanations and facilitating interpretability in critical domains where it is a key obstacle for the adoption of ai systems @xcite . however, a major drawback of the approach is that it is task-specific. in particular, the retrieval relies on the structure within in the data and needs to be adapted to data from different sources foot_0 . we hope our work fosters research on coupling llms with retrieval in domains such as finance and law, where the ability to extract insights and answer questions about vast amounts of domain-specific data has many practical applications. ## related work previous work has proposed training specialized llms for financial and legal domains @xcite @xcite . however, doing so requires a large amount of data, compute and cost. sun et al. (2023a) nay et al. ( 2023 ) test various gpt models with icl to answer multiple choice questions over tax laws. a retriever augmented setting is tested where a dense passage retriver, gtr @xcite , retrieves the top 4 relevant sections to the questions. since entire sections are passed to the llms, the text has to be truncated. this study extends past work by complementing llms with a retriever that extracts the relevant text from within the statutes, allowing for larger contexts and more in-context examples in the prompt. ## data we use two datasets containing questions that involve multi-step logical and arithmetic reasoning from the legal and financial domains respectively. ## methodology figure 2 shows an overview of our proposed approach, which consists of two main components: retrieval and answering. the retrieval step involves filtering paragraphs from text and rows from tables that are relevant to the question. the retrieved information is then passed to the answering model. ## discussion this study aims to utilize llms for challenging domain-specific qa tasks by using icl along with retrieval techniques that leverage the semistructured nature of financial and legal data. the proposed approach is simple and performs well compared to existing systems. it exploits icl which avoids the costly and time-consuming processes of data collection and training. since the proposed system produces a chain-of-thought with each output, it is easily interpretable and errors can be identified and rectified by human supervision @xcite . we hope this work will encourage researchers to delve deeper into the analysis and development of llm-integrated nlp systems and retrievalaugmented llms. ## limitations the retrieval algorithms in our study are specifically tailored to each dataset. despite good reasoning abilities, the evaluation reveals that arithmetic errors are common. further, inference with llms can be costly with latency higher than traditional approaches, making it sub-optimal for handling large volumes of data efficiently. these limitations point to interesting future directions such as using arithmetic tools as plugins @xcite for better performance and more generalizable retrieval algorithms. further, several domain-specific llms can be tested @xcite @xcite .