Q uebec Automobile Insurance Question-Answering With Retrieval-Augmented Generation (2024)

Abstract: Large Language Models (LLMs) perform outstandingly in various downstream tasks, and the use of the Retrieval-Augmented Generation (RAG) architecture has been shown to improve performance for legal question answering (Nuruzzaman and Hussain, 2020; Louis et al., 2024). However, there are limited applications in insurance questions-answering, a specific type of legal document. This paper introduces two corpora: the Quebec Automobile Insurance Expertise Reference Corpus and a set of 82 Expert Answers to Layperson Automobile Insurance Questions. Our study leverages both corpora to automatically and manually assess a GPT4-o, a state-of-the-art (SOTA) LLM, to answer Quebec automobile insurance questions. Our results demonstrate that, on average, using our expertise reference corpus generates better responses on both automatic and manual evaluation metrics. However, they also highlight that LLM QA is unreliable enough for mass utilization in critical areas. Indeed, our results show that between 5% to 13% of answered questions include a false statement that could lead to customer misunderstanding.

## introduction to protect their financial situation and property, vehicle owners and homeowners need to buy property damage insurance. however, most people have little to no proper knowledge of insurance products, and rely on insurance representatives to help them properly select and comprehend these products @xcite . as a result, in order to protect the public, insurance regulators, such as the "autorité des marchés financiers" (amf) in quebec, make sure that insurance representatives are well-trained and educated, and that insurers properly inform their customers @xcite . however, customers are increasingly interested in buying insurance products online @xcite . this change impacts how an insurer can adequately inform their customer. traditionally, customers buy products in person or through phone insurance representatives, which allows an insurance expert to help the customer understand the different products and buy the correct one. with an online sale, customers are left to gather information by themselves @xcite . moreover, insurance is regulated locally, which means that insurance products, coverages and laws are different from one jurisdiction to the next. consequently, while many resources are available online, such as "infoassurance" (ibc and gaa, 2024), only the limited set of resources from one's own locality are applicable, and customers must take care not to get information from elsewhere. the rapid progress in natural language processing and the growing availability of insurance data present unprecedented opportunities to bridge the gap between people and insurance knowledge. for instance, legal text summarization @xcite holds the potential to simplify complex legal documents for laypeople. similarly, insurance question-answering (qa) could offer affordable, expert-like assistance to non-expert customers. to this end, we present an end-to-end approach aimed at generating high-quality answers to quebec automobile insurance questions. our methodology harnesses the popular "retrieval-augmented generation" (rag) approach. the main contributions of this work are therefore: this paper is outlined as follows: first, we study the relevant questions-answering legal rag research and its related corpora in section 2. then, we propose our corpora in section 3, and in section 4, section 5 and section 6 we present a set of experiments aimed at evaluating the performances of gpt-4o at answering quebec automobile insurance questions. finally, in section 7, we conclude and discuss our future work. ## related work legal-domain qa rag the advent of large language models (llms) has led to advances in many previously arduous tasks, such as in the application of the rag concept in qa tasks, which has attracted a great deal of research interest in recent years @xcite . answering legal questions has always been more complex due to the inherent difficulties of exploiting specialized texts that stem from handing specialized terminology @xcite and intricate sentence structures @xcite . recently, @xcite has presented an end-to-end methodology to generate answers to any statutory law question leveraging a rag architecture, along with a longform legal question answering dataset comprising 1,868 expert-annotated legal questions in french. likewise, the insurance sector, with its complex documents and nuanced information, could benefit from these advancements. consequently, although research is mainly focused on the legal field, there is also a growing interest in the insurance sector, including for insurance rag. nuruzzaman and hussain (2020) have presented a chatbot that generates accurate and contextual responses by identifying intentions and entities while ensuring semantic relevance and meaning of responses. it is trained on domain-specific datasets to understand insurancespecific terms and information. it notably uses rag strategies to generate responses. likewise, @xcite focuses on a single-turn dialogue covering insurance qa on a korean dataset to respond to insurance customers. ## corpora this section describes the two corpora we created for our work: our french corpus of automobile insurance expertise references documentation for the province of @xcite , and our french corpus of 82 layperson questions about quebec automotive insurance and their expert answers and annotations. first, we will describe our process for creating each corpus foot_2 and then present some key statistics. ## qa methodology this section details our methodology for leveraging a large language model (llm) to answer insurance questions. our choice of architecture is similar to @xcite , ajmi (2024), and @xcite . we use a rag architecture to inject domain expertise into an llm generation for qa. like the previous authors, our rag architecture is inspired by the concept of "advanced rag" @xcite , an architecture that adds a preand post-retrieval steps to the traditional processing. our architecture was built using langchain @xcite , a python framework that consolidates the various components of the rag architecture. as illustrated in figure 1 , first, a retriever selects a small subset of insurance documents from our reference corpus (red), some relevant to the question and some not. then, a generator conditions its answer on the subset of articles returned by the retriever (blue). we describe these two components in details in the following subsections. ## experiments the goal of our experiments is to assess whether an llm can adequately answer technical questions with complex answers, namely quebec insurance questions, with or without a rag architecture. to achieve this, we conduct experiments to automatically and manually evaluate six approaches. ## results in this section, we present and discuss both our quantitative and qualitative results. we also have conducted an ablation study that also use each source individually in appendix b. ## conclusion and future works in conclusion, this paper introduced two new corpora: a quebec automobile insurance expertise reference corpus and a corpus of expert answers to laypeople automobile insurance questions. to generate answers to the questions in our second corpus, we leverage an rag architecture that uses our reference corpus. we experimented with six approaches: a zero-shot that did not use the rag architecture, an rag architecture without references, and four models that incrementally use more of our reference corpus. our results demonstrate that, on average, using our complete reference corpus generates better responses based on both automatic and manual evaluation metrics. our results show that between 5% to 13% of generated answers include a false statement that could mislead a customer, indicating that llm-based technical and sensitive qa is not yet robust enough for mass utilization by the public. in our future works, we plan to extend the references corpus to include amf proprietary documents, such as their insurance representative training manual, and increase the number of expertanswered questions. moreover, we would also like to experiment with other llms, and to conduct a real-life evaluation using real insurance customers. finally, we plan to improve performance with prompt engineering and llm fine-tuning.