C ar E xpert: Leveraging Large Language Models for In-Car Conversational Question Answering (2023)

Abstract: Large language models (LLMs) have demonstrated remarkable performance by following natural language instructions without fine-tuning them on domain-specific tasks and data. However, leveraging LLMs for domain-specific question answering suffers from severe limitations. The generated answer tends to hallucinate due to the training data collection time (when using off-the-shelf), complex user utterance and wrong retrieval (in retrieval-augmented generation). Furthermore, due to the lack of awareness about the domain and expected output, such LLMs may generate unexpected and unsafe answers that are not tailored to the target domain. In this paper, we propose CarExpert, an in-car retrieval-augmented conversational question-answering system leveraging LLMs for different tasks. Specifically, CarExpert employs LLMs to control the input, provide domain-specific documents to the extractive and generative answering components, and controls the output to ensure safe and domain-specific answers. A comprehensive empirical evaluation exhibits that CarExpert outperforms state-of-the-art LLMs in generating natural, safe and car-specific answers.

## introduction conversational question answering (cqa) has recently gained increased attention due to the advancements of transformer-based @xcite large language models (llms). these llms @xcite @xcite are nowadays widely adopted for performing question answering in both open-domain and domain-specific settings @xcite . as the source of additional knowledge conversational question answering systems are typically provided with text paragraphs @xcite , and knowledge graphs @xcite for generating informative dialogues in a domain-specific setting, where such what is the high beam assistant? how do i activate it? the high-beam assistant ensures that you do not often have to turn the high-beam headlight on and off yourself in order to avoid dazzling oncoming traffic. the system is activated automatically when the car is started. extracted from document generated from document systems typically engage in a multi-turn interaction with a user in form of speech or text. figure 1 demonstrates a conversation between a user and a conversational question answering system (carexpert) in a bmw car. leveraging llms end-to-end has several drawbacks @xcite @xcite . firstly, the generated answer is often hallucinated as the knowledge from the pre-trained weights of llms is limited to their training data collection time @xcite . furthermore, retrieval-augmented answer generation suffers from hallucination as well, due to wrong retrieval, complexity of the user utterance and retrieved document. secondly, llms can be exploited using adversarial instructions that may lead the system to ingest malicious input and generate unsafe output @xcite . in the context of a car, the aforementioned downsides imply that the answer could lead to unsafe handling of the vehicle due to a lack of instructions, preservation, warning messages, or appropriate information; or by providing erroneous or confusing information. addressing the aforementioned issues, in this paper we propose carexpert, an in-car conversational question-answering system, powered by llms. carexpert is a modular, language model agnostic, easy to extend and controllable conversational question-answering system developed to work on the text level. on a high-level carexpert performs question answering in two steps. first, given a user utterance it retrieves domain-specific relevant documents wherein the potential answer may exist. second, for predicting the answer, carexpert employs both extractive and generative answering mechanisms. specifically, there are four sub-tasks involved in the overall process: 1) orchestration, 2) semantic search, 3) answer generation, and 4) answer moderation. furthermore, carexpert tackles unsafe scenarios by employing control mechanisms in three ways: i) in the orchestrator using an input filter, ii) by defining prompts for controlling llmbased answer generation, and iii) by an output filter in the answer moderator. furthermore, carexpert employs a heuristic during answer moderation to select answers from multiple models (extractive and generative) and provide the user with the potential best answer as the output.to facilitate voicebased user interaction in the car for real-life use, we encapsulate carexpert with text-to-speech and speech-to-text services. figure 2 depicts a highlevel overview of the carexpert architecture. such modular design of carexpert allows flexible integration to various types of interfaces such as web browser and mobile app (i.e., bmw app). to assess the performance of carexpert we conduct exhaustive evaluations (both qualitative and quantitative). an empirical evaluation exhibits that carexpert outperforms off-the-shelf state-of-theart llms in in car question answering. the contribution of this paper can be summarized as follows: ## approach carexpert aims to generate domain-specific document-grounded answers. the task is divided into four sub-tasks: 1) orchestration, 2) semantic search, 3) answer generation, and 4) answer moderation. we describe the sub-tasks below. ## experimental setup data: the reader and retriever models in carexpert are fine-tuned and evaluated on car-specific data from various sources (owners' manuals, selfservice faqs, car configurator feature descriptions and press club publications). baselines: we choose dense passage retriever (dpr) @xcite , bm25 @xcite , sentence-transformer @xcite and splade @xcite as the baseline retriever. for answer generation we experiment with albert @xcite (extractive) and gpt-3.5-turbo 1 (generative) and luminous-extended 2 (generative). metrics to measure the performance of the retriever we use mean reciprocal rank (mrr@3). for evaluating extractive reader, we utilize tokenlevel metrics, such as f1-score and exact match (em). furthermore, we employ cosine similarity and meteor @xcite to capture the similarity of generated answer aginst the reference response. further details of the datasets, hyper-parameter settings, and metrics can be found in the appendix, in a.1, a.3 and a.4 respectively. ## experiments and results we conduct both qualitative and quantitative experiments to assess different parts contributing to the overall performance of carexpert. ## discussions and potential impact carexpert is built in a modular fashion, which allows for expansion and adaptability to diverse industrial use cases. furthermore, the proposed architecture enables the system to maintain, modify and scale the data more effectively. moreover, a pipeline approach such as carexpert improves the overall interpretability and debugging of a system. finally, the introduced system is controllable and domain-specific as it allows for explicit control over the design and behavior of each of the user carexpert (with paragraphs) gpt-3.5-turbo (w/o paragraphs) gpt-3.5-turbo (with paragraphs) does the car have a parking assist function? yes, park assist supports the following functions: when parallel parking: parking in reverse, parallel to the road. when perpendicular parking: parking forward or in reverse, perpendicular to the road. when driving out of parallel parking spaces. when driving out of perpendicular parking spaces. i'm sorry, but i would need more specific information about the car you're referring to. there are many different car models with various features, and the availability of parking assist functions can vary between manufacturers and models. can you provide me with the make and model of the car you're interested in? yes, the car has a parking assist function. the first paragraph mentions that park assist supports functions like parallel parking, perpendicular parking, and driving out of both types of parking spaces. the second paragraph further elaborates on the features of the automatic parking assistant, which includes [...] how to mount child seats? to install the child restraint system in the vehicle, proceed as follows: 1. mount the child restraint system in accordance with the instructions from the child seat manufacturer. 2. make sure that the child restraint system anchors engage correctly on both sides in the lower seat mounts. 3. pull out the seat belt strap completely. 4. secure the child restraint system with the seat belt. 5. allow the seat belt strap to be pulled in, then pull it tight against the child restraint system. mounting child seats properly is crucial for the safety of your child while traveling in a vehicle. the specific instructions for mounting a child seat may vary depending on the make and model of the seat and the vehicle. however, here is a general guide on how to mount child seats: 1. read the instructions: carefully read the instruction manual provided with your child seat. it will provide specific details about the installation process and any additional requirements. 2. choose the appropriate seat: [...] to mount child seats, follow these steps: 1. read and follow the instructions provided by the child seat manufacturer. 2. ensure that the child restraint system anchors engage correctly on both sides in the lower seat mounts. 3. pull out the seat belt strap compl. [...] 6. refer strictly to the installation instructions supplied with the child restraint system for specific guidelines. 7. each rear seat position is equipped with a head restraint. [...]. ## conclusion we have introduced carexpert, a new and controlled in-car conversational question-answering system powered by llms. specifically, carexpert employed semantic search to restrict the system generated answer within the car domain and incorporated llms to predict natural, controlled and safe answers. furthermore, to tackle hallucinated answers, carexpert proposed an extraction scorebased answer moderator. we anticipate that the proposed approach can not only be applicable for the in-car question answering but also be easily extendable and adapted for other domain-specific settings. in future, we plan to integrate multi-task models to handle multiple task using a single llm and reduce error propagation in the system.