Efficient Answer Retrieval System ( EARS ): Combining Local DB Search and Web Search for Generative QA (2024)

Abstract: In this work, we propose an efficient answer retrieval system **EARS**: a production-ready, factual question answering (QA) system that combines local knowledge base search with generative, context-based QA. To assess the quality of the generated content, we devise comprehensive metrics for both manual and automatic evaluation of the answers to questions. A distinctive feature of our system is the Ranker component, which ranks answer candidates based on their relevance. This feature enhances the effectiveness of local knowledge base retrieval by 23%. Another crucial aspect of our system is the LLM, which utilizes contextual information from a web search API to generate responses. This results in substantial 92.8% boost in the usefulness of voice-based responses. **EARS** is language-agnostic and can be applied to any data domain.

## introduction developing a virtual assistant is crucial for supporting clients as it provides 24/7 assistance, enhancing customer experience with instant responses and personalized interactions. it helps businesses scale their operations efficiently, reducing workload on human support teams and enabling them to focus on more complex issues. one of the essential components of a virtual assistant is a factual questionanswering (qa) system. this system is capable of handling all user queries, providing answers to factual requests, whether domain-specific (related to the services of a particular company) or opendomain. in this paper, we present a factual qa skill as one of the components of the virtual assistant, designed for the clients of mobile telesystems (mts) company 1 . the developed qa skill is proficient in addressing domain-specific inquiries about our services and products, along with open-domain ques-1 more than 84 million subscribers tions. our production-ready qa system seamlessly integrates two qa methodologies: knowledge base search and an llm-based solution, enriched with context from a search engine. the system has been incorporated into both a chat-based interface and a voice interface. given that the skill accepts input from both chat and automatic speech recognition (asr), the query goes through extensive preprocessing ( §3). after preprocessing, the user's query is processed by the retriever ( §4.1), which retrieves the top-𝑛 semantically relevant answer candidates from the knowledge base. next, the ranker ( §4.2) selects the most appropriate answer from these candidates. however, if the ranker lacks sufficient confidence in its choice, the question is passed to the generative search ( §4.3). this component employs a large language model (llm) that is prompted to generate an answer based on relevant context from a web search api. our approach can be followed to develop your own factual qa system, capable of effectively assisting users with both domain-specific and opendomain inquiries. our contribution consists of two key aspects: earlier qa systems were based on bm25 @xcite , a ranking function that estimates the relevance of documents to a given search query. subsequently, bm25f @xcite was developed, which enhances text relevance calculation by considering query terms across multiple specific fields. the bm25f class integrates the bm25 scores of the query term in these various fields. the introduction of word embedding methods sparked the development of pipelines where proximity in the latent space serves as a metric for semantic retrieval. @xcite evaluated various text embeddings, including elmo @xcite and glove @xcite , for question and paragraph embeddings. the transformer encoder-based architecture @xcite led to the introduction of multilingual e5 @xcite text embeddings, which are trained through a multi-stage pipeline. when both the question and the context are provided (in reading comprehension), bert-based models have brought about substantial improvements @xcite . in our pipeline, the local knowledge-based component relies on embeddings derived from the multilingual e5 model. knowledge base question answering (kbqa) relies on knowledge graphs (kgs) to find the correct answer. kbqa involves applying two main approaches: semantic parsing (translating the question into an executable logical form) and retrievalbased methods (inferring answers from kg). the wqaqua @xcite pipeline starts with knowledge base grounding, after which the possible sparql queries are constructed that return non-empty answers when executed. @xcite outlines a generative model for qa that draws on textual content and knowledge graphs to uncover supportive information. @xcite proposed an algorithm for extracting subgraphs from a kg, based on question entities and answer candidates. then the transformer-based model is provided with linearized subgraph to generate response. llms, when equipped with retrieval augmented generation (rag), perform exceptionally well across a variety of tasks @xcite . rag improves the accuracy of llms' outputs by retrieving supplementary knowledge through specialized retrievers. this process enriches the prompts given to llms with relevant information from retrieved documents, enabling them to generate more precise and detailed content. @xcite proposed a method to select a context for rag-based system that is retrieved from different sources, including kgs. the selection method is based on uncertainty estimation (ue) techniques. ## preprocessing the quality of search can be affected by the format of users' requests, which can stem from diverse sources such as chatbots, search bar widgets, or voice-based interfaces. these requests can be typed in lowercase, mixed case, or have capitalization errors. this is significant because the e5 @xcite embedder model we employ is case-sensitive. the optimal performance is achieved when the user's input is orthographically correct. therefore we develop the bert-based @xcite truecase component that fixes word casing. we train the model in token classification manner. for training, we utilize the tatoeba dataset @xcite . ## combining search pipeline general overview of our system can be found in figure 1 . once an input query (a question requesting a fact as an answer) is received by our system, it's processed in the following way: ## system performance we develop domain-specific validation sets that simulate probable user queries. additionally, we have a comprehensive golden set comprising 1,600 factual questions, along with their corresponding the usefulness metric is our key product metric for evaluating qa system responses. although we define a "useful" answer simply as one that addresses the posed question, there are many nuances (some of the are highlighted in appendix d which provides a detailed description of each usefulness value). the score can take values of 0, 0.5, or 1 for a single sample, and then these values are averaged over the validation set to get the final usefulness. preliminary outcomes of the current system are depicted in figure 2 . the no in figure 2b , one can observe the absence of the web search component's share, as we are unable to present web search results to the user through the audio channel; this is due to safety concerns and the lack of informativeness. furthermore, it's important to note that, by definition, a qa system should respond to all posed questions. "no information" cannot be considered a correct answer. even in cases where a user asks an nonsensical question that cannot be answered properly, the llm should provide a response indicating that the question does not have a definitively correct answer. therefore, our objective is to minimize the share of "no information" responses, as these responses offer no value or benefit to the service and are essentially useless. upon optimizing the qa system validation, we focused on accelerating the service and migrating it to the triton inference server foot_4 . by incorporating dynamic batching @xcite for the local search module and continuous batching @xcite for the llm, along with asynchronous external search engine queries for context, we achieved a 700% increase in rps and a 500% reduction in response time. currently, we are pursuing llm quantization for further efficiency. ## conclusion in this work, we introduced ears, a productionready factual question answering system. answers can be derived either from a local knowledge base or generated by an llm using context obtained from the web search api. we outlined the comprehensive workflow for addressing factual questions, making it reproducible for any language. the cornerstone of the pipeline is the ranker, which enhances retrieval quality by 23%. incorporating the llm boosts the quality of our chatbot's spoken answers by 92.8%. furthermore, we developed a suite of automatic metrics to reduce reliance on human annotators.