Fast and Light-Weight Answer Text Retrieval in Dialogue Systems (2022)

Abstract: Dialogue systems can benefit from being able to search through a corpus of text to find information relevant to user requests, especially when encountering a request for which no manually curated response is available. The state-of-the-art technology for neural dense retrieval or re-ranking involves deep learning models with hundreds of millions of parameters. However, it is difficult and expensive to get such models to operate at an industrial scale, especially for cloud services that often need to support a big number of individually customized dialogue systems, each with its own text corpus. We report our work on enabling advanced neural dense retrieval systems to operate effectively at scale on relatively inexpensive hardware. We compare with leading alternative industrial solutions and show that we can provide a solution that is effective, fast, and cost-efficient.

## introduction dialogue systems such as amazon lex, ibm watson assistant, or microsoft azure bot service operate mainly through intent detection. a subject matter expert (sme) creates a dialogue system by defining a fixed set of intents that a user might have and provides scripted responses for each of them. machine learning models are adopted to identify the user intent and route to the corresponding dialogue nodes and responses. it usually takes a considerable amount of human curated data to train an intent detection model. adding features or content to a dialogue system would require adding new intents and training the model all over again. to alleviate such limitations, an alternative approach to enabling the same user experience is to have a system automatically search through a corpus of text to find relevant responses to each user request. one motivation behind this approach is to replace the intent detection, so to make it flexible, quicker, and easier to set up and maintain a dialogue system, because the sme does not need to enumerate all the intents they expect a user to have. applying text retrieval in such a system can also complement intent detection: intent detection can handle the anticipated user needs and text search can handle unanticipated requests. in either case, the value of the text retrieval depends critically on how accurate it is. another big advantage of the text retrieval approach is that it could provide reasonable accuracy even when there is little or no labeled training data. a popular line of text retrieval methods is matching sparse terms and weighing those matches by how frequent they are in the document being found and how infrequent they are in the corpus. for example, bm25 @xcite is an extremely popular algorithm of this sort that provides an excellent balance between accuracy and computational cost. however, in the recent years, research has shown that neural network solutions can provide superior accuracy to sparse term matching approaches like bm25. in particular, neural dense retrieval approaches such as dpr @xcite and colbert @xcite have achieved outstanding results in retrieval and re-ranking even at zeroshot setting, and further boosted accuracy when in-domain training data is available. neural dense retrievers achieve high accuracy but usually involve models with hundreds of millions of parameters and require long training time. however, in real-world scenarios, a cloud service sometimes supports many different deployed dialogue applications at the same time, hence needs to be able to process requests for all of those applications at the same time. this can be extremely expensive if each application has a model that demands an enormous amount of memory and/or processing power when handling requests. a practical system needs to be able to balance the benefits of a sophisticated model with the costs of running it. furthermore, dialogue system administrators want to be able to add training data to an existing, deployed system and start getting improved results quickly. we explore various approaches to addressing these requirements, including scaling techniques such as distilled encoders and dimension reduction, self-directed iterative learning and asynchronous learning. we conduct thorough experiments on our datasets to benchmark these approaches, and show that we have emerging technology that achieves accuracy that is competitive with state-of-the-art research solutions with substantially less expensive resource requirements. ## related work in information retrieval (ir), popular relevancy algorithms such as tf-idf and bm25 @xcite match keywords with an inverted index and compute relevancy using heuristic functions. together with pre-processing methods such as stemming and removal of curated stop words, sparse-term-based retrieval works fairly well without training, and is widely adopted in real world applications. dense passage retrieval @xcite @xcite @xcite has gained a lot of attention lately with applications extending beyond retrieval tasks into areas including open-domain question answering, language model pre-training, fact checking, dialogue generation (e.g., rag @xcite , realm @xcite , multidpr @xcite , kilt @xcite , con-vdr @xcite , rocketqa @xcite ). in dense passage retrieval, the query q and each passage p are separately encoded into dense vectors, and relevance is modeled via similarity functions such as dot-product. recent works improve efficiency and effectiveness of single-vector dense retrieval systems, including model distillation @xcite , hard negative sampling @xcite , etc.. another line of related work is cross-encoder document ranking @xcite @xcite . query-document pairs are concatenated and sent through transformer-based encoders, an additional layer on top of the encoded representation is adopted to produce a relevance score of the docu-ment to the query, which is then used for ranking. @xcite and @xcite benchmark intent detection models on intent detection datasets such as clinc150 @xcite where sufficient training examples exist for each intent. on the other hand, our use case focuses on the scenarios where answer text is available but training examples are insufficient. the task we are dealing with is a real-world use case of answer text retrieval in an faq dialogue system. formally, we have a corpus p of answer text snippets @xcite . for each answer text passage p in p , we have a limited number of associated example queries q p . the system is expected to retrieve the most relevant answer text passage for each incoming user query q. it needs to deliver a good latency, and work well when the size of q p is small, i.e., when there are not many training examples available. most importantly, the resource consumption must be kept low. to address the use case, we start with two leading industrial solutions as baselines: the two aforementioned approaches each have their own strength. the classifier approach leverages query examples and machine learning, while the sparse-term-based retrieval approach utilizes answer text but not query examples, and does not involve training. we seek to get the benefits from both approaches. one option is to capture the cross-attention between query q and each candidate passage p by feeding ⟨q, p⟩ pair to a transformerbased encoder and learn over the encoded output @xcite @xcite . however, due to the need to cross-encode the incoming query together with each passage, this approach requires more computation by orders of magnitude and is not practical for our task setting. dense passage retrieval methods @xcite @xcite @xcite @xcite have gained a lot of attention lately and achieved state of the art results on various retrieval and ranking datasets. dense retrievers are efficient compared to other neural methods such as transformer-based cross-encoder models: passages are encoded and indexed offline, at inference time only the query needs to be encoded once; also they leverage ann (approximate nearest neighbor) algorithms to efficiently search for relevant dense vectors. dense retrievers are effective compared to traditional sparseterm-based ir methods such as bm25: they are not restricted by rigid keyword matching; they use transformers to encode both the queries and the passages, and benefit from transfer learning from large retrieval/re-ranking datasets. being effective and efficient, neural dense retrievers make an ideal solution for our task setting and requirements. ## approach we first briefly overview the work in neural dense retrieval and talk about the gaps from practical usage in section 4.1. in the remainder of section 4, we explain our efforts applying dense passage retrieval to the task and further reducing response time, memory footprint, and training time. ## conclusion we report on our work on enabling advanced neural dense retrieval systems to operate effectively at scale on relatively inexpensive hardware. on our real-world use case and datasets from dialogue systems, we show that we can provide a solution that achieves accuracy that is competitive with state-ofthe-art research solutions with substantially less expensive resource requirements and shorter response time.