DTAFA : Decoupled Training Architecture for Efficient FAQ Retrieval (2021)

Abstract: Automated Frequently Asked Question (FAQ) retrieval provides an effective procedure to provide prompt responses to natural language based queries, providing an efficient platform for large-scale service-providing companies for presenting readily available information pertaining to customers’ questions. We propose DTAFA, a novel multi-lingual FAQ retrieval system that aims at improving the top-1 retrieval accuracy with the least number of parameters. We propose two decoupled deep learning architectures trained for (i) candidate generation via text classification for a user question, and (ii) learning fine-grained semantic similarity between user questions and the FAQ repository for candidate refinement. We validate our system using real-life enterprise data as well as open source dataset. Empirically we show that DTAFA achieves better accuracy compared to existing state-of-the-art while requiring nearly 30× lesser number of training parameters.

## introduction faq retrieval system provides a natural language interface for querying faq collection and is increasingly becoming popular with large-scale service-providing companies. further, with the advent of personal assistants (like xiaoice, siri, alexa, google assistant, etc.), these "virtual agents" can provide answers and help users solve routine tasks by an additional interface to faqs, hotlines and forums -enabling a natural interaction with users @xcite . faq retrieval is a challenging task, majorly attributed to the fact that question-answer texts are short, making it harder to bridge the lexical and semantic gap between a user query and faq questions due to limited context @xcite . further, in certain cases, precise understanding of the user questions might be difficult due to informal representations, domainspecificity, abbreviations, and formal-colloquial term mismatches @xcite . in addition, faq retrieval systems should be able to handle both keyword as well as short span "natural language" questions. given the predominantly "customer-centric" nature, such systems generally demand higher precision and interpretability compared to traditional information retrieval methods. challenges. in modern interactive applications, the fluidity of natural language based humancomputer interactions provides an additional metric to capture quality of user experience. for example, consider a voice-based faq platform interfaced via a personal assistive system. in such cases, providing the user with the top-k "matching" results (from the faq platform) to choose from, impedes natural fluidity of interaction. an intelligent system should be able to automatically understand and/or infer the context, meaning and relevance to provide the best matching faq to address the user's concern. hence, in such scenarios the top-1 or "one-best" accuracy tends to precisely capture the quality-of-service. further, note that modern enterprises have global footprints with diverse product and service portfolios, and hence such faq systems should also be able to handle the challenge of multi-lingual customer base associated with globalization. unfortunately, "multi-linguality", particularly in faq retrieval systems, has been underaddressed in the literature; although being crucial to organizations for faster scaling of operations to geographically distributed markets. in this work, we propose the decoupled training architecture for faq retrieval (dtafa) framework geared towards enhanced "one-best" accuracy to alleviate the above challenges in modern interactive application settings. problem statement. faq retrieval engines attempt to understand the underlying intent of user questions and retrieve the most related documents or answers that may contain correct information @xcite . formally, consider @xmath0, a 1 ), • • • , (q n , a n )} to be a pre-curated collection (or repository) of questionanswer pairs, where q denotes a question related to the domain, and a represents the corresponding answer. given a user query q, the task then is to return related work and contributions dtafa provides a novel learning framework for multilingual faq retrieval with enhanced top-1 recommendation accuracy (or "one-best" accuracy), geared towards improving the overall quality of interactive automated customer experience. as shown in figure 1(b), dtafa leverages two "decoupled" deep learning architectures trained independently. the main fundamental intuition behind dtafa is simple but yet found to be effective; to decrease the search space first via a simple classification module which does not take into account the semantics of the label and then aiming to select from the reduced search space the most semantic similar to the label context give the label has enough context. prior art focuses mainly in dealing with the faq retrieval problem as either text classification or semantic textual similarity problem. for text classifi-cation, we have seen set of large-scale transformerbased pre-trained language models (plms) such as @xcite , roberta @xcite , and xlm @xcite . these plms are fine-tuned using task-specific labels and created new state of the art in many downstream natural language processing (nlp) tasks including faq retrival problems or more broadly text classification @xcite . on the other side, there have been several prior work that relies in measuring semantic similarities for faq-based qa such as matchpyramid @xcite , iwan @xcite , and pair2vec @xcite and more recently using q-to-a matching using an unsupervised way, and further introducing a second unsupervised bert model for q-to-q matching @xcite . however, adapting plm text classification based approaches do not take label textual semantics into account which they have have some useful lexical information that can be used for improving the system accuracy. in addition, these architectures impacts the inference time when deployed in production due to the huge number of model parameters. semantic textual similarity based methods usually do not scale when the number of faq pairs increases as there will be a need for performing matching to every pair to extract the corresponding answer. in that sense, we propose dtafa with an aim to solve such challenges relying on two decoupled deep learning architectures trying to leverage the advantages of each of the above approaches in a hybrid approach yielding to more practical implementation. our contributions, in a nutshell, are: (i) we propose dtafa, a novel framework for multi-lingual faq retrieval that captures lexical and semantic similarities and relationships among user queries, faq questions and their paraphrased versions to understand fine-grained differences to provide enhanced "one-best" accuracy; ## dtafa framework we next describe the detailed architecture and working of the different components in dtafa shown in figure 1(a) . dtafa hinges on two decoupled deep learning architecture based modules. the first module is trained to learn latent lexical relationships between the faq questions (qu) and their paraphrased variants (eq) for generating candidate top-k most relevant or similar questions within the faq collections. the top-k candidates are then fed to the second module, a probabilistic siamese lstm-based architecture, to capture finegrained differences in semantic context between the questions and their possible variants (proxies for real user queries) for further improving the accuracy of the final top-1 recommended result. to support multi-linguality and zero-shot learning for scaling to other languages, both modules in dtafa are based on laser sentence embeddings @xcite which are language-independent representations -similar sentences are mapped onto nearby vector spaces (in terms of cosine distance), regardless of the input language. however, instead of training using only one language and performing zero-shot learning on the others (the default setting @xcite )), we use three languages, namely english, spanish and chinese, for training across the components. we present dtafa in the context of faq retrieval, observe that it can easily be extended to other classification problems, where the textual labels contain enough semantic information. ## experimental setup in this section, we describe the experimental setup for comparing the performance of dtafa against state-of-the-art approaches. we consider the "onebest" accuracy, measured in terms of precision-at-rank-1. all models trained using nvidia titan rtx gpu. ## empirical results this section reports the empirical results obtained for dtafa (both monolingual and multi-lingual settings) as compared to the competing approaches described previously. to capture "one-best"' accuracy, we report the precision-at-rank-1 (p@1) performance, which captures the fraction of the top-1 answer retrieved by the system that are relevant to the user query. this indirectly captures the quality-of-service for speech-based assistive platforms. dtafa is currently in pre-deployment phase in our organization. ## conclusion we propose a novel multi-lingual faq retrieval framework (dtafa) for improving the accuracy of top-1 results ("one-best" performance). our framework combines the advantages of both classification and semantic textual similarity approaches in one single framework and hence, improves faq retrieval problem accuracy while keeping number of parameters less compared to other state-of-the-art approaches making it more practical approach in an industrial context. experiments on real enterprise data as well as open source dataset across 13 languages demonstrate the efficacy of our system over existing tradional approaches, both in monolingual and multi-lingual settings. we show dtafa to robustly generalize to multiple languages based on "zero-shot" transfer learning, providing upto 40% accuracy improvement on distant languages along with 30× lesser number of trainable model parameters.