Multi-Tenant Optimization For Few-Shot Task-Oriented FAQ Retrieval (2022)

Abstract: Business-specific Frequently Asked Questions (FAQ) retrieval in task-oriented dialog systems poses unique challenges vis à vis community based FAQs. Each FAQ question represents an intent which is usually an umbrella term for many related user queries. We evaluate performance for such Business FAQs both with standard FAQ retrieval techniques using query-Question (q-Q) similarity and few-shot intent detection techniques. Implementing a real-world solution for FAQ retrieval in order to support multiple tenants (FAQ sets) entails optimizing speed, accuracy and cost. We propose a novel approach to scale multi-tenant FAQ applications in real-world context by contrastive fine-tuning of the last layer in sentence Bi-Encoders along with tenant-specific weight switching.

## introduction business-specific frequently asked questions (faq) form an important part of many taskoriented dialog systems today. business faqs exhibit some common characteristics which differentiate them from community-based faqs. we show few examples of such faqs in since each question within an intent category represents it loosely, having a single question for *work done while the authors were working at verloop.io 1 https://github.com/verloop/few-shot-faqir an intent poses a challenge for intent detection. simple paraphrases do not suffice and therefore, 5-10 variations for each faq question representing the intent is sourced from domain experts. common approaches in faq retrieval include query-question similarity and query-answer entailment @xcite @xcite . query-question (q-q) methods focus on similarity between a user's query and a question to retrieve the relevant answer. query-answer (q-a) methods predict the relevance between the query and an faq answer to re-rank the results obtained via q-q. for faq retrieval, we leverage q-q similarity. the q-a entailment method is infeasible in some faq sets due to the generic nature of answers for several different questions, owing to little match between the q-a pair and responses being modeled as system action instead of text. we also prefer a single model based approach over stacked models such as retriever-reranker models @xcite as inference latency gets compounded in such systems. we compare several q-q retrieval approaches using pre-trained models and fine-tuned embeddings. we also model the problem as a few-shot intent detection and evaluate classifier based techniques. from an industry specific dialog solutions perspective, the approaches need to balance accuracy, real time inference latency and costs while having the ability to scale for multiple tenant (faq sets) requirements. our contribution is towards implementing an faq retrieval system for business faqs in real-world contexts. we describe a low-cost approach to deploy multi-tenant faq applications at scale while optimizing for inference latency and accuracy. we also evaluate different methods and models against few-shot intent detection and conversational datasets. ## related work the models based on transformer architectures have enabled the usage of transfer learning for various tasks such as text similarity @xcite . @xcite introduced an approach to combine the query-question similarity from tsubaki and query-answer relevance from bert. @xcite evaluated different providers on three intent recognition datasets, created using queries received by chatbots from real users. in a few-shot setting, recent approaches leverage the transformer architecture. @xcite demonstrated that using a dual sentence en-coder as base model works better in low-resource intent detection tasks as compared to bert. @xcite proposed the use of contrastive learning to improve the performance of bert based classifiers in few-shot intent detection tasks. pre-training methods for creating transferable language representations is a common approach. in-domain pre-training further increases the adaptability of the model to domain related downstream tasks. specifically for few-shot setting, fine-tuning pre-trained models with supervised learning is found to be effective @xcite . cross-encoder approaches with bert @xcite and roberta @xcite models have achieved sota scores using sentence pair training. training is done with question pairs. during inference, question pairs are created using query and train questions. these pairs are scored by a classifier to predict the final label. bi-encoder models have the advantage of being able to cache the representations and hence, are highly efficient during inference @xcite . the research on stilts @xcite found the advantages of intermediate training on data rich supervised tasks, giving pronounced benefits in data constrained regimes. @xcite discusses the gains achieved via transfer learning using pretrained sentence encoders as a retriever model. cross-encoder models entail a higher latency as all the question pairs are required to be classified during inference. @xcite explores usage of the classifier model as a re-ranking model following a bi-encoder based retriever to reduce inference time. we evaluate our faq retrieval use case with simple baseline approaches, in-domain pretrained models, classifier based approaches, cross-encoders and bi-encoders in a few-shot setting. ## datasets we choose datasets that are similar to our realworld tenant faqs and perform experiments on publicly available intent detection datasets for taskoriented dialogues. we use datasets from hint3 (curekart, powerplay11 and sofmattress) @xcite which reflects real-world faq intents and user queries. we also use dialoglue datasets @xcite ) -hwu64 @xcite , clinc150 @xcite and banking77 @xcite . dialoglue datasets further provide train sets for data constrained regimes with 5 and 10 samples per class. similarly, hint3 also contains subset variants of datasets which are created by retaining only samples whose entailment score using the elmo model was less than 0.6. we use the fewshot subsets for dialoglue and the full as well as subset variant of hint3 datasets. these datasets are suitable for few-shot analysis. all the datasets have separate train and test sets and we report our evaluations on the test sets. the clinc150 and hint3 datasets have separate in-scope and out-ofscope queries and we use them to evaluate oos performance. the dataset size, intent and domain distribution is listed in appendix a. ## methodology in this section, we elaborate on our experiments for model training and deployment. we compare different approaches of few-shot faq retrieval and describe our deployment strategy for the same. ## experimentation approaches fine-tuning for all bi-encoder and cross-encoder models is done using question pairs as elaborated in section 4. based on the training method outlined by @xcite , we train for a higher number of iterations to offset any fine-tuning instability. for the bi-encoders, we fine-tune only the final layer of the model using a contrastive or an online triplet loss. the pre-trained bert and convbert classifier models were fine-tuned using a softmax cross-entropy loss. the cross-encoder was also fine-tuned with the same approach using question pairs with a binary cross-entropy loss. for predicting the query label, we create q-q pairs limited to 5 questions per intent. the model predicts a score for these combinations, where the label for the question with the highest score is considered to be the predicted label. we trained all models for 10 k iterations with a learning rate of 2e-5, batch size of 16, adamw optimizer with 10% linear warm up and gradient normalization. ## evaluation metrics for evaluation, we consider a combination of faq retrieval metrics along with intent detection metrics. similar to the metrics used in @xcite , we consider top-k accuracy (same as success rate referred in the paper), mrr@k (mean reciprocal rank), ndcg@k (normalized discounted cumulative gain) and map@k (mean average precision). from an intent detection perspective, we also evaluate out-of-scope accuracy across different thresholds. we use accuracy as the metric for evaluation of oos queries. in case the top predicted label score is less than a specified threshold, then the top-k results are shown as suggestions to the user. metrics like mrr, ndcg and map put emphasis on the ordering of the final responses and hence are useful for evaluating the effectiveness of this approach. ## results from our experiments, we find that the finetuned bi-encoder models perform the best across all datasets. all the bi-encoder embedding ap-proaches significantly outperform the other approaches in few-shot setting in spite of being constrained with the last layer fine-tuning strategy. for hint3 datasets, we use the benchmarking results as reported by @xcite for comparison. we find that the fine-tuned bi-encoder shows better performance than all benchmarked chatbot solutions on both full and constrained datasets. we noticed improvement while finetuning the pre-trained bi-encoder models using the question pair approach. we also note that online triplet loss based fine-tuning performed marginally better as compared to contrastive loss for all the base models other than mpnet model on haptik dataset. we observe that the question pair / triplet training strategy mitigates the effect of fewer samples. reducing sample sizes of the q-q pairs from 200 k to 50 k balanced samples hurts the performance across all datasets. fine-tuning the task adapted pre-trained models shows much better results in comparison to the base pre-trained models in most of the dialoglue datasets but the gains are either less or deteriorate for haptik datasets. in haptik datasets, the median examples per intent is quite low as compared to dialoglue, with some intents having only 1 sample. there would be no triplets formed for such samples. we also see a wide variation in the number of examples per intent ranging from 3 to 95 in the curekart dataset. this would reduce the number of triplets for such intents. while cross-encoder strategies are supposed to be superior, we find that the cross-encoder finetuning suffers due to the restrictive training strategy and performs sub-par when compared to the bi-encoders. the bert based classifier also shows poor performance which appears to degrade on datasets with more classes. in comparison to training bert model with all layers unfreezed as reported in the arora et al. (2020), we see that the last layer finetuning strategy severely impacts the model performance. we also observe that models pre-trained on indomain data such as convbert is superior to base bert even as a feature extractor. interestingly, we see that even baseline approaches with bm25 and tf-idf show better results than pre-trained bert in such settings. fine-tuning the convbert model on the supervised classification task shows an improvement over the pre-trained model. if we look at the oos accuracy for the fine-tuned minilm-l6 model, the similarity scores are normally high and lower thresholds do not have much impact. we further observe that each dataset has different thresholds where the best trade-off between oos and in-scope accuracy is achieved. the top-3 accuracy of minilm-l6 on all datasets including the challenging powerplay11 dataset is above 70%. we choose the fine-tuned bi-encoder models based on superior performance with our training strategy. ## deployment smaller models give us more scalability at the cost of accuracy while larger models tend to have lower throughput and higher memory requirements. the model selection is done based on the specific business need which dictate these three metrics. we choose minilm-l6 as it handles higher load at significantly lower latency as compared to the best performing model mpnet, with a slight trade-off in accuracy. ## conclusion we evaluated various methods for retrieval of business faqs by modeling the problem as an faq retrieval and a few-shot intent detection task. we proposed a realistic multi-tenant deployment solution with trade-offs in accuracy while balancing for cost and latency. our last layer weight-switching strategy works well where the model has to be finetuned on tenant specific tasks. we observed that fine-tuned bi-encoder embeddings work best in a few-shot setting even under a constrained finetuning strategy.