Dense Template Retrieval for Customer Support (2022)

Abstract: Templated answers are used extensively in customer support scenarios, providing an efficient way to cover a plethora of topics, with an easily maintainable collection of templates. However, the number of templates is often too high for an agent to manually search. Automatically suggesting the correct template for a given question can thus improve the service efficiency, reducing costs and leading to a better customer satisfaction. In this work, we propose a dense retrieval framework for the customer support scenario, adapting a standard in-batch negatives technique to support unpaired sampling of queries and templates. We also propose a novel loss that extends the typical query-centric similarity, exploiting other similarity relations in the training data. Experiments show that our approach achieves considerable improvements, in terms of performance and training speed, over more standard dense retrieval methods. This includes methods such as DPR, and also ablated versions of the proposed approach.

## introduction a very common practice to make customer support more efficient is the use of templates for replies. the templates of replies are designed by customer support administrators to systematise the reply to frequent requests. then, given a customer's request, an agent can pick a response from within a collection of predefined templates, this way saving time when replying to repetitive questions. besides improving the throughput of human agents, the use of templates also assures uniformity in the handling of different customers, as requests with the same underlying problem should be handled with the same type of reply. however, customer support centers can have hundreds of templates, and finding the best template for a question is not an easy task, particularly for unexperienced agents. automatically * this work was developed at cleverly and zendesk, under the context of tiago mesquita's master of science (m.sc.) thesis at ist, university of lisbon. sorting and suggesting customer support templates @xcite @xcite can facilitate agent's work, reducing reply times, accelerating the learning curve of new agents, helping agents to focus on more added valued tasks, and overall providing a better customer support at reduced costs. the recent advances in large pre-trained language models @xcite , together with their successful use in question-answering @xcite and information retrieval @xcite @xcite , motivate the use of dense retrieval for template selection. dense retrieval can be used to rank instances from the template collection, facilitating the selection of the correct template. still, template ranking has specific characteristics when compared with more common retrieval scenarios in the literature: i) we have a strict many-to-one relation between queries and templates, in contrast to other common retrieval tasks @xcite ; ii) the collection of templates is relatively small and generally in the order of hundreds, although also dynamically updated over time; iii) the length of the queries (emails with questions from customers) tends to be relatively long. given real-time and computation constraints in the template suggestion problem, we focus on biencoder models @xcite , which at prediction time only need to compute dense representations of queries and make fast comparisons with pre-computed representations of template candidates. we discard cross-encoder models that, despite often achieving higher retrieval performance @xcite , can have problems processing long queries and/or documents, and cannot take advantage of pre-computed template representations, involving higher computational costs that scale with the number of template candidates, and being often unsuitable for real-time applications. still, our contributions are model independent, being also applicable to cross-encoders. specifically, we make the following main research contributions: experiments with real-world customer support datasets show that both the in-batch sampling strategy and the expanded loss lead to improvements, in terms of template suggestion and training speed. ## related work template retrieval for customer support has seen limited research in recent times. most previous work has addressed the task as template classification with simple machine learning approaches (e.g., support vector machines or naïve bayes) on top of representations obtained from bags-of-words @xcite or tailored pattern matching @xcite . a combination of retrieval and generative approaches is explored by @xcite , but without taking advantage of modern transformer-based pre-trained language models. to the best of our knowledge, public literature on the topic has not explored recent advances in dense retrieval. most of the recent dense retrieval methods follow a bert-based dual-encoder architecture and use a similarity function (e.g., cosine similarity) to produce ranking scores @xcite . the simplicity of the similarity function is crucial, allowing efficient retrieval through recent developments in approximate nearest neighbour (ann) search, such as faiss @xcite . these methods enable search speeds comparable with simpler sparse retrieval methods (e.g., bm25 @xcite ), whilst retaining better performance in most scenarios. given the simplicity of dual-encoder architectures, most research has focused on improving the training procedure, namely by careful selection of the querydocument pairs. for each query, the model should maximize similarity with all related documents (i.e., positives), whilst minimizing it for all unrelated documents (i.e., negatives). most approaches have focused on the problem of selecting negative documents, usually falling under one of two categories: 1) efficient batching techniques, or 2) techniques that prioritize batch generation but generally sacrifice training efficiency. in the first category, methods aim to maximize the amount of negatives available within the batches. the most efficient way to achieve this is by sharing negatives between all queries in the batch, an approach known as in-batch negatives @xcite . more recent studies have proposed sharing negatives between multiple gpus, allowing for massive batch sizes under parallel model training @xcite . although the previous techniques maximize the number of negatives seen during training, most of the instances are easily distinguishable, providing weak contributions to the loss function (i.e., easy negatives). studies under the second category focus on the careful selection of negatives per query, looking for those that are useful for learning (i.e., hard negatives). the earliest approaches leveraged other retrieval methods to pool hard negatives, namely bm25 @xcite , picking highly ranked although irrelevant documents @xcite . despite being effective at picking static hard negatives, these approaches fail to adapt, as the model learns to rank the instances accordingly. ance @xcite addresses this issue by using the model itself to pool hard negative examples, dynamically adjusting the selection as the training progresses. in practice, dynamically generating and indexing embeddings for large document collections is costly, requiring separate gpus to periodically maintain and refresh the index with prior checkpoints, whilst training in parallel. ltre @xcite and later adore @xcite further refined these ideas, by freezing the weights of the document encoder and eliminating the need to refresh the index. although most approaches generally focus on a single category, some have tried to leverage techniques from both. dpr @xcite was one of the first studies to explore this idea, combining randomly pooled in-batch negatives with bm25 hard-negatives per query. recently, star @xcite took the idea even further, by using static hard negatives pooled from the pretrained model, but sharing them between all queries in the batch, similarly to in-batch negatives. results showed that the combination provides an effective tool for stabilizing the biases introduced by the use of static negatives. our technique also combines ideas from both categories, adapting in-batch negatives to the specific scenario of template retrieval, and performing hard negative sampling from all the in-batch negative examples. most previous studies on dense retrieval have generally also considered loss functions that enforce query-centric similarity relations, as these are explicitly related to the retrieval task. however, as shown in pair @xcite , models may benefit from exploring passage-centric similarity relations, potentially improving the representations. more recently, @xcite proposed dance, which showed improvements by considering a loss function that combines query retrieval and document retrieval tasks. in our template retrieval scenario, these ideas are further explored, by taking into account all four possible relations between pairs of queries and templates. ## simple dense template retrieval we formally define the problem of template retrieval as follows: given a query q, the model must retrieve, from a collection of templates, the template t that better answers the query. ## improved dense template retrieval to improve on the method outlined in the previous section, we propose two innovative contributions. the first relates to the in-batch sampling strategy, whilst the second refers to an expanded loss function that exploits different similarity relations for queries and templates. ## conclusions this paper discussed challenges associated with retrieving templates for answering customer support questions, proposing a dense retrieval framework to address the task. this framework features innovative contributions in terms of (a) extending in-batch negatives to support unpaired sampling of queries and templates, and (b) a novel loss function that considers more similarity relations from the training data within each batch. experiments on two different datasets of customer support interactions attest to the improvements brought forward by the proposed ideas. for future work, we plan to adapt and test the proposed techniques in other tasks that involve unbalanced corpora and large texts, such as general faq retrieval or question answering @xcite .