Retrieval Based Response Letter Generation For a Customer Care Setting (2022)

Abstract: Letter-like communications (such as email) are a major means of customer relationship management within customer-facing organizations. These communications are initiated on a channel by requests from customers and then responded to by the organization on the same channel. For decades, the job has almost entirely been conducted by human agents who attempt to provide the most appropriate reaction to the request. Rules have been made to standardize the overall customer service process and make sure the customers receive professional responses. Recent progress in natural language processing has made it possible to automate response generation. However, the diversity and open nature of customer queries and the lack of structured knowledge bases make this task even more challenging than typical task-oriented language generation tasks. Keeping those obstacles in mind, we propose a deep-learning based response letter generation framework that attempts to retrieve knowledge from historical responses and utilize it to generate an appropriate reply. Our model uses data augmentation to address the insufficiency of query-response pairs and employs a ranking mechanism to choose the best response from multiple potential options. We show that our technique outperforms the baselines by significant margins while producing consistent and informative responses.

## introduction in modern business operations, customer care services are essential to support customers needing product information, making complaints, and in general, positively addressing their expectations. this support service plays a vital role in ensuring a good customer experience and is a key factor in developing goodwill. while non-specific, general knowledge about a product can now be conveniently retrieved through a web search, the exchange of specific information naturally entails a conversation between the customer and an agent who represents the organization. traditionally, this task has been carried out by a trained human through chat or email exchanges. however, doing this manually at scale takes an enormous human effort. the process is time-consuming, laborintensive and error-prone given the massive volume and diversity of customer queries. automation of the response generation process can go a long way toward solving this problem. unfortunately, the rule-based systems that are in existence today struggle to capture the linguistic complexity of typical communications. recently, the advent of transformer-based pretrained language models such as bert @xcite , gpt-2 @xcite , t5 @xcite ) has brought about substantial progress in understanding and generating fluent text. nevertheless, task-oriented dialog (tod), a process that aims to assist a user to complete a certain task through response generation, is yet to be mastered due to the challenges in producing text that is informative and relevant to the prompt @xcite . the task is especially difficult because of the scarcity of annotated datasets needed to train a supervised model. furthermore, creating an effective customer feedback system has additional challenges. in the existing and very popular tod datasets @xcite , user utterances are usually fact-finding queries annotated with slot labels (e.g. query: 'find a park near area51', slot-values: {destination: 'park', close_to: 'area51'}). a tod framework @xcite first identifies the slots and then uses the slot-value pairs to retrieve facts from a knowledge base to reply. unfortunately, these frameworks do not realistically address customer-care automation tasks where neither is the user prompt labeled with slot tags nor is there a knowledge base with relevant facts. fur-thermore, a user prompt may not be limited to an inquiry about facts but may also include a complaint, suggestion, compliment, request, etc. in this work, we factor in these challenges and present a response generation framework that automatically produces and ranks response letters addressing customers' queries or feedback, with a minimally annotated dataset. the contribution of this work is in two areas: our response generation framework is to be deployed in production as a part of an evolving pipeline for automating the customer service process. it will initially serve as a suggestion system to collect feedback from real human agents and is then expected to progress to enabling increased levels of automation. ## dataset we use two proprietary datasets from the restaurant and adhesive tape industry, named dinecare (dc) and tapetech (tt), consisting of 8448 and 14938 unique email exchanges between customers and agents respectively. each exchange contains a case id, product and reason codes of the service, the customer query letter, and the human agent's response letter. examples are included in and may lack textual description. the reason code stands for the type of customer query; therefore we observe similar responses across queries having the same reason code. ## framework in this section, we first describe a potential baseline approach for response generation and then the retrieval-guided response generation framework. ## experiment details we used a small gpt-2 model of 124m parameters provided by huggingface @xcite . grid search was used to tune the hyper-parameters to the following set of optimal values: {weight decay: 0.1, warm-up steps: 1e2, gradient accumulation steps: 16, learning rate: 5e-4, dropout rate: 0.1, epoch: 5, optimizer: adam, γ = 0.4, τ = 0.6}. the training took around 1 hour for the dc dataset and 2.5 hours for the tt dataset on an nvidia tesla v100-sxm2-16gb (gpu device). after training, 4 responses were generated for each query using 4 decoding combinations: 1. (top-k with temperature), 2. (top-p with temperature), 3. (top-k,top-p) and 4. (top-k,top-p with temperature) where @xmath0, @xmath1.8 and @xmath2.7. for preprocessing and evaluation, we use nltk and nlg-eval @xcite . ## related work research in machine-generated response systems originated at least four decades ago. at the end of the last century, @xcite introduced a concept of utterance recognizer and response generator for task-oriented dialog (tod) systems. the past few years have witnessed several response generation models, particularly using neural approaches to conversational ai. recently, combining the idea of gpt @xcite and transfer-learning based training scheme, wolf et al. (2020) produced improved dialog systems. similarly, zhang et al. (2020) presented dialogpt, a tunable large-scale conversational response generation model based on gpt-2 (radford et al., 2019). for tod system, kale and rastogi (2020) and du et al. (2020) proposed schema and template guided generation respectively which use slot-value tagged knowledge representations as input. lately, several works (we @xcite @xcite have put forward a retrieve and refine approach to combine plain-text knowledge in conversational response generation. these works have inspired us to adopt similar generation ideas for our task. ## conclusion the study proposes a neural response generation framework to reduce human labor in a real-world customer care setting, where a structured knowledge base is scarce. our framework extracts knowledge from historical records of conversations to generate an informative response. our evaluation shows the efficacy of the ranking system and provides evidence for the operational applicability of the framework. we plan to extend the framework with a response validation module for further improvement.