RSVP : Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training (2023)

Abstract: The dialogue systems in customer services have been developed with neural models to provide users with precise answers and round-the-clock support in task-oriented conversations by detecting customer intents based on their utterances. Existing intent detection approaches have highly relied on adaptively pre-training language models with large-scale datasets, yet the predominant cost of data collection may hinder their superiority. In addition, they neglect the information within the conversational responses of the agents, which have a lower collection cost, but are significant to customer intent as agents must tailor their replies based on the customers’ intent. In this paper, we propose RSVP, a self-supervised framework dedicated to task-oriented dialogues, which utilizes agent responses for pre-training in a two-stage manner. Specifically, we introduce two pre-training tasks to incorporate the relations of utterance-response pairs: 1) Response Retrieval by selecting a correct response from a batch of candidates, and 2) Response Generation by mimicking agents to generate the response to a given utterance. Our benchmark results for two real-world customer service datasets show that RSVP significantly outperforms the state-of-the-art baselines by 4.95% for accuracy, 3.4% for MRR@3, and 2.75% for MRR@5 on average. Extensive case studies are investigated to show the validity of incorporating agent responses into the pre-training stage.

## introduction task-oriented dialogue systems aim to assist users in finishing specific tasks, and have been deployed in a wide range of applications, e.g., tour guides @xcite , medical applications @xcite , and especially in the ecommerce domain such as booking flights and hotels @xcite . with the rapidly expanding market and customers' growing demands 1 code is available at https://github.com/tommytyc/rsvp . for high-quality shopping experiences, providing precise answers efficiently through customer services has become increasingly vital for companies. recently, companies, especially e-commerce companies, are developing their own ai models @xcite @xcite to automate repetitive tasks and reduce the burden on agents. the common approach is to frame this problem as an intent detection task @xcite , which involves classifying a customer's utterance into one of the pre-defined intents. previous methods tackling intent detection tasks with deep neural models @xcite are able to learn semantic representation from customer utterances. however, these methods rely on large-scale high-quality labeled data requiring expensive and time-consuming annotations @xcite . recent studies on intent detection primarily focus on pre-training language models with public dialogue datasets and fine-tuning them for downstream intent detection tasks @xcite @xcite . these methods have proven to be effective, but they heavily depend on the availability of additional public pre-training datasets, where large amounts of unlabeled data, especially when the language is not in english, can be hard to acquire and collect due to the high cost @xcite . furthermore, most of these studies only utilize customer utterances to detect intent, while neglecting the usefulness of metadata (e.g., agent responses) from in-house customer services systems, which can be efficiently collected with customer utterances and without the requirement of annotation cost. however, how to effectively utilize the agent responses to benefit intent detection is an unknown yet challenging problem, as the agent responses cannot be obtained before detecting the intent of customer requests during real-time deployment. in light of this, we aim to incorporate agent responses with customer utterances in the training set to produce fine-grained knowledge. we hypothesize that if the model is able to provide an accurate response for a given utterance, it must have inferred the hidden intent behind the utterance. as shown in figure 1 , a customer may raise a question about the manipulation of a product. the agent response shows the concrete solution, "press the button on the device", to the customer utterance due to the agent's understanding of the intent being the manipulation approach to the product. if the pre-trained model understands that the agent's response will include the manipulation approach to the product, the intent detection model will be able to classify the utterance into the intent: productmanipulation. in this work, we propose rsvp, a two-stage framework composed of a pre-training step and a fine-tuning step to utilize the agent responses to effectively learn the intent of customer utterances by introducing two pre-training tasks to learn a much more reliable conversational text encoder which has been further pre-trained to predict an accurate response. the first task, response retrieval, follows a dual encoder paradigm and models the customer utterance as a dense retrieval to a batch of agent responses. this reformulation enables the pre-trained language model to discriminate between the correct response and the others. the second task, response generation, constructs a conversational question-answering problem with the utterances and responses. by directly learning to answer the customer utterances, the pre-trained language model can improve the extracted semantic knowledge of the agent responses. finally, the pre-trained language model will be used to finetune the intent detection task. as the pre-training tasks have aligned the model better with the agent responses, we expect that the adapted model can infer the intent better for each incoming utterance. our contributions can be summarized as follows: ## related work existing intent detection approaches have shown promising results by leveraging the ability of transfer learning with two-stage pre-training and finetuning process. specifically, these works @xcite @xcite involve pre-training models in a self-supervised manner with mask language modeling (mlm) on large public dialogue datasets, and fine-tuning them on the target intent datasets. @xcite presented a continued pre-training framework, where they pre-trained the model with mlm and cross-entropy losses from a few publicly available intent datasets to improve the few-shot performance on cross-domain intent detection tasks. additionally, a line of studies has introduced pre-training methods incorporating a variety of relevant nlp tasks, such as natural language inference (nli), to leverage knowledge from other public datasets. @xcite recast intent detection as a sentence similarity task and pre-trained the language model on nli datasets with bert @xcite pairwise encoding, and they came up with a nearest-neighbor framework to take advantage of the transfer learning. however, their system is computationally expensive due to the full utilization of training data in both the training and inference stages. @xcite pre-trained a language model on intent detection datasets with self-supervised learning and fine-tuned it with supervised-contrastive learning @xcite . @xcite designed an additional regularization loss term to tackle the anisotropy problem, as they found that supervised pre-training may yield a feature space where semantic vectors fall into a narrow cone. @xcite reformulated intent detection as a questionanswering task and used intent label names as the answer to the customer utterances. our method shares some similarities with @xcite , who proposed pre-training the language model on a public reddit dataset @xcite with a response ranking task, and then fine-tuned it on an intent detection task by contrastive learning. in contrast, in order to take the customer service agent's response into consideration and relieve the large-scale data labeling effort, we pre-train directly on the customer service dialogue with our two pre-training tasks: response retrieval, which introduces a response selection task by retrievalbased batch contrastive loss, and response generation, which formulates mimicking agent responses by a text generation task. ## conclusion this paper presents rsvp, a novel two-stage framework to address the challenging intent detection problem in real-world task-oriented dialogue systems. rsvp leverages customer service agents' responses to pre-train with customer utterances, offering a refreshing view to utilize the metadata in the dialogue systems. it allows us to explore the implicit intent lying in both utterances and responses, and takes advantage of contrastive learning to improve the model's robustness. rsvp consistently outperforms other strong baseline models on two real-world customer service benchmarks, showcasing the effectiveness of this framework. we believe rsvp can serve flexibly for task-oriented dialogue intent detection problems, and several interesting research directions could be further investigated within this framework, such as more metadata related to the conversation, pre-training tasks inte-grating the label names with responses, etc.