Automating Template Creation for Ranking-Based Dialogue Models (2020)

Abstract: Dialogue response generation models that use template ranking rather than direct sequence generation allow model developers to limit generated responses to pre-approved messages. However, manually creating templates is time-consuming and requires domain expertise. To alleviate this problem, we explore automating the process of creating dialogue templates by using unsupervised methods to cluster historical utterances and selecting representative utterances from each cluster. Specifically, we propose an end-to-end model called Deep Sentence Encoder Clustering (DSEC) that uses an auto-encoder structure to jointly learn the utterance representation and construct template clusters. We compare this method to a random baseline that randomly assigns templates to clusters as well as a strong baseline that performs the sentence encoding and the utterance clustering sequentially. To evaluate the performance of the proposed method, we perform an automatic evaluation with two annotated customer service datasets to assess clustering effectiveness, and a human-in-the-loop experiment using a live customer service application to measure the acceptance rate of the generated templates. DSEC performs best in the automatic evaluation, beats both the sequential and random baselines on most metrics in the human-in-the-loop experiment, and shows promising results when compared to gold/manually created templates.

## introduction dialogue response generation has been an active area of research in recent years. response generation can be used in human-to-bot conversational systems @xcite or to generate quick replies in human-to-human conversational systems @xcite . response generation approaches fall under two broad categories: (1) direct sequence generation using an encoder-decoder architecture @xcite or (2) response ranking, in which the model developer specifies a predefined template pool and an encoder model is used to score pairs of conversation history and candidate template response @xcite @xcite . using template ranking rather than direct sequence generation allows model developers to limit generated responses to pre-approved messages, preventing the model from producing impolite or ungrammatical responses. in addition, sequence generation models have a tendency to favor safe, generic responses @xcite @xcite , and template ranking models can be used to ensure that the system generates information-rich responses that drive the conversation towards an end goal. however, manually creating templates is time-consuming and requires domain expertise. for certain use cases such as customer service, templates need to be continually updated to reflect policy changes, further adding to this cost. in addition, manually created templates may differ subtly from actual agent utterances in model training data and thus may not be selected by the ranking model. in this paper, we explore automating the creation of a template pool for a customer service chat application through clustering historical agent utterances and choosing representative utterances from each cluster. to the best of our knowledge, research on automatic template creation using utterance clustering has been limited. the structure of this paper is as follows. in section 2, we describe the data and text preprocessing methods we used to extract template candidates from historical chat transcripts. in section 3, we describe our proposed approach for template generation: an end-to-end approach that uses an auto-encoder structure to jointly learn the utterance representation and construct template clusters. in addition, we describe a strong baseline that we propose for comparison: a sequential approach in which we first learn the utterance representation and then construct template clusters. in section 4, we describe the automatic and human-in-the-loop evaluations that we conducted and our findings, and in section 5 we draw conclusions and propose future research directions. ## data we select template responses from a dataset of agent utterances extracted from historical chat transcripts. to construct this dataset, we collect anonymized transcripts of conversations between customers and customer service agents (csas) in two domains: (1) cancel membership (cm), and (2) tracking shows delivered but order not received (dnr). in the anonymized transcripts, all unique customer identifiers (uci) are replaced with a special token: "generic slot". we further extract all agent utterances foot_0 in these transcripts and exclude those occurring only once in the data. the intuition behind this is that if an utterance only occurred once, it is not likely to be useful as a template. we end up with approximately 550k agent utterances in each domain. the dnr domain contains longer utterances than the cm domain (an average of 12 words per sentence vs. 11 for cm) and a larger vocabulary size (22.9k for dnr vs. 19.2k for cm). ## approach we cluster agent utterances using a novel end-toend approach, deep sentence encoder clustering (dsec), in which the utterance representation and the clustering model are jointly learned. we compare this against two baselines: (1) a weak baseline in which templates are sampled randomly from the dataset, and (2) a sequential baseline in which the utterance representation and the clustering model are learned sequentially. for the baseline system, we use dense features to represent each utterance and explore the use of different embedding types-glove @xcite , elmo @xcite , and bert @xcite as well as the effect of using in-domain data on the performance of the system. for both dsec and the sequential baseline, after the clusters have been obtained, we create the template pool by selecting the highest-confidence utterance in each cluster. the confidence is either the probability that the utterance falls in the cluster (for dsec), or the distance between the utterance and its cluster centroid (for the sequential baseline). domain cluster description utterances cm informing the customer of confirmation e-mail your refund of generic slot will be credited to your original payment method within 7 to 10 business days. cm confirming refund request i see that you have used the membership benefits, and because of that i can offer generic slot refund. sounds good? cm greeting good afternoon. cm asking for confirmation can you please confirm the last four digits and the expiration date of the payment method that has been charged? dnr confirming refund options would you like the refund to be back on your gift or credit card? dnr apology i do apologize for the inconvenience if it was tagged as delivered but nowhere to be found. dnr confirming order status it seems that the package was already lost and mismarked as delivered. ## experiments & results we evaluate clustering performance using both automatic and human-in-the-loop evaluations. for all experiments, we fix the cluster number at 50 for all models to ensure that the template pool has good coverage of common situations. ## conclusion & future work we present dsec, an end-to-end sentence encoding and clustering approach that can help automate template creation for template-based conversational models. the purpose is to avoid the human effort required to manually create a template pool when training a response generation model for a conversational system. we evaluate the proposed approach on two customer service datasets and find that it outperforms both a strong sequential baseline and a random baseline in most cases. in addition,