Semantic Diversity for Natural Language Understanding Evaluation in Dialog Systems (2020)

Abstract: The quality of Natural Language Understanding (NLU) models is typically evaluated using aggregated metrics on a large number of utterances. In a dialog system, though, the manual analysis of failures on specific utterances is a time-consuming and yet critical endeavor to guarantee a high-quality customer experience. A crucial question for this analysis is how to create a test set of utterances that covers a diversity of possible customer requests. In this paper, we introduce the task of generating a test set with high semantic diversity for NLU evaluation in dialog systems and we describe an approach to address it. The approach starts by extracting high-traffic utterance patterns. Then, for each pattern, it achieves high diversity selecting utterances from different regions of the utterance embedding space. We compare three selection strategies based on clustering of utterances in the embedding space, on solving the maximum distance optimization problem and on simple heuristics such as random uniform sampling and popularity. The evaluation shows that the highest semantic and lexicon diversity is obtained by a greedy maximum sum of distance solver in a comparable runtime with the clustering and the heuristics approaches.

## background in the past years, voice-first dialog systems have become ubiquitous in the market, with an ever increasing number of features, languages and customer requests. a crucial component of these systems is the natural language understanding (nlu) model. the nlu model maps customer requests onto specific actions that the device has to perform. in practice, this means classifying an utterance into a domain, intent and slots @xcite . for instance, given the customer's utterance "play madonna", an nlu model returns: (music, playmusicintent, play artistname) where music is the domain, playmusicintent is the intent and the slot is artistname. when a new algorithm for nlu is proposed in a research environment, the evaluation is typically performed by aggregating metrics such as slot error rate (ser) @xcite and semantic error rate (semer) @xcite on a large test set of utterances. however, in a production environment, aggregated metrics alone are not sufficient, as they may hide failures on specific business critical utterances. thus, whenever a change is introduced into an nlu model, failures need to be manually reviewed to determine whether they represent an issue for the customers. the manual review of failures is a crucial, and yet very time-consuming operation. hence, the question: how to create a test set that makes the analysis more efficient including a diversity of patterns, utterances and possibile failure causes? the problem of maximizing semantic diversity in text is common in tasks such as text summarization @xcite ), text generation @xcite , keyphrase extraction @xcite , machine translation @xcite , data augmentation in dialog systems @xcite . however, to the best our knowledge, semantic diversity has never been used to create test sets for the evaluation of natural language understanding models in dialog systems. in this paper, we introduce an approach to automate the creation of test sets with high semantic diversity for the evaluation of the nlu model in a dialog system. the approach works as follows. first, we filter the dataset extracting a set of high-traffic pattern. then, for each pattern, we map utterances into an embedding space to represent the semantics of the different slot values. finally, we create test sets comparing three selection algorithms based on partitioning the space in groups and selecting representatives or on directly solving a maximum sum of distance optimization problem to achieve high diversity. ## approach the approach can be divided in three major steps: pattern extraction, encoding and selection (fig. 1 ).