Optimizing NLU Reranking Using Entity Resolution Signals in Multi-domain Dialog Systems (2021)

Abstract: In dialog systems, the Natural Language Understanding (NLU) component typically makes the interpretation decision (including domain, intent and slots) for an utterance before the mentioned entities are resolved. This may result in intent classification and slot tagging errors. In this work, we propose to leverage Entity Resolution (ER) features in NLU reranking and introduce a novel loss term based on ER signals to better learn model weights in the reranking framework. In addition, for a multi-domain dialog scenario, we propose a score distribution matching method to ensure scores generated by the NLU reranking models for different domains are properly calibrated. In offline experiments, we demonstrate our proposed approach significantly outperforms the baseline model on both single-domain and cross-domain evaluations.

## introduction in spoken dialog systems, natural language understanding (nlu) typically includes domain classification (dc), intent classification (ic), and named entity recognition (ner) models. after ner extracts entity mentions, an entity resolution (er) component is used to resolve the ambiguous entities. for example, nlu interprets an utterance to alexa (or siri) "play hello by adele" as in the 'music' domain, 'play music' intent, and labels "hello" as a song name, "adele" as an artist name. er queries are then formulated based on such a hypothesis to retrieve entities in music catalogs. often times nlu can generate a list of hypotheses for dc, ic, and ner, and then a reranking model uses various confidence scores to rerank these candidates @xcite . since er is performed after nlu models, the current nlu interpretation of the utterance is limited to the raw text rather than its underlying entities. even in nlu reranking @xcite , only ## related work early reranking approaches in nlu systems use a single reranker for all the domains. robichaud et al. @xcite proposed a system for multi-domain hypothesis ranking (hr) that uses lambdamart algorithm @xcite to train a ranking system. the features in the ranking system include confidence scores for intents and slots, relevant database hits and contextual features that embed relationship to previous utterances. the authors showed improved accuracy in top domains using both non-contextual and contextual features. crook et al. adapted a similar reranking scheme for multi-language hypothesis ranking @xcite . the set of features in the reranker include binary presence variables, for example presence of an intent, coverage of tagged entities and contextual features. they adapted the lambdamart algorithm to train a gradient boosted decision trees model @xcite for cross language hypothesis ranking, and demonstrated comparable performance of the cross language reranker to the language-specific reranker. these models did not explicitly use er signals for reranking. in addition, reranking is done across domains. such single reranker approach is not practical in nlu systems with a large set of independent domains. in contrast, our approach emphasizes domain independence, allowing reranking to be performed for each domain independently. furthermore, we rely on er signal as a means to improve reranking. to the best of our knowledge, the most related work to ours is su et al. @xcite , which proposed a re-ranking scheme to maximize the accuracy of the top hypothesis while maintaining the independence of different domains through implicit calibration. each domain has its nlu reranker, and the scores for the hypotheses from reranking are compared across all the domains to pick the best hypothesis. the feature vector for each reranker is composed of intent, domain and slot tagging scores from the corresponding domain. additionally, a cross entropy loss term is used to ensure calibra-tion across domains. in a series of experiments, they demonstrated improvement of semantic understanding. our work is an extension of that work as we utilize er signals, in addition to the dc, ic, and ner scores, and introduce a new loss term to improve the reranking accuracy. to resolve the score non-comparable problem in a multi-domain system, traditional calibration methods utilize platt scaling or isotonic regression to calibrate the prediction distribution into a uniform distribution @xcite @xcite @xcite . however, this does not work in our scenario since the data in different domains are imbalanced, which causes domains with big traffic to have lower confidence scores. instead of using probability calibration methods, we propose a solution based on power transformation to match the prediction score distribution back to the original score distribution, thus making the scores comparable even after er information is added to nlu reranking. ## reranking model the baseline nlu reranking model is implemented as a linear function that predicts the ranking score from dc, ic, and ner confidence scores. we augment its feature vector using er signals and introduce a novel loss term that penalizes the hypotheses that do not have a matched entity in the catalog. similar to @xcite , we tested using a neural network model for reranking, but observed no improvements, therefore we focus on the linear model. ## score distribution matching before adding the er features, the reranking scores are calibrated through training on the same utterance data with similar models. however, adding the er features in nlu reranking for a single domain may lead to incomparable scores with other domains. using the loss function in eq (3), we have the following theorem: proof. for the expected semer loss s u , since it is the linear combination of semer i , the solution of the minimization problem will be: p 1 → 1, p 2 = p 3 = p 4 = p 5 → 0. this leads to:y 1 → ∞, y 2 = y 3 = y 4 = y 5 → -∞. then for the expected cross entropy loss c u , let x @xmath0, the minimization of c u becomes: the first part (i 1 ) is monotonically increasing, while the second part (i 2 ) is monotonically decreasing when x @xmath1. this also leads to: y 1 → ∞, y 2 = y 3 = y 4 = y 5 → -∞. thus, solving the minimization problem minl u is equivalent to solving the linear system: when @xmath2, where f + is the feature matrix for the positive labels, f -is the feature vector for the negative labels, w is the weight vector we need to solve, and 1 + , 1 -are the unit vectors with the same dimension as the number of positive samples and negative samples respectively. we can rewrite eq ( 6) into: f @xmath3, and its solution will be the projection associated with the loss in eq (3) of y onto the solution space spanned by the column vectors of matrix f . now define this projection as p f ( y). for the feature matrix of the nlu model in eq ( 2) , we have f @xmath4, and for the feature matrix of nlu-er model in eq ( 1) we have f @xmath5, er s 1 , er s 2 , . . . , er sq , 1 def ault ]. since f n is the submatrix of f er , we have spanf n ⊂ spanf er , thus: in theorem 4.1, we show that the candidate hypothesis from a more complicated model will be more likely to have a higher score than the domains using the original reranker model. thus the domains using the nlu-er reranker are no longer comparable to the domains using the original model. we observed this scenario in our experiments empirically. when we only experiment with music domain, it will generate higher confidence scores and have more false positives. to solve this problem, since we would like the confidence scores for each domain to have stabilized variance and minimized skewness, we propose to use power transformation, which is able to map data from any distribution to an approximately standard gaussian distribution. in our case, the confidence scores from eq (1) might be zero or negative, thus we consider the yeo-johnson transformation with λ = 0 and λ = 2: we have the inverse function: ## conclusion in this work, we proposed a framework to incorporate er information in nlu reranking. we developed a new feature vector for the domain reranker by utilizing entity resolution features such as hits or no hits. to provide the er features to the nlu reranker, we proposed an offline solution that distills the er signals into a gazetteer. we also introduced a novel loss term based on er signals to discourage the domain reranker from promoting hypotheses with er no match and showed that it leads to better model performance. finally, since domain rerankers predict the ranking scores independently, we introduced a score matching method to transform the nlu-er model score distribution to make the final scores comparable across domains. our experiments demonstrated that the music domain reranker performance is significantly improved when er information is incorporated in the feature vector. also with score calibration, we achieve moderate gain for the cross-domain scenario.