Collaborative Data Relabeling for Robust and Diverse Voice Apps Recommendation in Intelligent Personal Assistants (2021)

Abstract: Intelligent personal assistants (IPAs) such as Amazon Alexa, Google Assistant and Apple Siri extend their built-in capabilities by supporting voice apps developed by third-party developers. Sometimes the smart assistant is not able to successfully respond to user voice commands (aka utterances). There are many reasons including automatic speech recognition (ASR) error, natural language understanding (NLU) error, routing utterances to an irrelevant voice app or simply that the user is asking for a capability that is not supported yet. The failure to handle a voice command leads to customer frustration. In this paper, we introduce a fallback skill recommendation system to suggest a voice app to a customer for an unhandled voice command. One of the prominent challenges of developing a skill recommender system for IPAs is partial observation. To solve the partial observation problem, we propose collaborative data relabeling (CDR) method. In addition, CDR also improves the diversity of the recommended skills. We evaluate the proposed method both offline and online. The offline evaluation results show that the proposed system outperforms the baselines. The online A/B testing results show significant gain of customer experience metrics.

## introduction intelligent personal assistants such as alexa, siri, and google assistant have been becoming more and more popular and making people's daily lives convenient. ipas can fulfill users' requests by answering questions ranging from weather to stock price. to enrich user experience, a large amount of third-party (3p) voice apps (aka skills) have been developed. these voice apps extend ipas built-in capabilities to better serve customers. they can perform operations like ordering food, playing a game, or helping a user sleep by playing soothing sounds. the supported 3p skills can number up to hundreds of thousands. ipas understand user's request using spoken language understanding (slu) system. the request goes through a series of components to get response, as illustrated in figure 1 . the first component is automatic speech recognition (asr), which converts speech to its transcription also called utterance. at the second stage, the utterance is interpreted by nlu system. nlu as the critical component of slu interprets the meaning of an utterance by using several nlp technologies including domain classifier (dc), intent classifier (ic), and named entity recognition (ner). after nlu, the arbiter is responsible to select the most relevant voice app (skill) for a given nlu interpretation. sometimes the arbiter may fail to find a relevant skill that can handle the user request. it could be a system error such as automatic speech recognition (asr) error, natural language understanding (nlu) error. another reason could be that the feature requested by the user is not supported yet by the dialog system or the requested content is not found such as music, video, book, recipe, etc. to reduce customer friction and recover the conversation, we propose a skill recommender system which proactively suggests 3p skills to users for unhandled requests, even if the users are not aware of the skills. the proposed skill recommender system is composed of two components: a shortlister, and a reranker. figure 2 shows the system architecture. given an utterance, the shortlister, also known as the candidate generator retrieves k most relevant skills out of the skill catalog. the retrieved skills are passed to the reranker that ranks the skill candidates by using skill specific information and the utterance. finally, the top-1 skill is presented to the user. this system is not meant to replace the original nlu or arbiter components. it is specifically designed to serve as a fallback for utterances that are not handled by the existing system (i.e., unclaimed utterances) using the increasing catalog of 3p skills. traditional recommender systems such as video recommendation recommend a ranked list of items to a user. the user scans the list and select the one he/she likes the most @xcite . the feedback from the user is treated as label (accept/reject) for learning a model. however, due to the limitation of voice user interface (vui), we can only present the top-1 skill to users, as listening to the playback of a long list is tedious and can significantly degrade user experience @xcite . this limitation results in partial observation problem. namely, users cannot observe the full recommendation list and make a decision, which imposes difficulties in learning a ranking model. to solve partial observation problem, we propose a novel method called collaborative data relabeling (cdr). cdr mitigates the partial observation problem by trying to answer a counterfactual question, "what if we present another skill to the user?". cdr answers this question by matching a similar request and using feedback from that request to relabel the original ranked list. recommender systems usually focus on optimizing the accuracy of predictions while ignoring the diversity of the recommended items, which can degrade user experience if similar items get recommended over and over again @xcite @xcite @xcite . cdr improves the diversity of recommended skills by relabeling different skill candidates that serve the same intent. the relabeled skills force the model to learn to diversify their prediction distribution among multiple skill candidates. at the beginning, we do not have data for training the model. to collect training data, we build a rule-based system. similar to the proposed system, the rule-based system also has a two-stage architecture. we use the data collected from this system to train and evaluate our proposed model offline. the proposed model is put into production for online a/b testing after it has achieved satisfying offline results. online experimental results show significant gains of user experience metrics such as higher volume of acceptances, lower friction rates, etc. overall, the contributions of this work are summarized as following: online experimental results show significant gains of user experience metrics. ## skill recommender system our skill recommender system consists of two components, shortlister and reranker, as shown in figure 2. given the input utterance text, shortlister selects top-k relevant skills from the skill catalog. we implement shortlister as a keyword-based search engine. to build the skill search engine, we index skill metadata including skill name, skill descriptions, example phrases, and invocation phrases. at retrieval time, the relevancy score between an utterance and a skill is computed as the sum of tf-idf score @xcite of every word in the utterance. the skills with top k relevancy scores are returned. the reranker model takes in the skill candidates generated by shortlister and returns a ranked list of skills based on the utterance and skill specific information. the reranker is a deep learning model with a listwise ranking loss function. figure 3 shows the reranker model architecture. the utterance is encoded by a bert encoder @xcite . the features of skills include skill id, skill name, and skill score returned by shortlister. skill id is represented using an embedding vector; skill name is encoded into an embedding vector by bert. the skill score feature is converted into a bin and encoded as an embedding vector. the skill feature embedding vectors are concatenated to form a single embedding vector. the utterance embedding vector is concatenated with every skill embedding vector to form a sequence of utterance-skill embedding vectors. as the skill candidates returned by shortlister is ordered by relevance score, to capture such sequential information, these sequence of embedding vectors are put into a bi-lstm layer @xcite . the outputs from the bi-lstm layer is converted to probability scores by using softmax function. each skill has a corre-sponding probability score. the skills are reranked according to the predicted probability scores. for a list of skill candidates, the user feedback for the skill candidates is @xmath0, ..., y k }, y @xmath1, 1} indicating whether the user accepts or rejects the skills and the predicted probabilities by the reranker model is @xmath2, ..., s k }. we use listwise ranking objective function @xcite . the objective function is formulated as ## collaborative data relabeling compared to traditional recommender system such as video recommendation where users view the full recommended list and select the best one they like, the skill recommender system has its unique challenge. limited by vui, we can only present the top-1 ranked skill to user, which results in a partial observation problem. with partial observation, users have no chance to view and compare other skills in the list. we do not know if the user would like the other skills more than the top-1. without comparing the top-1 skill with the other skills, it is hard to learn a ranking model, as ranking in essence is about comparing. to solve partial observation problem, we propose collaborative data relabeling (cdr) approach. in skill recommendation, only the top-1 skill is presented to users. given a user x who invoked voice assistant with some utterance, to know her responses to skills s y and s z that were not presented to her, we found two users y and z who spoke similar utterances and were suggested with skills s y and s z , respectively and use their responses to relabel user x's feedback to skills s y and s z . the intuition of cdr is to answer a counterfactual question, namely, "what if we had presented another skill to user?". to answer this question, we find k nearest neighbors of a user request (utterance) and use their feedback to relabel the original ranked list of skill candidates, which is inspired by matching method @xcite in causal inference. in causal inference, matching is an approach to estimate the treatment effect by comparing the treated units to non-treated units with similar characteristics. cdr has a similar working mechanism. given a user utterance, to know the user's response to an unpresented skill, we find a similar utterance whose invoker has interacted with that skill and use his/her response to relabel it as either positive or negative, as illustrated in figure 4 . usually, there are more than one neighbors that have interacted with the skill, in which case we use majority vote to decide the final label. as the final labels are decided by multiple neighbors, they are more reliable than those of just one user, which results in learning a more robust model. recommender systems are confronted with an over-fitting problem that only a small portion of items are recommended to users @xcite , which can hurt user satisfaction as they can quickly get bored by always being suggested with similar types of items. this problem is especially relevant for skills that serve the same intent with different content. for example, when users ask to play a soothing sound to help them sleep, always suggesting the same sleep sound can get users bored, while there exist many types of sleep sounds in the skill store such as frog, ocean, rain, waterfall sleeping sound, etc. suggesting diverse skills can improve user satisfaction @xcite . the proposed cdr method improves diversity by relabeling different skill candidates as positive, which forces the model's predictive distribution to be dispersed among more skills. diversified suggestions can lead to drop in accuracy @xcite , which imposes difficulties in faithfully evaluating the real user satisfaction metrics. to evaluate how diversity can influence user satisfaction, we use manual annotation. the detail of manual annotation schema will be explained in section 5.1.2. to capture semantic meaning of utterances for similarity measurements, we use fine-tuned bert encoder to encode utterances into embedding vectors. the bert model is fine-tuned using data with a multi-task objective function, specifically, intent classification and named entity recognition. we also experiment with pre-trained bert encoder and find that it does not work well for capturing semantics of an utterance, which has also been discovered by several works such as @xcite and @xcite . we use the average pooling of the contextual embedding vectors in the last layer as the utterance embedding vector. we measure the similarity score between two utterances using cosine similarity between their embedding vectors. ## conclusions and future work in this work, we proposed a skill recommender system to suggest skills for unhandled voice commands in intelligent personal assistants that aims to reduce the user friction and recover the conversation. compared to traditional recommender systems, the skill recommender system faces the challenge of partial observation. to resolve this challenge, we proposed collaborative data relabeling. in addition, it improves the diversity of recommended skills. collaborative data relabeling as a simple and effective approach is especially useful for industrial deployment. we evaluated the proposed system offline before put it online for a/b testing. the online experimental results showed significant gains of user experience metrics. in the future, we will try contextual bandits and let the model learn to explore the under-exploited skills.