Noise Robust Named Entity Understanding for Voice Assistants (2021)

Abstract: Named Entity Recognition (NER) and Entity Linking (EL) play an essential role in voice assistant interaction, but are challenging due to the special difficulties associated with spoken user queries. In this paper, we propose a novel architecture that jointly solves the NER and EL tasks by combining them in a joint reranking module. We show that our proposed framework improves NER accuracy by up to 3.13% and EL accuracy by up to 3.6% in F1 score. The features used also lead to better accuracies in other natural language understanding tasks, such as domain classification and semantic parsing.

## introduction understanding named entities correctly when interacting with virtual assistants (e.g. "call jon", "play adele hello", "score for warrior kings game") is crucial for a satisfying user experience. however, ner and el methods that work well on written text often perform poorly in such applications: utterances are relatively short (with just 5 tokens, on average), so there is not much context to help disambiguate; speech recognizers make errors ("play bohemian raspberry" for "play bohemian rhapsody"); users also make mistakes ("cristiano nando" for "cristiano ronaldo"); non-canonical forms of names are frequent ("shaq" for "shaquille o'neal"); and users often mention new entities unknown to the system. in order to address these issues we propose a novel named entity understanding (neu) system that combines and optimizes ner and el for noisy spoken natural language utterances. we pass multiple ner hypotheses to el for reranking, enabling ner to benefit from el by including information from the knowledge base (kb). we also design a retrieval engine tuned for spoken utterances for retrieving candidates from the kb. the retrieval engine, along with other techniques devised to address fuzzy entity mentions, lets the el model be more robust to partial mentions, variation in named entities, use of aliases, as well as human and speech transcription errors. finally, we demonstrate that our framework can also empower other natural language understanding tasks, such as domain classification (a sentence classification task) and semantic parsing. ## related work there have been a few attempts to explore ner on the output of a speech pipeline @xcite @xcite . among these, our ner model is closest to @xcite and @xcite ; however, unlike the former, we use a richer set of features rather than phonemes as input, and unlike the latter, we are able to use a deep model because of the large volume of data available. el has been well explored in the context of clean @xcite @xcite and noisy text inputs @xcite @xcite , but as with ner, there have been only a few efforts to explore el in the context of transcribed speech @xcite , although crucially, both these works assume gold standard ner and focus purely on the el component. traditionally, a pipelined architecture of ner followed by el has been used to address the entity linking task @xcite @xcite . since these approaches rely only on the best ner hypothesis, errors from ner propagate to the el step. to alleviate this, joint models have been proposed @xcite proposed an ner+el model which re-ranks candidate mentions and entity links produced by their base model. our work differs in that we use a high precision ner system, while they use a large number of heuristically obtained noun phrase (np) chunks and word n-grams as input to the el stage. @xcite jointly train an ner and el system using a probabilistic graphical model. however, these systems are trained and tested on clean text and do not address the noise problems we are concerned with. ## architecture design for a given utterance, we first detect and label entities using the ner model and generate the top-l candidate hypotheses using beam search. the el model consists of two stages: (i) candidate retrieval and (ii) joint linking and re-ranking. in the retrieval stage, for each ner hypothesis, we construct a structured search query and retrieve the top-c candidates from the retrieval engine. in the ranking stage, we use a neural network to rank these candidate entity links within each ner hypothesis while simultaneously using rich signals (entity popularity, similarity between entity embeddings, the relation across multiple entities in one utterance, etc.) from these entity links as additional features to re-rank the ner hypotheses from the previous step, thus jointly addressing both the ner and el tasks. ## datasets and training methodology to create our datasets, we randomly sampled around 600k unique anonymous english transcripts (machine transcribed utterances), and annotated them with ner and el labels. utterances are subject to apple's baseline privacy practices with respect to siri requests, including that such requests are not associated with a user's apple id, email address, or other data apple may have from a user's use of other apple services, and have been filtered as described in section 7. we then split the annotated data into 80/10/10 for train, development and test sets. for both the ner and el tasks, we report our results on test sets sampled from the "music", "sports" and "movie & tv" domains. these are popular domains in the usage and have a high percentage of named entities: with an average of 0.6, 1.1 and 0.7 entities for each utterance in the 3 domains respectively. to evaluate model performance specifically on noisy user inputs, we select queries from the test sets that are marked as containing speech transcription or user errors by the annotators and report results on this "noisy" subset, which constitutes 13.5%, 12.7% data for movie&tv and music domain respectively when an entity exists. 3 to evaluate the relation feature, we also look at the "related" subset where a valid relation exists in the utterance. this subset consists 13.4% and 5.3% of data for the music and sports domain with at least one entity. 4 we first train the ner model described in section 3.1. next, for every example in our training dataset, we run inference on the trained ner model and generate the top-5 ner hypotheses using beam search. following this, we retrieve the top 25 candidates for each of these hypotheses using our search engine combined with the ground truth ner and el labels and fed to the el model for training. to measure the ner model performance, we use the standard ner f1 metric used for the conll-2003 shared task @xcite . to measure the quality of the top-5 ner hypotheses, we compute the oracle top-5 f1 score by comparing and choosing the best alternative hypothesis among the 5 and calculate its f1 score, for each test utterance. in this manner, we also know the upper bound that el can reach from reranking ner hypotheses. as described in section 3.2, the el model is optimized to perform two tasks simultaneously: entity linking and reranking of ner hypotheses. hence to evaluate the performance of the el model, we use two metrics: reranked ner-f1 score and the el-f1 score. the reranked ner f1 score is measured on the ner predictions according to the top el hypothesis, and is defined in the same way as the previous ner task. to evaluate entity linking quality, we adopt a strict f1 metric similar to the one used for ner. besides entity boundary and entity type, the resolved entity also needs to be correct for the entity prediction to be counted as a true positive. for ner model training, we use standard minibatch gradient descent using the adam optimizer with an initial learning rate of 0.001, a scheduled learning rate decay of 0.99, lstm with a hidden layer of size 350 and a batch size of 256. we apply a dropout of 0.5 to the embedding and bilstm layers, and include token level gazetteer features @xcite to boost performance in recognizing common entities. we linearly project these gazetteer features and concatenate the projection with the 200 dimensional word embeddings and 100 dimensional character embeddings which are then fed into the bilstm followed by the crf. for el, the character cnn model we use has two layers, each with 100 convolution kernels of size 3, 4, and 5. character embedding are 25 dimensional and trained end to end with the entity linking task. the mlp for embedding similarity takes the concatenation of two name embeddings, as well as their element-wise sum, difference, minimum, maximum, and multiplication. it has two hidden layers of size 1024 and 256, with output dimension 64. similarity features of mentions in the prediction are averaged, while the other features like ner confidence and entity popularity are concatenated to the representation. the final mlp for scoring has two hidden layers, with size 256 and 64. we train the model on 4 gpus with synchronous sgd, and for each gradient step we send a batch of 100 examples to each gpu. ## conclusion in this work, we have proposed a named entity understanding framework that jointly identifies and resolves entities present in an utterance when a user interacts with a voice assistant. our proposed architecture consists of two modules: ner and el, with the el serving the additional task of possibly correcting the recognized entities from ner by leveraging rich signals from entity links in the knowledge base while simultaneously linking these entities to the knowledge base. with several design strategies in our system targeted towards noisy natural language utterances, we have shown that our framework is robust to speech transcription and user errors that occur frequently in spoken dialog systems. we have also shown that featurizing the output of neu and feeding these features into other language understanding tasks substantially improves the accuracy of these models.