PENTATRON : PE rsonalized co NT ext-Aware Transformer for Retrieval-based c O nversational u N derstanding (2022)

Abstract: Conversational understanding is an integral part of modern intelligent devices. In a large fraction of the global traffic from customers using smart digital assistants, frictions in dialogues may be attributed to incorrect understanding of the entities in a customer’s query due to factors including ambiguous mentions, mispronunciation, background noise and faulty on-device signal processing. Such errors are compounded by two common deficiencies from intelligent devices namely, (1) the device not being tailored to individual customers, and (2) the device responses being unaware of the context in the conversation session. Viewing this problem via the lens of retrieval-based search engines, we build and evaluate a scalable entity correction system, PENTATRON. The system leverages a parametric transformer-based language model to learn patterns from in-session customer-device interactions coupled with a non-parametric personalized entity index to compute the correct query, which aids downstream components in reasoning about the best response. In addition to establishing baselines and demonstrating the value of personalized and context-aware systems, we use multitasking to learn the domain of the correct entity. We also investigate the utility of language model prompts. Through extensive experiments, we show a significant upward movement of the key metric (Exact Match) by up to 500.97% (relative to the baseline).

## introduction intelligent devices are ubiquitous in the modern computing. the scientific modules that drive these devices involve conversational understanding, ambient computing, natural language reasoning and self-learning @xcite @xcite . a user's interaction with a device, however, is susceptible to errors arising from a myriad of sources including wrong pronunciation, inaccuracies in the subject mentions in a sentence, environmental noise, hardware and software error @xcite . correct interpretations of user queries, especially entities, is central to delivering the best user experience. two important factors that contribute strongly to high-precision entity recognition are (1) personalization, ie, learning users' unique patterns, and (2) contextualization, ie, deriving cues from the information in a user-device interaction session. in this paper, we design and evaluate an entity correction system, pentatron, with both personalization and contextualization baked into its architecture. figure 1 : (above) one multi-turn dialogue session with defective source query which contains one erroneous entity 'wallace' and its successful rephrase with correct entity 'wallows'. (below) concatenation of queries and responses using special tokens to form a single sequence as encoder input. ## problem setup and solution design the overall architecture of the pentatron system is described in figure 3 . figure 3 : for a given user, the input request string from the pentatron orchestrator is processed by a transformer model and also by a named entity recognition model, both trained on historical user requests, to encode the request and extract mentions, respectively. a semantic search is applied on the request embeddings and the precomputed entity embeddings of the user to find the best match following which, post-processing is applied to feed the result into downstream components. ## conclusions and future directions in this work, we build a system called penta-tron which significantly improves user experience in intelligent devices by operating on entities and reducing friction in multi-turn dialogues. there are several future directions we plan to work on, including operationalizing large-scale unbiased personalized and context-aware systems, and designing self-learning @xcite ) using techniques such as reinforcement learning. we also plan to investigate the utility of a multi-level index to improve entity coverage and mitigate the cold-start problem for new customers. dynamic index building and deployment in low-latency applications is an ongoing direction.