Personalized Dense Retrieval on Global Index for Voice-enabled Conversational Systems (2023)

Abstract: Voice-controlled AI dialogue systems are susceptible to noise from phonetic variations and failure to resolve ambiguous entities. Typically, personalized entity resolution (ER) and/or query rewrites (QR) are deployed to recover from these error modes. Previous work in this field achieves personalization by constraining retrieval search space to personalized indices built from user’s historical interactions with the device. While constrained retrieval achieves high precision, predictions are limited to entities in recent user history, which offers low coverage of future requests. Further, maintaining individual indices for millions of users is memory intensive and difficult to scale. In this work, we propose a personalized entity retrieval system that is robust to phonetic noise and ambiguity but is not limited to a personalized index. We achieve this by embedding user listening preferences into a contextual query embedding used in retrieval. We demonstrate our model’s ability to correct multiple error modes and show 91% improvement over baseline on the entity retrieval task. Finally, we optimize the end-to-end approach to fit within online latency constraints while maintaining gains in performance.

## introduction as conversational ai agents assert a ubiquitous presence in millions of households, the expectation for a seamless user experience grows. users expect the ai agent to understand natural language queries and diverse accents, remember individual preferences, and function well in noisy environments. however, some interactions lead to user friction where a user does not get what they request. friction primarily arises from (1) system errors and (2) ambiguity. system errors accumulate across various stages of the spoken dialog system pipeline, such as automatic speech recognition (asr), natural language understanding (nlu), and more. for example, the asr model may confuse the user request "play low" with a phonetically similar request "play love" (figure 1a ). ambiguity arises when a user's input is unclear due to abbreviations, or lack of context. a common scenario in the music domain is when a user requests a song without specifying the artist. as illustrated in figure 1b , without context into a user's listening preferences, the system struggles to deliver the user's preferred track due to numerous matching entities in the catalog. these scenarios result in prolonged interactions where a user reformulates the query or abandons the request all together. in practice, these challenges are typically addressed through query rewriting (qr) @xcite and building robust entity resolution @xcite components. in search-based approaches, queries and candidate target entities are embedded in latent space where vector search is performed to retrieve the most relevant target per query. in qr, the output space is formed by historical user requests; in er it comprises catalog entities. personalization and contextualization are key for high-precision retrieval in entity-centric domains @xcite . current approaches rely on user-specific indices to constrain the output search space to requests/entities a user has associated with in the past to achieve personalization. however, personalized indices offer low coverage of future queries @xcite , since users regularly explore new content through novel queries. in this work we present an efficient personalized retrieval system that extends beyond the personalized index. rather than using historical interactions to constrain the output @xcite , we embed them into continuous representations to form user embeddings that become inputs into the retrieval model. the user embeddings are joined with semantic query representations to form a contextual embedding that is subsequently used in retrieval. figure 2 visually differentiates our work from existing global and personalized search-based retrieval approaches. to form the user embeddings, we propose en-tity2vec; a domain-aware continuous entity representation learning method that captures item similarities beyond semantics and phonetics. this differentiates our approach from previous work in contextualization @xcite , where multi-turn dialogues are concatenated with the query as input into a semantic encoder. to illustrate, consider a sequence of queries in a user session: "play dancing queen by abba" "play i will survive by gloria gaynor" "play bad girls" previous approach cannot leverage semantic signal in the sequence to disambiguate the final request for "bad girls", whereas our domain-aware embeddings would derive that user likes 70's disco music and resolve it to bad girls by donna summers as opposed to the more recent and popular song bad girls by mia. in section 6, we demonstrate that our approach improves by 91% over an index-based personalized baseline. further, we explain how we optimize the end-to-end system for runtime deployment. ## related work query rewriting (qr) in voice-enabled conversational ai systems is a popular way to refine asr output into forms that can be accurately handled by downstream systems @xcite @xcite . @xcite propose rewrites based on a markov chain model trained on historical user reformulation patterns. @xcite re-frame the problem as neural retrieval where queries and rewrite candidates are jointly encoded in vector space, followed by nearest-neighbor search on the query. the embedding-based search enables generalization to previously unseen queries. @xcite and @xcite improve precision by explicitly modeling diverse user preferences through personalized indices. however, the index is constrained to historical interactions which results in low recall ceiling when users request for new entities @xcite . collaborative filtering to diversify the index is suggested in @xcite , but index size is still limited by memory constraints. our approach overcomes this limitation by expanding search to the full catalog, while maintaining high precision and personalization power. our model architecture is inspired by @xcite and @xcite , who fuse embeddings from multiple sources in encoder-based qr and entity2vec ). on the query side (left) we embed the query text and up to 50 recent entities that user interacted with. on the entity side (right), we embed the entity tokens and fetch the entity2vec embedding. component embeddings are combined in the merger layer to form the final representation of query and entity. model weights are optimized to maximize cosine similarity between groundtruth (query,entity) pairs. and er systems. we take a similar approach to merge user and query embeddings in our encoder, which we describe in section 3.2. ## model we frame the problem as entity retrieval. following a dual-encoder framework for dense retrieval @xcite , we learn vector representations of queries and entities in a joint space. the encoder is optimized with a contrastive learning objective @xcite , ensuring that queries and relevant entities are embedded closely in the latent vector space. at inference, we leverage faiss @xcite to perform nearest neighbor search on a pre-computed global entity index to retrieve the closest candidates for an input query. ## data we build a dataset of voice search queries from user requests in a production system. for this work we target a subset of user utterances requesting music playback (e.g., "play flowers by miley cyrus"). expansion to other domains is in scope for future work. all user data are de-identified. ## results & discussion we observe that fine-tuning sbert is crucial to adapt the semantic encoder to our entity-centric domain (global, fine-tuned vs global, sbert in in the query. for example, the query "play phone booth" moved closer to song payphone and further away from phone play post-fine-tuning on our task. in ## conclusion in this work we present a novel approach for personalized search-based entity retrieval on noisy queries in voice-operated ai dialogue systems. we achieve personalization by encoding historical user preferences in a contextual query vector representation, followed by vector search on a global entity catalog. we empirically confirm that our approach significantly improves on existing baselines that rely on fixed-size historical indices to guide model output to personalized predictions. future work includes incorporating more contextual features in the user embeddings. for example, user preferences may vary based on time of day, day of week, and seasonality trends. similarly including external knowledge such as lyrics, release dates, etc., in the entity embeddings will be explored.