Graph Meets LLM : A Novel Approach to Collaborative Filtering for Robust Conversational Understanding (2023)

Abstract: A Personalized Query Rewriting system strives to minimize defective queries to ensure robust conversational functionality by considering individual user behavior and preferences. It’s designed as a search-based system, maintaining a user index of past successful interactions with the conversational AI. However, this method faces challenges with unseen interactions, which refers to novel user interactions not covered by the user’s historical index. This paper introduces our Collaborative Query Rewriting approach, which utilizes underlying topological information to assist in rewriting defective queries arising from unseen user interactions. This approach begins by constructing a “User Feedback Interaction Graph” (FIG) using historical user-entity interactions. Subsequently, we traverse through the graph edges to establish an enhanced user index, referred to as the “collaborative user index”. This paper then further explores the use of Large Language Models (LLMs) in conjunction with graph traversal, leading to a significant increase in index coverage for unseen interactions. The effectiveness of our proposed approach has been proven through experiments on a large-scale real-world dataset and online A/B experiments.

## introduction defective queries frequently occur during user interactions with conversational ai systems such as alexa, siri or google assistant. these are induced by user ambiguities or mistakes, along with errors in automatic speech recognition (asr) or natural language understanding (nlu). defective queries impact the robustness of the conversational ai system, as they hinder users from receiving the intended results and often require further clarification. query rewriting (qr) is a subsystem within the conversational ai that plays a crucial role in reducing defective queries. by automatically refining or correcting these defective queries, qr enhances the overall robustness of the ai system and significantly improves the user experience. personalized query rewriting (personalized qr) takes into account individual preferences or unique error patterns identified from a user's historical interactions with the conversational ai. it plays a crucial role in addressing a wide range of user-specific defects, particularly in the torso and tail distribution. for instance, when a user presents a defective query like "play abcdefg", a non-personalized qr system might rewrite it to "play alphabetic song" based on the high overall transition probability from "abcdefg" to "alphabetic song". however, for this particular user, the query was intended for the song "abcdefu" by the american singer gayle. a personalized qr system is often designed as a search-based approach, which requires a user history index to capture historical non-defective user experiences. the user history index includes each user's own historical successful queries, rephrases, rewrites, and related metadata & statistics. during runtime, given a user query (e.g. "play abcdefg"), the system checks if a successful historical query utterance (e.g. "play abcdefu by gale") in the user history index closely matches the current query. the user interactions covered by the user history index are called the "seen interactions". despite the effectiveness of personalized qr for reducing defects in conversational ai, we have identified the challenge posed by "unseen interactions" not covered by the user history index. we have observed that users frequently engage in new experiences, leading to approximately 50% of the queries/interactions not being covered by the user history index. we refer to these queries/interactions as "unseen". moreover, unseen queries/interactions have a defect rate roughly 7% higher. this under-scores the potential benefits of query rewriting for these unseen interactions. we introduce our approach "collaborative query rewriting" (collaborative qr), designed to overcome the constraints of the user history index. this approach is inspired by our observation that users who interact with similar entities through a conversational ai often make similar queries or experience comparable defects (see figure 1 ). the cornerstone of our approach is the "user feedback interaction graph" (fig), which captures users' previous interactions with various entities through the conversational ai in a user-entity interaction graph (see section 3.2). our key idea is to leverage fig to form a collaborative user index consisting of additional rewrite candidates not found in the user history index. ## experiments in this section, we first demonstrate the effectiveness of collaborative qr, that the collaborative user index built through graph traversal can boost defect reduction, and we are able to achieve competitive precision performance with the much enlarged in3 we stack transformer layers as the l1 encoder model that can run within the latency budget. ## conclusion in this paper, we initially highlight the potential advantages of query rewriting in addressing users' unseen interactions. we then propose the "collaborative query rewriting" approach that aims to reduce defective queries arising from unseen interactions. performance degradation due to an enlarged index was rectified by implementing additional transformer layers for the l1 retrieval model and incorporating guardrail and graph features in the l2 ranking model. furthermore, we investigated the potential of an llm in enhancing the collaborative qr approach. we found great potential for an llm to significantly improve the coverage of the collaborative user index that can lead to a significant 5 to 6 times more reduction of query defects.