CGF : Constrained Generation Framework for Query Rewriting in Conversational AI (2022)

Abstract: In conversational AI agents, Query Rewriting (QR) plays a crucial role in reducing user frictions and satisfying their daily demands. User frictions are caused by various reasons, such as errors in the conversational AI system, users’ accent or their abridged language. In this work, we present a novel Constrained Generation Framework (CGF) for query rewriting at both global and personalized levels. It is based on the encoder-decoder framework, where the encoder takes the query and its previous dialogue turns as the input to form a context-enhanced representation, and the decoder uses constrained decoding to generate the rewrites based on the pre-defined global or personalized constrained decoding space. Extensive offline and online A/B experiments show that the proposed CGF significantly boosts the query rewriting performance.

## introduction large-scale conversational ai agents such as alexa, siri and google assistant help millions of users to perform a lot of tasks, such as playing music, controlling light devices at home, etc. in general, such conversational ai agents have multiple components including automatic speech recognition (asr) and natural language understanding (nlu). asr is responsible for converting the speech signal of the user's query (e.g., "play michael jackson music") to a text transcript. following this, nlu provides domain/intent classification (e.g., domain: music, intent: playmusic) and entity labelling (e.g., artist-name: michael jackson), which are used to fulfill the user's request. however, sometimes there are frictions due to speech recognition or nlu errors. for example, asr errors may lead to an erroneous transcript "play alien bridges", when the user actually meant "play leon bridges". due to such errors, the downstream nlu system is affected, capturing a wrong * work done when working at amazon. many existing qr systems use search-based pipelines for either global-wise query rewriting @xcite or personalized query rewriting @xcite . these systems typically have two steps: retrieval and ranking. users' historical defect-free interactions with conversational agents are used to construct the global or personalized index. when a new request arrives, the system compares it to those utterances in the index using a retrieval model such as a dual encoder with billion-scale similarity search (e.g., faiss) @xcite and retrieves top n candidates from the index. then a ranking model is used to rank these candidates with both neural semantic and ir features as input. the system picks the top 1 ranked candidate as the final rewrite. such a search-based system is widely used in the large scale conversational ai agents since it can effectively control the output because of the use of the index and thus reduce the risky rewrites. however, there are also limitations in such retrieval-based systems. first, the query and rewrite candidate affinity is mainly captured through a vector dot product and lacks token level modeling. second, a large memory footprint is needed to store dense representations when a large index is used in the retrieval step. in this work, we propose to leverage generationbased models under the constrained generation framework (cgf) for the query rewriting task. since little work has incorporated the previous context information in query rewriting, although its importance is recognized @xcite , we use the previous dialog context and the user's current request in the encoder. the decoder uses constrained decoding in inference to force the generated rewrite to be in a predefined candidate set. the proposed cgf enables us to mitigate the aforementioned shortcomings from the search-based system since the autoregressive formulation allows the model to directly capture relations between the contextual input and target rewrites and thus effectively cross encode both. moreover, the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with the vocabulary size, not the index count. though the neural language generation approaches are known to hallucinate content, our proposed constrained decoding approach with a predefined candidate set makes the generation model faithful to the model input and avoids the potential hallucinations or bad rewrites. we conducted extensive offline experiments for both global and personalized query rewriting to show the effectiveness of the proposed approach. our online experimental results also demonstrate that the proposed cgf indeed generates rewrites of better quality. ## cgf for query rewriting as shown in figure 1 , we introduce the sequenceto-sequence (seq2seq) model to generate the rewrite, where a bidirectional encoder takes the context and current request as input, and an autoregressive decoder relies on the pre-defined index to perform the constrained decoding in order to generate the target rewrite. ## conclusion in this work, we propose cgf, a novel paradigm for query rewriting: generate target rewrite autoregressively with context-enhanced encoding and constrained decoding. cgf is a general framework for different query rewriting purposes where one can original request (w/o context): agent response (before rewrite): user: how old is tommy in it here's what i found from the web: defect reduction: a 24-year-old man whose... 50.20% cgf rewrite: agent response (after rewrite): how old is tommyinnit tommyinnit is 17 years old. original request (w/o context): agent response (before rewrite): user: play night talk by drake i couldn't find night talk defect reduction: cgf rewrite: agent response (after rewrite): play knife talk by drake here's knife talk , by drake (feat. 41.52% 21 savage, project pat) , on amazon music. original request (w/ context): agent response (before rewrite): user: play little yancy agent: lil' fancy from apple music.y sorry, i'm having trouble finding the song. defect reduction: user: play little yankees praise part 22.71% cgf rewrite: agent response (after rewrite): play little yancy praise party here is little yancy praise party.