Unified Contextual Query Rewriting (2023)

Abstract: Query rewriting (QR) is an important technique for user friction (i.e. recovering ASR error or system error) reduction and contextual carryover (i.e. ellipsis and co-reference) in conversational AI systems. Recently, generation-based QR models have achieved promising results on these two tasks separately. Although these two tasks have many similarities such as they both use the previous dialogue along with the current request as model input, there is no unified model to solve them jointly. To this end, we propose a unified contextual query rewriting model that unifies QR for both reducing friction and contextual carryover purpose. Moreover, we involve multiple auxiliary tasks such as trigger prediction and NLU interpretation tasks to boost the performance of the rewrite. We leverage the text-to-text unified framework which uses independent tasks with weighted loss to account for task importance. Then we propose new unified multitask learning strategies including a sequential model which outputs one sentence for multi-tasks, and a hybrid model where some tasks are independent and some tasks are sequentially generated. Our experimental results demonstrate the effectiveness of the proposed unified learning methods.

## introduction large-scale conversational ai agents such as alexa, siri, and google assistant, are becoming increasingly popular in real-world applications to assist users in daily life. however, some of the user interactions lead to dissatisfied experiences, where users do not get what they requested or the assistant has to engage with the user again to clarify the user request. these user frictions arise from errors in the system, including automatic speech recognition (asr) and natural language understanding (nlu), as well as user ambiguity and background noise. the goal of qr @xcite is to identify the queries that lead to friction and rewrite them to queries without changing the users' intention, in order to mitigate defective interactions. besides, in a multi-turn dialogue session with agent, users sometimes tend to use incomplete utterances which usually omit or refer back to entities or concepts that appeared in the previous dialogue, namely ellipsis, and co-reference. thus, we also always rely on contextual carryover @xcite to rewrite the incomplete query into a context-dependent and self-contained query. although query rewriting has received a lot of attention in recent years, it has always been studied in two separate directions, i.e., reduce defects and contextual carryover. recent works integrate all functionalities into pre-trained sequence-to-sequence (seq2seq) language models and report impressive results @xcite . this synergy has resulted in a great deal of recent work developing transfer learning methodology for nlp. inspired by that, we propose a unified contextual query rewriting framework, that can utilize one model to perform the rewrite for both friction reduction and contextual carryover. moreover, we involve two additional tasks in the unified model: nlu interpretation of rewrite and rewrite trigger prediction. nlu interpretation of rewrite is a task to predict the domain, intent, and entity slots information. trigger prediction is a task that enables the model to decide to trigger a rewrite or not. these tasks can not only be used for downstream modules but also serve as auxiliary tasks to boost the primary task query rewriting performance. specifically, we leverage the bart model @xcite which is a largescale seq2seq framework. we unify defect reduction and contextual carryover as one qr task, where the model input is a dialogue context along with the current request, and the output is the rewrite. for nlu interpretation and trigger prediction tasks, we also cast them as a text-to-text generation task, where the model output the trigger decision and the nlu interpretation for the current request. motivated by the concept of in-context learning @xcite , to steer the model to solve different sub-task, we plug a task-specific prompt, into the model input. this way, the generations of different sub-tasks are decoupled, leading to better flexibility of the model regarding generation for each sub-tasks. besides the traditional text-to-text unified learning approach @xcite in which each task's prediction is generated independently, we explored variants of unified learning approach, including sequential unified learning where one sequence is used to generate multi-task results using target prompts and hybrid unified learning where some tasks are independent and some tasks are sequentially generated. we conducted extensive offline experiments to study the proposed unified learning approaches. our experimental results demonstrate the effectiveness of the proposed approaches. our production simulation validates the positive impact of the proposed model which indeed generates rewrites of better quality. ## related work query rewriting in dialogue systems, query rewriting benefits dialogue state tracking especially co-reference resolution @xcite , and reducing users' friction by replacing the users' utterance @xcite @xcite . @xcite and @xcite propose to leverage the search-based model, which consists of a dssm based retrieval layer and a tree ranking layer, to handle global and personalized query rewriting. @xcite use generation-based approaches to tackle the co-reference and omission-specific scenarios. @xcite propose a constrained generation based seq2seq model for query rewriting. contextual carryover contextual carryover has been an important component in dialogue systems for resolving co-reference and omission. @xcite leverage an encoder-decoder architecture for making independent carryover decision for each slot in the context. later, @xcite propose a framework to jointly predict whether a subset of related slots should be carried over from dialogue history. @xcite formulate the contextual carryover problem as a contextual query rewriting problem (cqr). @xcite present a few-shot generative approach to conversational query rewriting. in this work, we unify the query rewiring task with cqr by one text-to-text generation model, with generated rewrite handling contextual slot carryover cases. unified learning transfer learning in natural language processing (nlp) has gained popularity due to its demonstrated effectiveness. this approach involves pre-training a model on a data-rich task and fine-tuning it for a specific task @xcite @xcite . later, the efficacy of transfer learning has been further improved by a unified framework that converts all text-based language problems into a text-to-text format presented through the t5 model @xcite . in this paradigm, instead of adapting the pre-trained language model (lm) to downstream tasks via objective engineering, downstream tasks are reformulated to look more like those solved during the original lm training with the help of a textual prompt. ## methodology before introducing the unified learning model, we first establish the baseline for the query rewriting. we formulate the query rewriting as a sequenceto-sequence (seq2seq) task and fine-tune a pretrained bart model for the rewrite generation. as shown in figure 1 , the model takes the current turn and its previous dialog context as input and generates the target text autoregressively. the seq2seq architecture comprises a bidirectional encoder that takes the context and current request as input, and an autoregressive decoder that performs constrained decoding to generate the target rewrite. ## experiments we conduct two sets of experiments to evaluate the proposed unified learning models. the first set of experiments compares the three proposed unified models in terms of their effectiveness on the query rewrite, trigger prediction, and nlu hypothesis generation. the second set of experiments evaluates the benefit of integrating contextual carryover into the query rewriting task. ## conclusion in this work, we propose unified learning approaches for qr. the proposed approach unifies several tasks into one text-to-text model. besides, the proposed approach unifies general rewrite tasks with contextual carryover tasks. we explored multiple unified learning scenarios such as parallel multitask learning, sequential multi-task learning, and hybrid multi-task learning. our experimental results and production simulation demonstrated the superiority of the unified learning model.