C ycle KQR : Unsupervised Bidirectional Keyword-Question Rewriting (2022)

Abstract: Users expect their queries to be answered by search systems, regardless of the query’s surface form, which include keyword queries and natural questions. Natural Language Understanding (NLU) components of Search and QA systems may fail to correctly interpret semantically equivalent inputs if this deviates from how the system was trained, leading to suboptimal understanding capabilities. We propose the keyword-question rewriting task to improve query understanding capabilities of NLU systems for all surface forms. To achieve this, we present CycleKQR, an unsupervised approach, enabling effective rewriting between keyword and question queries using non-parallel data.Empirically we show the impact on QA performance of unfamiliar query forms for open domain and Knowledge Base QA systems (trained on either keywords or natural language questions). We demonstrate how CycleKQR significantly improves QA performance by rewriting queries into the appropriate form, while at the same time retaining the original semantic meaning of input queries, allowing CycleKQR to improve performance by up to 3% over supervised baselines. Finally, we release a datasetof 66k keyword-question pairs.

## introduction information retrieval and question answering (qa) systems aim to fulfill user requests, from traditional web search to voice assistants. users can issue queries in two forms @xcite : as keyword queries ("iphone 14 price") or as fullyformed natural language questions ("how much is an iphone 14?"). natural language understanding (nlu) components are expected to correctly interpret semantically equivalent queries, regardless of the form. figure 1 ( @xcite shows examples of queries with the same intent, but in different forms. work on question paraphrases has grown recently, with current approaches utilizing encoderdecoder architectures for generating questions @xcite . however, nlu systems that handle different input forms must support two important functionalities beyond questions: to address these, we introduce the keywordquestion rewriting task, which aims to rewrite questions into semantically equivalent keyword queries, and vice versa. this task enables training models for the above desiderata, allowing nlu systems to handle either keywords or questions. further, this capability enables asking clarification questions about user's input query @xcite , (cf. figure 1 (b) ), by rewriting it into a question. while previous work has tackled the problem of rewriting keywords into questions @xcite , it focuses on unidirectional (@xmath0. we propose a bidirectional and unsupervised approach in this paper. specifically, we propose cyclekqr, an approach based on cycle-consistent training @xcite , which is an unsupervised approach that effectively maps two non-parallel sets of keywords and questions via two cycles, keywords-to-question (k2q) and questionto-keywords (q2k). these two cycles learn interactively from each other, and k2q & q2k are trained using reconstruction losses. although unsupervised, cyclekqr is more robust since the its rewriting models are trained simultaneously, while a supervised approach has to train both models in two rounds. as there are no existing datasets for keyword-question rewriting we also construct and release a dataset of 66k keyword-question pairs. we focus on (i) intrinsic and (ii) extrinsic evaluation strategies. for intrinsic evaluation, we measure the bidirectional rewriting quality of competing approaches using human studies and automated metrics such as rouge and bleu. our extrinsic evaluation focuses on qa systems, where the ability to correctly interpret the user query is critical, and if necessary, query rewriting approaches are applied to ensure answer accuracy @xcite . more concretely, we measure the downstream impact on qa performance using the rewritten keyword or question queries. 2 we assess the impact on two types of qa systems: open domain and knowledge base qa systems. experiments show that qa systems perform poorly on query forms that they are not trained on. we show that cyclekqr can effectively rewrite between keyword and question queries without annotations, and at the same time improve qa performance. cyclekqr obtains qa performance improvements of up to 3% over a supervised baseline using the same neural architecture. this is due to the ability to cope with noisy input and target query pairs, given that cyclekqr computes a reconstruction loss in each cycle, thus allowing it more flexibility in the rewriting step, and avoiding overfitting on noisy pairs. in sum, we make the following contributions: • introduce a keyword-question rewriting task for improved query understanding; • propose cyclekqr, an unsupervised approach for keyword-question rewriting; • release a publicly available dataset with 66k pairs for keyword-question rewriting. our task focuses on bidirectional keyword-question rewriting, with the aim of improved query understanding, while retaining their semantic meaning. two techniques have been proposed for paraphrasing and style transfer: rule-based and generative approaches. @xcite compare rulebased paraphrasing approaches including phrase replacement (e.g. using wordnet), translation and tense change. gröndahl and asokan (2020) employ simple rules (e.g. changing the tense of the sentences) to generate multiple candidates, and select the best one using a ranker. @xcite propose a neural encoder-decoder model to generate and control the output questions using question templates. @xcite first normalize a given sentence by removing its style and then apply gpt2 to transfer it to a desired style. rules and templates used in existing works are domain and dataset specific, and obtaining them for any target domain and dataset is time-consuming, hence, we focus on encoder-decoder architectures. question-keyword rewriting. the first attempts to rewrite keyword-based queries into questions focused on using templates @xcite @xcite . question templates are typically obtained from user-clicked questions, and for a given a keyword query, the most relevant template is used to generate a question. @xcite propose a statistical model to generate a synthetic set of question-keyword query pairs. given a question, they generate a keyword query by randomly sampling based on its length and the included terms. the pairs are then used to train a neural model @xcite . the resulting keyword queries are highly noisy and require additional filtering before being used. moreover, it does not guarantee that the output will retain the meaning of the question. ## task definition we define the task of rewriting keywords into question queries and vice-versa through two rewriting functions: (i) keywords-to-question (k2q), and (ii) question-to-keywords (q2k). let q be a set of natural language questions, and k a set of keyword queries. k2q rewrites a keyword query @xmath1, i.e. @xmath2. similarly, its inverse function, q2k, rewrites a question into its corresponding keyword query, @xmath3. for example, given the keyword query "houston tx county", k2q will output the question "what county is houston tx in?". conversely, given the question "where was martin luther king born?", q2k will output "martin luther king birthplace". both k2q and q2k must preserve the semantic meaning of the original input into the generated output, i.e. both the input and output must convey the same information need. ## cyclekqr: cycle-consistent training for keyword-question rewriting figure 2 shows an overview of our approach. we now describe how we use cycle-consistency to train k2q and q2k for keyword-question rewriting. overview: for the k2q and q2k, cyclekqr uses two encoder-decoder models @xcite with parameters denoted as θ and ϕ. to train them, we follow iterative back-translation @xcite , in which the output of each model is used to generate intermediate training data for the other. the core idea behind cycle-consistent training is to jointly train the k2q and q2k functions, allowing cyclekqr to align the latent spaces of keyword and question queries. in the following, we describe the two training cycles. ## conclusion we introduced the bidirectional keyword-question rewriting task, which improves qa performance by rewriting queries into the desired form for a given qa system. furthermore, we presented cy-clekqr, which learns the k2q and q2k functions simultaneously through cycle-consistent training in an unsupervised manner. cyclekqr allows for the two rewriting functions to be less susceptible to noise coming from unsupervised data, and can be easily adapted to different qa systems and scenarios, for which non-parallel data is available. we carried out a detailed evaluation of different competing rewriting approaches for two qa scenarios (open domain and knowledge base), assessing their performance through an intrinsic and extrinsic evaluation. experimental results show that rewriting queries into the correct form improves qa performance, and that cyclekqr is able to provide highly accurate rewrites, which retain the original query intent and improve the ranking of the correct answer for the underlying qa system. finally, we contributed a keyword-question rewriting dataset, consisting of a total of 66k keyword-question pairs which can be used to facilitate research on keyword-question rewriting.