On the Robustness of Question Rewriting Systems to Questions of Varying Hardness (2022)

Abstract: In conversational question answering (CQA), the task of question rewriting (QR) in context aims to rewrite a context-dependent question into an equivalent self-contained question that gives the same answer. In this paper, we are interested in the robustness of a QR system to questions varying in rewriting hardness or difficulty. Since there is a lack of questions classified based on their rewriting hardness, we first propose a heuristic method to automatically classify questions into subsets of varying hardness, by measuring the discrepancy between a question and its rewrite. To find out what makes questions hard or easy for rewriting, we then conduct a human evaluation to annotate the rewriting hardness of questions. Finally, to enhance the robustness of QR systems to questions of varying hardness, we propose a novel learning framework for QR that first trains a QR model independently on each subset of questions of a certain level of hardness, then combines these QR models as one joint model for inference. Experimental results on two datasets show that our framework improves the overall performance compared to the baselines.

## introduction in conversational question answering (cqa) @xcite , several sequential questions need to be answered one by one given a relevant article. to answer a question in cqa, we need to understand the historical context of the question. for example, to answer the question "when did he begin writing these pieces?", we need to know what he refers to in the conversation context. in our work, we address questionin-context rewriting (qr), which aims to rewrite a context-dependent question into an equivalent selfcontained question in cqa, e.g., replacing he in the topic words: benigno aquino iii; senate @xcite q1: what changes did he make while in the senate? a1: i don't know. q2: when was he elected? → q ′ 2 : when was benigno aquino iii elected to senate? a2: may 15, 2007 q3: was he a republican or democrat? a3: genuine opposition (go), a coalition comprising a number of parties, including aquino's own liberal party, ... q4: are there any other interesting aspects about this article? → q ′ 4 : are there any other interesting aspects about benigno aquino iii article aside from political affiliation or when benigno was elected? a4: aquino was endorsed by the pentecostal jesus is lord church. in section 6.3, our first study is on evaluating the performance of a qr model under questions varying in hardness. one issue in this process is that there is a lack of classified questions in different rewriting hardness. though we can rely on human labor to annotate the questions, it is expensive and not scalable. instead, we propose a simple yet effective heuristic method to classify the questions automatically. we measure the discrepancy between a question and its rewrite, where the larger the discrepancy, the more difficult to rewrite the question. the intuition is that if a question is very dissimilar to its rewrite, more information has to be filled into the rewrite, which means the question is harder to rewrite. we specifically use the bleu score to measure the discrepancy, and lower scores mean larger discrepancies. using this method, we then split the questions into three subsets: hard, medium, and easy, and evaluate the baseline systems using these subsets. in order to verify the classified subsets and find out what makes questions different in rewriting difficulty, in section 6.3.2, we further evaluate the question characteristics in hard, medium, and easy subsets through human evaluation. we first manually summarize the commonly used rules for rewriting questions from the training set, and then annotate the questions using the labels of summarized rewriting rules, followed by counting the number of these rewriting rules used in these subsets. finally, to enhance the robustness of a qr model to questions varying in difficulties, we propose a novel learning framework in section 5, where we first separately train a qr model on each hard, medium, and easy subset, and then combine these models into a joint model for inference. training one sole model on each subset is to let the model better learn domain-specific information to deal with one specific type of questions (hard/medium/easy). by combining the models together, we have a joint model capable of rewriting questions differing in rewriting hardness. specially, we introduce adapters @xcite to reduce parameters when building private models and we present sequence-level adapter fusion and distillation (slaf and slad) to effectively combine the private models into a joint model. our contributions in this paper include: the difficulty of answering a question given a relevant document has been studied in the question answering community @xcite . @xcite examine 12 reading comprehension datasets and determine what makes a question more easily answered. perez et al. (2020); @xcite ; talmor and berant (2018); @xcite study how to make a hard question more easily answered. however, there is no work to date that studies whether rewriting difficulties exist in qr and how to measure the difficulties. some other work is similar to qr but focuses on other tasks such as dialogue tracking @xcite @xcite and information retrieval @xcite @xcite . varying rewriting difficulties can result in multiple underlying data distributions in the qr training data. the shared-private framework has been studied to learn from training data with multiple distributions @xcite . one issue of the shared-private framework is parameter inefficiency when building private models we use adapter tuning @xcite to build the private models. adapter tuning was recently proposed for adapting a pre-trained language model, e.g., bert @xcite , to downstream tasks @xcite , and its effectiveness has been verified by previous work @xcite @xcite . we are the first to apply it to reduce model parameters in the shared-private framework. how to combine the knowledge stored in multiple adapters is also important. @xcite propose adapter fusion to build an ensemble of adapters in multi-task learning. we propose sequence-level adapter fusion in our work. ## question-in-context rewriting question-in-context rewriting (qr) aims to generate a self-contained rewrite from a contextdependent question in cqa. given a conversational dialogue h with sequential question and answer pairs {q 1 , a 1 , • • • , q n , a n }, for a question q i from h with its history h @xmath0, a 1 , • • • , q i-1 , a i-1 }, we generate its rewrite q ′ i . ## difficulty of question rewriting the difficulty of rewriting a question varies across questions. we propose a simple yet effective heuristic to formulate rewriting difficulty as the discrepancy between a question and its rewrite. to generate a self-contained rewrite, we need to identify relevant information from the conversation context to incorporate it into the original question. we observe that if the discrepancy is large, we need to identify more missing information from the conversation context which makes the rewriting task more difficult. in this work, we use bleu score to measure the discrepancy. bleu has been widely used to measure how similar two sentences are @xcite . given a question q and its rewrite q ′ , we define the difficulty score z for rewriting q as: a low z score indicates a larger discrepancy between q and q ′ , making it more difficult to rewrite q into q ′ . besides bleu, we also study the effectiveness of rouge, lengths of q and q ′ , and |q|/|q ′ | in §6.5 to measure rewriting difficulty. ## difficulty-aware qr with adapters previous work on qr learns to rewrite questions with only one shared model @xcite , which cannot adequately model all questions with different rewriting difficulties. instead of using only one shared model, we propose a novel method to classify a question into several classes by measuring its rewriting difficulty ( §5.1), learn a private model for each class ( §5.2), and finally combine the private models for inference ( §5.3). different questions with varying rewriting difficulties result in multiple data distributions in the training set. by dividing the training data into several classes with varying rewriting difficulties, we can better learn the data distributions with the help of private models @xcite . ## conclusion in this work, we study the robustness of a qr system to questions varying in rewriting hardness. we use a simple yet effective heuristic to measure the rewriting difficulty. we further propose a novel method to deal with varying rewriting difficulties. tested on canard and qrecc, we show the ef