Answering Unanswered Questions through Semantic Reformulations in Spoken QA (2023)

Abstract: Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech that can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.

## introduction question answering (qa) is a longstanding nlp task, and voice assistants like alexa have made spoken qa ubiquitous. users often address such assistants with spontaneous speech, as they would a human. however, differences between spoken and written language @xcite , such as the presence of disfluencies, informal or incomplete speech, and different syntax have been shown to pose challenges for nlp tasks @xcite @xcite . qa system mostly use written data, and such phenomena impact question understanding and answer retrieval @xcite , leading to irrelevant answers or unanswered questions, leaving users unsatisfied. recently, language generation has been used to improve qa through question rewriting (qr). for example, qr is used in conversational systems to answer contextual questions in multi-turn dialogues @xcite . while qa models can be improved with fine-tuning, real-world systems have multiple qa backends and retraining is expensive, making input rewriting a practical solution @xcite . this has the added benefit that a single qr model may improve multiple qa systems. we propose applying qr to reformulate difficult or unanswered questions. we analyzed millions of answered and unanswered real-world questions from a leading voice assistant to understand the factors impacting qa failure ( §3). in addition to the well-known issue of disfluencies, we identify novel challenges from question structure and specificity. to address them, we propose three linguisticallyinformed reformulation operations that only require the question ( §4). the operations, shown in figure 1 , are designed to improve answerability foot_0 based on common speech patterns, so that for a previously unanswered question, the same qa system is able to provide an answer for its reformulation. while question repair has been studied, our root wh-and question generalization operators are novel contributions of this work. our results demonstrate that our approach can achieve: live deployment of our model ( §6.4) achieves positive impact for millions of users with unanswered questions, and explicit relevance feedback from customers shows high satisfaction. question complexity: depending on the qa system, some questions may be more difficult to answer. it has been shown that questions requiring multi-hop reasoning are more challenging @xcite , often leading to no answers or wrong answers. questions are affected by the broader types of syntactic complexity explored in the field @xcite @xcite . regardless of complexity, questions may also be unanswerable due to incorrect framing or false suppositions @xcite . other work has analyzed questions in different datasets, showing that wh-* words (e.g. who, what, when) are the dominant way to start a question @xcite , and that these words and related phrases (e.g., "how much", "how large") are associated with reduced answering complexity @xcite . in our work we consider how controlled syntactic restructuring can address the above challenges to reduce answering complexity. question rewriting: rewriting questions is a natural extension of query reformulation approaches used to improve information retrieval @xcite . question rewriting has been applied to improve qa in different ways. question paraphrasing has been used as a data augmentation approach to retrain qa systems to improve robustness (gan and ng, 2019). @xcite propose using a reinforcement learning agent between the original question and a black box qa system. the agent probes the qa system with several reformulations to learn how to elicit the best answer. @xcite propose a question refinement system to rewrite malformed questions. rewriting operations: text rewriting is based on specific linguistic changes. nassar et al. (2019) note that text simplification changes can be lexical (rare words replaced by more common ones) and syntactic (complex structures are split, reordered, or deleted). @xcite notes that paraphrasing questions is more difficult as the interrogative structure is separate from the declarative, and can have many variations. they quantified paraphrasing operations and showed that interrogative reformation accounted for 50% of changes, followed by lexical substitution (25%) and semantic changes (16%). recent work on sentence rewriting has followed this direction, by breaking down reformulation into predefined editing operations @xcite . our work is inspired by all of the above, but differs in several ways. we expand on the known issues in qa by analyzing real voice assistant data to identify prevalent challenges to tackle; we consider malformed question correction as a prerequisite for dealing with challenges of complex questions. additionally, prior rewriting approaches aim to improve qa via retraining, or by building a rewriter tailored to a single qa system. we take a different approach that does not rely on answer data or qa system feedback, and build a general model that can benefit multiple qa systems in a federated architecture. instead of uncontrolled paraphrasing, we deal with question complexity via controllable reformulations that distinguish between lexical modification, interrogative clause restructuring, and semantic changes. we propose novel linguistic restructuring operations to deal with complex syntax, and generalize high-specificity elements. scope: we limit our work to questions that were not answered due to retrieval failure, but may potentially have relevant answers if reformulated. they must be valid questions (seek knowable knowledge) whose information need can be understood (by humans) and re-stated. qa may fail for other reasons; we do not consider such issues e.g., inter alia, asr errors, invalid or difficult to understand questions, subjectivity, and other reasons for retrieval failure. a quantitative and qualitative study was undertaken by domain experts (details in appendix a), and identified the below challenges (c1-7) as contributing to a significant proportion 2 of failed requests, and potentially solvable by reformulation. c1. malformed utterances: questions with disfluencies and syntactic errors were more likely to fail e.g., fig. 1 (q1 ). correction methods have previously been used to fix these @xcite . c2. lexical gaps questions framed colloquially or lacking appropriate parlance for a topic e.g., fig. 1 (q2), were associated with failure. this is caused by lexical gaps @xcite arising from language mismatch between the user input and answer sources, as qa systems use formal knowledge sources for retrieval. lexical substitution and rephrasing may address this challenge. c3. complex syntactic structure: utterances with complex structure, such as multi-clause questions, can lead to qa failure. such phrasing is more common in spoken language, and can be simplified via syntactic restructuring, e.g., fig. 1 2 3 4 5 ). c4. polar propositions: yes-no questions are asked to confirm a specific proposition, e.g., "do box turtles live in japan?". answering polar questions is more difficult than wh-questions for both humans @xcite and qa systems @xcite , due to the entailment and inferences required to arrive at an answer. this can be simplified by reformulating to a factoid whquestion, e.g., "where do box turtles live?". 2 the exact numbers cannot be divulged for confidentiality. c5. false presuppositions: additionally, polar questions may contain false presuppositions that cause retrieval failure @xcite . we hypothesize rewriting such questions to wh-questions may retrieve relevant answers, e.g., fig. 1 ## surf question reformulation model we now describe our proposed semantic question reformulation model (surf) and the reformulation operators that it supports. ## results and discussion we now turn to a discussion of the results for the intrinsic (accuracy) and extrinsic (answer rate and relevance) evaluation strategies. reformulation accuracy and answer relevance. for answer relevance, in brackets are shown the extrapolated estimations of the absolute percentages of answered questions from ## conclusion we tackled the problem of improving spoken qa, and analyzed questions from live data to identify key challenges that could be addressed with reformulation. based on this we proposed surf with novel linguistically-motivated reformulation operators to solve the identified challenges. offline experiments show the effectiveness of our novel root transformation and generalization operations, with up to 24% of unanswered questions being answered via reformulations with high answer relevance. live deployment in a leading voice assistant has positively impacted millions of requests. we showed reformulation helps qa systems adapt to spoken user questions. we presented key insights from a deployed solution showing that performance can be significantly increased, without changing the underlying qa backends, by simply improving questions in their syntax and semantics.