Pseudo Ambiguous and Clarifying Questions Based on Sentence Structures Toward Clarifying Question Answering System (2022)

Abstract: Question answering (QA) with disambiguation questions is essential for practical QA systems because user questions often do not contain information enough to find their answers. We call this task clarifying question answering, a task to find answers to ambiguous user questions by disambiguating their intents through interactions. There are two major problems in building a clarifying question answering system: data preparation of possible ambiguous questions and the generation of clarifying questions. In this paper, we tackle these problems by sentence generation methods using sentence structures. Ambiguous questions are generated by eliminating a part of a sentence considering the sentence structure. Clarifying the question generation method based on case frame dictionary and sentence structure is also proposed. Our experimental results verify that our pseudo ambiguous question generation successfully adds ambiguity to questions. Moreover, the proposed clarifying question generation recovers the performance drop by asking the user for missing information.

## introduction question answering (qa) is a conventional task of natural language processing to provide answers for given user questions. the advance of neural network-based qa systems has led to a variety of benchmark datasets of the qa task @xcite . these benchmarks define the problem of qa as predicting a corresponding phrase (span) in documents to a given question when the system has both questions and target documents. most qa tasks defined in existing benchmark qa datasets assumes that the given questions have enough information for answering. however, real questions given by users are often ambiguous because users frequently forget to mention important terms or may hesitate. it is thus not always easy to derive clear answers for such ambiguous user questions. for example, when a user says, "what is the masterpiece drawn by leonardo da vinci?", the system cannot determine an answer because leonardo da vinci created several notable masterpieces (figure 1 ; ambiguous q). taylor @xcite defined four level categories of user states in information search. what is the masterpiece drawn by leonardo da vinci? which museum displays the masterpiece? what is the motif? a woman the louvre the answer is "la joconde" problem 1 existing qa datasets do not include such ambiguous questions problem 2 how does the system generate appropriate clarifying questions? ambiguous q clarifying q1 clarifying q2 solution 1 pseudo ambiguous question generation using existing datasets solution 2 clarifying question generation using semantic structures problem on a qa system using question paraphrasing @xcite and building ambiguous question answering datasets @xcite . however, it is not easy to build a dataset that covers any variation of ambiguous questions because of the diverse variety of ambiguity in questions (figure 1 ; problem 1). moreover, even if we can define the variation of ambiguity; it is still challenging to find appropriate clarifying questions for the disambiguation to shape the system answers (figure 1 ; problem 2). sentence structures have an essential role in clarifying the meaning because we control the sentence clarity by modifiers in syntax. this indicates that the sentence generation system can also control sentences' clarity by focusing on sentence structures. based on this idea, in this work, we propose a pseudo ambiguous question generation method for covering variations of the ambiguous question, which are derived from clear questions collected in existing qa datasets (figure 1 ; solution 1). the proposed method focuses on the syntax structures of question sentences to add ambiguity by eliminating some parts while considering grammatical roles from syntax point of view. we also propose a clarifying question generation method based on the case frame, which uses the syntax and semantic information of ambiguous questions (figure 1 ; solution 2). the clarifying question generation makes it possible to disambiguate the user's meaning by interacting with the user directly to improve the qa system performance. we conducted two experiments to investigate the quality of proposed generation systems. qualities of the pseudo ambiguous questions are evaluated by both the qa system and the human subjective test. the performance of the clarifying question generation is investigated by qa system performance using both the ambiguous questions and answers to the clarifying questions given by crowdworkers. section 2 sets forth our problem definition and system overview. section 3 describes the pseudo ambiguous question generation method. section 4 explains the proposed clarifying question generation method that uses sentence structures. section 5 shows the evaluation setting and system performance to verify the ability of our generation system. we clarify the position of our system in relation to existing systems in section 6, and then conclude this work in section 7. ## system overview our final goal is to build a clarifying question answering system that can ask a question back to users if the given questions do not contain sufficient information to distinguish the answer. we call such questions as ambiguous questions. figure 2 shows the overall system. we extract questions from existing qa datasets to modify them to pseudo ambiguous questions because building ambiguous question datasets is costly @xcite . most of the existing qa datasets consist of pairs of clear questions and corresponding text spans on target documents. these questions are defined clearly to distinguish the answer terms from the document. in other words, if human experts receive these questions, they can find the answer from the documents even if it takes a lot of time. our proposal eliminates some important parts of these questions to generate pseudo ambiguous questions using their syntax information. in the example presented in figure 2 , the system adds ambiguity to the question by removing the verbal phrase that corresponds to the verb "developed." when did the writer have a series? pseudo ambiguous q generated clarifying q qa dataset what was the first comic book written by the writer who had a series developed into a 2010 film with bruce willis and morgan freeman? transmetropolitan what was the first comic book written by the writer who had a series developed into a 2010 film with bruce willis and morgan freeman? conversion clarifying q generation case frame 2010 ## pseudo ambiguous question generation it is not realistic to collect all possible varieties of ambiguous questions because possible ambiguous questions given to the qa system are diverse and depend on the situation that the users are facing. in this paper, we present a method to generate pseudo ambiguous questions by modifying questions in existing qa datasets. we apply syntax parsing to question sentences to focus on modifiers, which have a role in clarifying the question's intent, and then eliminate them from the questions to make the sentences ambiguous. this section describes the generation process and its evaluations. what was the first comic book written by the writer who had a series developed into a 2010 film with bruce willis and morgan freeman? what was the first comic book written by the writer who had a series developed into a 2010 film with bruce willis and morgan freeman? ## clarifying question generation we built clarifying question generation system toward a clarifying question answering system, asking a question back to the questioners. the proposed system generates clarifying questions using predicate-argument structures; it finds predicates in ambiguous questions and generates questions to clarify their arguments. we used the case frame dictionary @xcite for the generation, which consists of frequencies of cases and arguments depending on predicates. this section describes the technical details of clarifying question generation. ## experiments we evaluated the proposed clarifying question generation system. we gave the pseudo ambiguous question generated by the method presented in section 3 to the clarifying question generation described in section 4. ## related works we built a generation system that clarifies user's requests by clarifying questions when the user's questions are ambiguous. there are two major approaches for building a qa system that can withdraw additional information to the initial ambiguous user query. one approach is based on paraphrasing, which paraphrases ambiguous sentences to clear sentences. the other major approach is using clarifying or confirmation questions, which is similar to our system. this section describes relationships to these works.