D o QA - Accessing Domain-Specific FAQ s via Conversational QA (2020)

Abstract: The goal of this work is to build conversational Question Answering (QA) interfaces for the large body of domain-specific information available in FAQ sites. We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing. Compared to previous work, DoQA comprises well-defined information needs, leading to more coherent and natural conversations with less factoid questions and is multi-domain. In addition, we introduce a more realistic information retrieval (IR) scenario where the system needs to find the answer in any of the FAQ documents. The results of an existing, strong, system show that, thanks to transfer learning from a Wikipedia QA dataset and fine tuning on a single FAQ domain, it is possible to build high quality conversational QA systems for FAQs without in-domain training data. The good results carry over into the more challenging IR scenario. In both cases, there is still ample room for improvement, as indicated by the higher human upperbound.

## introduction the overarching objective of our work is to access the large body of domain-specific information available in frequently asked question sites (faq for short) via conversational question answering (qa) systems. in particular, we want to know whether current techniques are able to work with limited training data, and without needing to gather data for each target faq domain. in this paper we present doqa, a task and associated dataset for accessing domain-specific faqs via conversational qa foot_0 . the dataset contains 2,437 information-seeking question/answer dialogues on three different domains figure 1 : a dialogue about cooking. on top, the original post, comprising a topic and an excerpt of the answer passage. in italics, dialogue acts (cf. section 3). previous work on conversational qa datasets include coqa @xcite and quac @xcite . the main focus of coqa are reading comprehension questions, which are produced with access to the target paragraph. the topic of the questions are delimited by the paragraph, which leads to specific questions about details in the paragraph. @xcite observed that a large percentage of coqa answers are named entities or short noun phrases. in quac, the topic of the conversation is set by a title and first paragraph of a wikipedia article about people. the user makes up questions about the person of interest. note that, contrary to our setting, there is no real information need in any of those datasets, which can lead to less coherent conversations: any question about the paragraph or person of interest is valid, respectively. ## related work conversational qa systems stem from the body of work on reading comprehension, whose goal is to test the capacity of a system to understand a document by answering any question posed over its content. recent work on the field has resulted in the creation of multiple datasets @xcite @xcite @xcite . these datasets are typically composed of multiple question/answer pairs, often along with a reference passage from which the answer is curated. whereas the questions are always in free text form, some datasets represent the answers as a contiguous span in the reference passage, while others contain free form answers. the former are usually referred as extractive, whereas the latter are called abstractive. all in all, in these qa datasets the queries are unrelated to each other, and thus there is no dialogue structure involved. @xcite propose to answer complex queries by decomposing them into sequences of single, co-referent queries. the question sequence can be seen as different turns in a dialogue, and each question refers and refines previous ones. the authors present the sequentialqa dataset, which comprises 6k question sequences posed over the content of wikipedia tables. in the case of our task, it is the user who makes several questions in sequence. more similar to our work, coqa @xcite and quac @xcite are two conversational qa datasets comprising qa dialogues that fulfill the information need of a user by answering questions about different topics. similarly to our, both datasets are built by crowdsourcing, where one person (the questioner) is presented with a topic and has to pose free-form questions about it. another person (the answerer) has to select an answer to the question by choosing an excerpt from the relevant passage describing the topic. some of the questions in both datasets are unanswerable, and access to previous questions and answers are needed in order to answer some of the questions. coqa contains 127k questions with answers, obtained from 8k conversations about passages from broad domains, ranging from children stories to science. the answers are also excerpts from the relevant passage, but answerers have the choice of reformulating them. the authors report that 78% of the answers had at least one edit. although reformulating answers can yield to more natural dialogues, @xcite showed that span based systems can in principle obtain a performance up to 97.8 points f1, showing that editing the answers does not yield to systems with better quality. in coqa, both questioner and answerer have access to the full passage, which guides the conversation towards the specific information conveyed in it. quac is a dataset that contains 14k informationseeking question answering dialogues. the dialogues in quac are about a specific section in wikipedia articles about people. the answerer has access to the full section text, whereas the questioner only sees the section's title and the first paragraph of the main article, which serves as inspiration when formulating the queries. quac also contains dialogue acts in each turn, which are useful when collecting the dialogues, as they can be used by the answerer to indicate to questioner whether to continue making questions about the last answer or drift to other aspects of the topic. we will compare coqa and quac in more detail in section 4. previous conversational qa datasets provide the relevant document or passage that contain the answer of a query. however, in many real world scenarios such as faqs, the answers need to be searched over the whole document collection. in related question answering research, @xcite and @xcite combine retrieval and answer extraction on a large set of documents. in @xcite the authors propose decomposing complex questions into a sequence of simple questions, and using search engines to answer those single questions, from which the final answer is computed. we find that requiring the system to search for relevant documents and passages is more realistic, and doqa is the first conversational qa task incorporating this scenario. in contemporary work, @xcite al. (2019 ) present a question answering dataset for the technical support domain which focuses on actual questions posed by users and has a real-world size with only 600 training instances. it also requires systems to examine 50 documents per query. our work has similar motivations for setting up more realistic tasks, and is complementary in the sense that we cover non-technical domains and conversatioal qa. community question answering has been also the focus of two related tasks @xcite , where, given a new question and a collection of pre-existing questions and answers, the systems need to rank the answers that are most useful for answering the new question. ## dataset collection this section describes our conversational qa dataset collection process which consists of an interactive task designed for two crowdworkers in amazon mechanical turk (amt). ## dataset analysis overall statistics in this section we present an quantitative and qualitative analysis of doqa and we compare them to similar conversational datasets like quac and coqa, stressing its similarities and differences. we also have the results of a short survey that workers had to respond to at the end of each hit. on the one hand, the user had to give feedback on how satisfied was with the answers of the expert in a scale of 1-5. the average satisfaction was 3.9. on the other hand, the expert had to give feedback on how sensible were the questions and the helpfulness of the answers. the average scores obtained were 4.27 and 4.10, respectively, which makes the amt task satisfactory. naturalness one of the main positive aspects of our dataset is the naturalness of the dialogues that other similar datasets like quac do not have. the answers of doqa come from a forum where the answer text is directed to a person who posted the question, and does not come from a much formal text like wikipedia, as it is the case of quac. the naturalness and casual register of the former it is more adequate than the formal register of the latter for a conversational qa system. the dialogue in figure 1 is a clear example of such naturalness, where the expert answers to the user with casual and directed expressions like "you may want" and "you may be having". to verify whether dialogues in doqa are more natural than the ones in quac, we sample randomly 50 dialogues in doqa cooking domain and quac and performed a/b testing to determine which of the two dialogues is more natural. this test showed that 84% of the times doqa dialogues are more natural. this naturalness is probably caused because a dialogue in doqa is started by a user with a very specific aim or topic to solve in mind, and thus, follow-up questions are very related to previous answers, and all the questions are set within a context. in contrast, dialogues in quac do not show so clear objective and questions seem to be asked randomly. dialogues in doqa are ended when the initial information need of the user is satisfied and this adds naturalness to dialogues. further analysis of the samples showed that answers in doqa seem to be more spontaneous because they have more orality aspects, such as higher level of expressivity ("normally when i try they end up burned not crispy!", "my biggest worry here would be...", "hey let's not be hasty"), opinions ("i came across a suggestion to cover the lid...", "i'd recommend simply adding...", "it sounds like fermentation to me") and humor ("well yeah but booze is booze"). contrarily, answers in quac are more hermetic and do not show any features of orality or spontaneity that a dialogue should have. all these features make doqa dialogues look more natural. we also analyzed the remaining 16% cases where doqa dialogues appear less natural. in most of these dialogues there were responses that did not really answer the question. the following question (q) and answer (a) pairs are good examples of it: (q) "is the taste going to be significantly different?" (a) "there is cornstarch in confectioner's sugar"; (q) "how about reheating?" (a) "when you defrost it, do so in your fridge leaving it overnight so that it defrosts gradually"; (q) "can i use my potatoes or carrots if they already have some roots?" (a) "the green portions of a potato are toxic". in some of these cases the correct answer for the respective question is not in the answer text provided to the expert. if this was the case, the expert should answer "i don't know", instead of giving a nonsense answer. question types ## task definition given a textual passage and a question, traditional qa systems find an answer to the question within the passage. conversational qa systems are more complex, as they need to deal with a sequence of possibly inter-dependent questions. that is, the meaning of the current question may depend on the dialogue history. for this reason, a dialogue history comprised by previous question/answer pairs is also provided to the system. in addition, some dialogue acts have to be predicted as an output: yes/no answers, which are required for affirmation questions, and continuation feedback, which might be useful for information-seeking dialogues. we denote the answer passage as p, the dialogue history of questions and respective ground truth answers as {q 1 , a 1 , ...q k-1 , a k-1 }, current question as q k , the answer span a k which is delimited by its starting index i and ending index j in the passage p, and dialogue act list v. the dialogue act list contains {yes,no,-} values for predicting affirmation and {follow-up,don't follow-up} for continuation feedback. ## baseline models we present two strong baseline models to address our task. although the state-of-the-art evolves quickly, our choice has the benefit of simplicity and strong performance. bert we took the fine-tuning approach for qa of bert, which predicts the indexes i and j of the a k answer span given p and q k as input. this baseline has shown strong performance on qa datasets such as squad @xcite . ## evaluation evaluation metrics given the similarity between quac and doqa, we use the same evaluation metrics and criteria used in quac. f1 is the main evaluation metric and is computed by the overlap at word level of the prediction and reference answers. as the test set contains multiple answers for each question we take the maximum f1 among them. note that when computing f1 quac filters out answers with low agreement among human annotators. an additional f1-all is provided for the whole set. we also report heq-q (human equivalence score on a question level) which measures the percentage of questions for which system f1 exceeds or matches human f1. ## conclusion and future work the goal of this work is to access the large body of domain-specific information in the form of frequently asked question sites via conversational qa systems. we have presented doqa, a dataset for accessing domain specific faqs via conversational qa that contains 2,437 information-seeking dialogues on the cooking, travel and movies domain (10,917 questions in total). these dialogues are created by crowdworkers that play the following two roles: the user asks questions about a certain topic posted in stack exchange, and the domain expert who replies to the questions by selecting a short span of text from the long textual reply in the original post. the expert can rephrase the selected span, in order to make it look more natural. in contrast to previous conversational qa datasets, our dataset responds to a real information need, is multi-domain, more natural and coherent. doqa introduces a more realistic scenario where the passage with the answer needs to be retrieved. together with the dataset, we presented results of a strong conversational model, including transfer learning from wikipedia qa datasets to our faq dataset. our dataset and experiments show that it is possible to access domain-specific faqs using conversational qa systems with little or no in-domain training data, yielding quality which is comparable to those reported in quac. for the future, we would like to exploit the abstractive answers in our dataset, explore more sophisticated systems in both scenarios and perform user studies to study how real users interact with a conversational qa system when accessing faqs.