Q a D ial M o E : Question-answering Dialogue based Fact Verification with Mixture of Experts (2022)

Abstract: Fact verification is an essential tool to mitigate the spread of false information online, which has gained a widespread attention recently. However, a fact verification in the question-answering dialogue is still underexplored. In this paper, we propose a neural network based approach called question-answering dialogue based fact verification with mixture of experts (QaDialMoE). It exploits questions and evidence effectively in the verification process and can significantly improve the performance of fact verification. Specifically, we exploit the mixture of experts to focus on various interactions among responses, questions and evidence. A manager with an attention guidance module is implemented to guide the training of experts and assign a reasonable attention score to each expert. A prompt module is developed to generate synthetic questions that make our approach more generalizable. Finally, we evaluate the QaDialMoE and conduct a comparative study on three benchmark datasets. The experimental results demonstrate that our QaDialMoE outperforms previous approaches by a large margin and achieves new state-of-the-art results on all benchmarks. This includes the accuracy improvements on the HEALTHVER as 84.26%, the FAVIQ A dev set as 78.7%, the FAVIQ R dev set as 86.1%, test set as 86.0%, and the COLLOQUIAL as 89.5%. To our best knowledge, this is the first work to investigate a question-answering dialogue based fact verification, and achieves new state-of-the-art results on various benchmark datasets.

## introduction fact verification, aiming to validate the factuality of claims against a corpus of documents, is an important nlp area @xcite and has been explored to various applications such as detecting fake news, rumor, and deceptive opinions @xcite @xcite @xcite . the majority of existing research focuses on media such as news, tables and wikipedia passages @xcite , while rarely consider the fact verification in the question-answering dialogue. in a dialogue safety domain, related studies either focus on enabling dialogue agents to resist adversarial attacks @xcite or on forestalling aggressive or biased responses from dialogue agents @xcite @xcite . however, misinformation online can spread quickly and cause public health crises due to the abuse of dialogue agents, especially questions about the pandemic of @xcite . the first example in figure 1 shows a popular question about covid-19 asked by information seekers online. the question-answering dialogue may be more vulnerable to be manipulated, since internet users can answer the question with multiple facts or speculative and vague expressions @xcite that deliberately distribute misinformation. for improving the robustness of fact verification systems, they must also be valid for verifying the responses in question-answering dialogues. the majority of previous works for the fact verification mainly focused on reasoning against pieces of evidence from wikipedia passages, while rarely considered questions sought by internet users. however, the questions also contain rich information to support the fact verification. figure 1 shows two examples for the question-answering dialogue based fact verification, where the questions were posed by real users. we can see that the questions contain some critical parts (e.g., "animals", "people", "who"), which indicate the confusions of information seekers and the vulnerable part of responses. taking the consideration above, we explore the fact verification in the question-answering dialogue and investigate how to exploit questions in the verification process. in this paper, we present qadialmoe, a neural network approach for question-answering dialogue based fact verification with mixture of experts. inspired by that mixture of experts is applied in both dialogue systems @xcite and fact verification fields @xcite , we implement each expert with the same neural architecture to focus on different parts of inputs (e.g., the relationship between the response and the question). specifically, to make our approach more generalizable, we propose a prompt module to generate questions in case that the original data only has responses. then each expert takes the same feature as the input from the output of feature extractor module and learns to deal with the meaning of questions, responses and evidence. we design a management module to guide the training of experts by assigning a unique attention score to each expert, and combine their verification results efficiently. however, previous models tend to incorrectly predict a response as supported when there is a significant overlap between the response and the evidence. similarly, it incorrectly predict supported or refuted for a nei response because of the word overlap. note that nei is short for notenoughinfo. to alleviate this problem, we introduce an attention guidance module to generate a prior assumption and guide the manager paying more attention to the input part with few word overlap. we conduct experiments on three benchmark datasets healthver @xcite , faviq @xcite and colloquial @xcite . experimental results demonstrate that our model outperforms previous systems by a large margin and achieves new state-of-the-art results on all of them. the main contributions of this paper can be summarized as follows: in this paper, we study the task of questionanswering dialogue based fact verification. given a question q, a response r and evidence e from wikipedia passages, the goal is to verify the factuality of the response by the question and evidence with the label supported or refuted. beyond the label as supported or refuted, the classification task has one more label called neutral or nei, which means no enough information and cannot make a decision. then, the task becomes a 3-way classification task. prior works have used question-answering dialogue data to create fact verification benchmarks @xcite @xcite @xcite . most fact verification processes only use evidence while rarely considering questions. however, we believe that the additional question that contains rich information is helpful to support the final prediction. in this study, we employ questions and evidence to validate the responses, which we formulate as the questionanswering dialogue based fact verification task. ## methodology in this section, we present our proposed framework qadialmoe, that leverages a set of experts to simultaneously consider the meaning of questions, responses and evidence from wikipedia passages. the overall model structure is illustrated in figure 2. our method consists of three components: the feature extractor ( §3.1) with a prompt module and a transformer encoder backbone, the mixture of experts module ( §3.2) for dealing with different parts of input, and the management module ( §3.3) for guiding the training of experts and combining their ability of verification effectively. ## conclusion in this paper, we present qadialmoe, a new method for fact verification in question-answering dialogue that exploits the mixture of experts to focus on various interactions among responses, ques-tions and evidence. we also generate synthetic questions with a prompt module to make our approach more generalizable. a manager with an attention guidance module is applied to guide the training of experts and assign a reasonable attention score to each expert. experimental results on three datasets healthver, faviq and colloquial demonstrate that qadialmoe outperforms previous approaches by a large margin and achieves new state-of-the-art results on all of them. the ablation studies and analysis further indicate that questions and evidence play an equal important role in our proposed framework. we hope our work can facilitate fact verification in a question-answering dialogue domain, and open the way to efficiently exploit questions and evidence in the verification process.