F a VIQ : FA ct Verification from Information-seeking Questions (2022)

Abstract: Despite significant interest in developing general purpose fact checking models, it is challenging to construct a large-scale fact verification dataset with realistic real-world claims. Existing claims are either authored by crowdworkers, thereby introducing subtle biases thatare difficult to control for, or manually verified by professional fact checkers, causing them to be expensive and limited in scale. In this paper, we construct a large-scale challenging fact verification dataset called FAVIQ, consisting of 188k claims derived from an existing corpus of ambiguous information-seeking questions. The ambiguities in the questions enable automatically constructing true and false claims that reflect user confusions (e.g., the year of the movie being filmed vs. being released). Claims in FAVIQ are verified to be natural, contain little lexical bias, and require a complete understanding of the evidence for verification. Our experiments show that the state-of-the-art models are far from solving our new task. Moreover, training on our data helps in professional fact-checking, outperforming models trained on the widely used dataset FEVER or in-domain data by up to 17% absolute. Altogether, our data will serve as a challenging benchmark for natural language understanding and support future progress in professional fact checking.

## introduction fact verification, the task of verifying the factuality of the natural language claim, is an important nlp application @xcite and has also been used to evaluate the amount of external knowledge a model has learned @xcite . however, it is challenging to construct fact verification data with claims that contain realistic and implicit misinformation. crowdsourced claims from prior work such as fever @xcite are figure 1 : an example of a refute claim on faviq, constructed using ambiguity in the information-seeking question, e.g., through a crossover of the year of the film being released and being filmed. in this paper, we show it is possible to use information-seeking questions @xcite and their ambiguities @xcite to construct a large-scale, challenging, and realistic fact verification dataset. information-seeking questions are inherently ambiguous because users do not know the answers to the questions they are posing. for example, in figure 1 , the question is ambiguous because the filming of the movie and the release of the movie can both be seen as the creation time. we introduce a new dataset faviq-fact verification derived from information-seeking questions, which uses such ambiguities to generate challenging fact verification problems. for instance, the claim in figure 1 requires the model to identify that the movie released in 2001 is in fact filmed in 2000 and to return refute. like this, claims generated through the crossover of the disambiguation of information-seeking questions are likely to contain misinformation that real users are easily confused with. we automatically generate such claims by composing valid and invalid question-answer pairs and transforming them into textual claims using a neural model. the data is further augmented by claims from regular questionanswer annotations. in total, faviq consists of 188k claims. we manually verified a subset of claims to ensure that they are as natural as human-written claims. our analysis shows that the claims have significantly lower lexical bias than existing crowdsourced claims; claims involve diverse types of distinct entities, events, or properties that are semantically close, being more realistic and harder to verify without a complete understanding of the evidence text. our experiments show that a model with no background knowledge performs only slightly better than random guessing, and the state-of-the-art model achieves an accuracy of 65%, leaving significant room for improvement. furthermore, training on faviq improves the accuracy of verification of claims written by professional fact checkers, outperforming models trained on the target data only or pretrained on fever by up to 17% absolute. together, our experiments demonstrate that faviq is a challenging benchmark as well as a useful resource for professional fact checking. to summarize, our contributions are three-fold: ## related work fact verification fact verification is crucial for real-world applications @xcite and as a benchmark to evaluate the knowledge in a model @xcite . one line of work has studied professional fact checking, dealing with claims collected by professional fact checkers in specific domains @xcite @xcite . while such data contains realistic claims that have occurred in the real world, it is expensive to construct as it requires labor from professional fact checkers. moreover, it is less suitable as a benchmark due to lack of a standard evidence corpus such as wikipedia foot_0 and ambiguities in labels. 3 other fact verification datasets are collected through crowdsourcing (e.g., fever @xcite and its variants @xcite ) by altering a word or negating the reference text to intentionally make true or false claims. this process leads to largescale datasets but with strong artifacts and unrealistic claims @xcite @xcite . consequently, a trivial claim-only baseline with no evidence achieves near 80% @xcite , verified in section 4.1). while more recent work proposes new crowdsourcing methods that alleviate artifacts @xcite , their claims are still written given particular evidence text, being vulnerable to subtle lexical biases that can be hard to explicitly measure. we construct a fact verification dataset from highly ambiguous information-seeking questions. our claims have significantly less lexical bias than other crowdsourced ones (figure 3 ), contain realistic misinformation that people are likely to be confused about ( qa to verification task prior work has also used qa data to create entailment or fact verification benchmarks. most make use of synthetic or annotated questions (demszky et al., 2018; jiang et al., 2020; pan et al., 2021; chen et al., 2021) 4 support: deckard shaw killed han in 'fast and furious tokyo drift'. refute: dominic toretto killed han in 'fast and furious tokyo drift'. ## experiments we first evaluate state-of-the-art fact verification models on faviq in order to establish baseline performance levels (section 4.1). we then conduct experiments on professional fact-checking datasets to measure the improvements from training on faviq (section 4.2). ## conclusion & future work we introduced faviq, a new fact verification dataset derived from ambiguous information-seeking questions. we incorporate facts that real users were unaware of when posing the question, leading to false claims that are more realistic and challenging to identify without fully understanding the context. our extensive analysis shows that our data contains significantly less lexical bias than previous fact checking datasets, and include refute claims that are challenging and realistic. our experiments showed that the state-of-the-art models are far from solving faviq, and models trained on faviq lead to improvements in professional fact checking. altogether, we believe faviq will serve as a challenging benchmark as well as support future progress in professional fact-checking. we suggest future work to improve the faviq model with respect to our analysis of the model prediction in section 4.1.2, such as improving retrieval, modeling multi-hop inference, and better distinctions between entities, events and properties. moreover, future work may investigate using other aspects of information-seeking questions that reflect facts that users are unaware of or easily confused with. for example, one can incorporate false presuppositions in questions that arise when users have limited background knowledge @xcite . as another example, one can explore generating nei claims by leveraging unanswerable information-seeking questions. furthermore, faviq can potentially be a challenging benchmark for the claim correction, a task recently studied by @xcite that requires a model to correct the refute claims.