P ub H ealth T ab: A Public Health Table-based Dataset for Evidence-based Fact Checking (2022)

Abstract: Inspired by human fact checkers, who use different types of evidence (e.g. tables, images, audio) in addition to text, several datasets with tabular evidence data have been released in recent years. Whilst the datasets encourage research on table fact-checking, they rely on information from restricted data sources, such as Wikipedia for creating claims and extracting evidence data, making the fact-checking process different from the real-world process used by fact checkers. In this paper, we introduce PubHealthTab, a table fact-checking dataset based on real world public health claims and noisy evidence tables from sources similar to those used by real fact checkers. We outline our approach for collecting evidence data from various websites and present an in-depth analysis of our dataset. Finally, we evaluate state-of-the-art table representation and pre-trained models fine-tuned on our dataset, achieving an overall F_1 score of 0.73.

## introduction fact-checking is the task of establishing the veracity of factual information, commonly performed manually by journalists. in addition to classifying how truthful claims are, human fact checkers also provide evidence for their judgements. to support this process with computational tools, researchers have compiled several datasets for evidence-based automated fact-checking (afc), which include information about the sources supporting or refuting the claims alongside veracity labels @xcite @xcite @xcite . while a large share of the datasets used in evidence-based afc focus on textual evidence (e.g. @xcite @xcite ), some recent datasets also cover structured data, for instance in the form of web tables @xcite . this is useful, as human fact checkers often need to consider a range of data modalities to verify claims. however, two main limitations remain. first, existing table factchecking datasets consist largely of claims which have been 'artificially' created via online crowdsourcing, starting from randomly selected evidence tables. second, the datasets use single sources of evidence, for instance wikipedia; this is different from how human fact checkers go about the taskmore often than not, they consult multiple primary sources, including websites, databases, and public reports. to overcome these limitations, we propose pub-healthtab foot_0 , a new table fact-checking dataset, using the pubhealth dataset @xcite as a seed. pubhealth has a number of advantages. it contains public health claims that human fact-checkers work on. the authors compared the complexity of these claims to real-world political claims, as well as to claims created by crowdworkers @xcite . as a proxy for complexity, they determined the reading skills needed to understand the claims. they established that public health claims are much more challenging, requiring high school levels of reading of 10 to 12 rather than 6 to 8 for political and crowdsourced claims. pubhealth also includes multiple sources of evidence for the claims, however, the evidence is purely text-based. in our dataset, we include web tables as evidence, extracted from different websites, similar to those used by human fact-checkers. we designed a hybrid dataset pipeline, which takes pubhealth claims and links them, via wikipedia articles, to other websites containing potential evidence tables. we used crowdsourcing in three ways: to establish the relevance of the extracted tables; to adjust pubhealth claims to support or refute the tables; and finally to assess the quality of the new claims. the result is a dataset of 1, 942 claim-table pairs about public health, drawing on evidence from more than 300 websites. we analysed the dataset to spot potential biases in the way we collected the data and compared pubhealthtab with other table-based fact-checking datasets. moreover, we experimented with several bert-based models and table representations to understand how our dataset performs on state-ofthe-art afc, achieving an overall f 1 score of 0.73. both allowed us to identify areas of future improvement, in particular to refute claims against evidence consisting of mostly numerical data or with noisy text headers. ## the pubhealthtab dataset figure 1 shows an overview of the data construction pipeline. in the top half, we automatically create pairs of claims and tables. we start from the pubhealth claims, assess them for relevance and then match the remaining ones with web tables (see section 3.1). in the bottom half, we use crowdsourcing to filter tables, adjust claims to tables, and check for quality (see section 3.1.2). in step 1 we removed ambiguous and out-ofdomain claims from the pubhealth dataset using a lexicon of 4132 medical terms from: wikipedia; foot_2 medical dictionaries from harvard university, foot_3 university of michigan foot_4 , and schulich school of medicine and dentistry foot_5 ; as well as the concept-net knowledge graph. 7 we retained the claims that contained at least one token matching the lexicons. for the other claims, we carried out ner to detect medical entities that the lexicons might have missed, using scispacy @xcite . we kept the claims for which we could find an entity in the claim text whose conceptnet node was liked to a lexicon term via the "hascontext" relation. 8 in step 2 we linked the claims from step 1 to wikipedia articles using two entity linking services: elq @xcite and wat, foot_8 for better coverage. we then took the websites referenced by the articles as a source of evidence tables. in step 3, from all wikipedia references, we kept those in english that could be scraped and which contained at least one table html tag (⟨table⟩). we heuristically removed all tables that were used purely for formatting reasons, and then ranked the remaining tables based on their bm25 similarity to the claim text. the result of this step was a set of 1915 claimtable pairs (1010 claims and 1422 tables from 1196 websites), which was fed to the crowdsourced half of the pipeline. ## dataset analysis we analysed the pubhealthtab dataset for biases and correlations, and compared it to other table fact-checking datasets. we applied three methods: ## experiments and results we experimented with several table representation techniques and state-of-the-art models on pub-healthtab to understand related challenges. ## conclusion we introduced pubhealthtab, a table-based dataset for evidence-based fact checking centred on realworld public health claims. our dataset comprises 1, 942 claim-table pairs, with tabular evidence data extracted from websites similar to those used by fact checkers. we described the dataset creation process and the steps taken to minimise biases and correlations. we evaluated state-of-the-art representation and modelling approaches and showed that the roberta model achieves the highest performance on pubhealthtab across all representation methods compared to other models. in contrast to previous table-based fact-checking datasets that contain tables from single data sources, stateof-the-art models struggle to correctly classify refute claims from pubhealthtab against evidence consisting of mostly numerical data or with noisy text headers, making pubhealthtab a challenging dataset for table-based fact-checking research. 13 figure 6: author-annotated crowdsourcing example.