CHEF : A Pilot C hinese Dataset for Evidence-Based Fact-Checking (2022)

Abstract: The explosion of misinformation spreading in the media ecosystem urges for automated fact-checking. While misinformation spans both geographic and linguistic boundaries, most work in the field has focused on English. Datasets and tools available in other languages, such as Chinese, are limited. In order to bridge this gap, we construct CHEF, the first CHinese Evidence-based Fact-checking dataset of 10K real-world claims. The dataset covers multiple domains, ranging from politics to public health, and provides annotated evidence retrieved from the Internet. Further, we develop established baselines and a novel approach that is able to model the evidence retrieval as a latent variable, allowing jointly training with the veracity prediction model in an end-to-end fashion. Extensive experiments show that CHEF will provide a challenging testbed for the development of fact-checking systems designed to retrieve and reason over non-English claims.

## introduction misinformation is being spread online at increasing rates, posing a challenge to media platforms from newswire to social media. in order to combat the proliferation of misinformation, fact-checking is an essential task that assesses the veracity of a given claim based on evidence @xcite . fact-checking is commonly conducted by journalists. however, fact-checking is a time-consuming task, which can take journalists several hours or days @xcite . thus, there is a need for automating the process. although misinformation spans both geographic and linguistic boundaries, most existing works focused on english @xcite @xcite @xcite claim: 2019年, 共有12.08万人参加成都中考，但招 生计划只有4.3万。in 2019, a total of 120,800 students participated in the high school entrance examination in chengdu, but schools only enrolled 43,000 students. document: 今年共有12.08万人参加中考，这个是 成都全市, 包括了20个区，高新区和天府新区的总 参考人数。 月前，教育局公布了2019年的普高招 生计划。招生计划数进一步增加，上普高的机会更 大了... 中心城区(13个区)招生计划为43015人。 this year, 120,800 people participated in the high school entrance examination. this number is for the entire city of chengdu, including 20 districts, high-tech zone and tianfu new district. a month ago, the education bureau announced the 2019 high school enrollment plan. the number of enrollment will be increased, indicating that there is a greater chance of going to high school... the plan of the central area (including 13 districts) is 43,015. verdict: refuted; domain: society challenges: evidence collection; numerical reasoning to bridge this gap, we introduce a chinese dataset for evidence-based fact-checking (chef). chef includes claims that are not only relevant to the chinese world, but also originally made in chinese. it consists of 10,000 real-world claims, collected from 6 chinese fact-checking websites covered multiple domains and paired with annotated evidence. to ensure annotation consistency, dataset natural domain #claims language evidence type source retrieved annotated fever (thorne et al., 2018) ✗ multiple 185,445 english text wiki ✓ ✓ hover (jiang et al., 2020) ✗ multiple 26,171 english text wiki ✓ ✓ tabfact (chen et al., 2020) ✗ multiple 92,283 english table wiki ✗ ✓ infotabs (gupta et al., 2020) ✗ multiple 23,738 english table wiki ✗ ✓ ant (khouja, 2020) ✗ multiple 4,547 arabic ✗ ✗ ✗ ✗ vitaminc (schuster et al., 2021) ✗ multiple 488,904 english text wiki ✗ ✓ danfever (nørregaard and derczynski, 2021) ✗ multiple 6,407 danish text wiki ✓ ✓ feverous (aly et al., 2021) ✗ multiple 87,026 english text/table wiki ✓ ✓ politifact (vlachos and riedel, 2014) ✓ politics 106 english meta/text fc ✗ ✗ punditfact (rashkin et al., 2017) ✓ multiple 4,361 english ✗ ✗ ✗ ✗ liar (wang, 2017) ✓ multiple 12,836 english meta fc ✗ ✗ verify (baly et al., 2018) ✓ politics 422 mul(2) text internet ✓ ✗ multifc (augenstein et al., 2019) ✓ multiple 36,534 english meta/text internet ✓ ✗ snopes (hanselowski et al., 2019) ✓ multiple 6,422 english text fc ✗ ✗ scifact (wadden et al., 2020) ✓ science 1,409 english text paper ✗ ✗ pubhealth (kotonya and toni, 2020b) ✓ health 11,832 english text fc ✗ ✗ answerfact (zhang et al., 2020) ✓ product 60,864 english meta/text amazon ✓ ✗ fakecovid (shahi and nandini, 2020) ✓ health 5,182 mul(3) ✗ ✗ ✗ ✗ xfact (gupta and srikumar, 2021) ✓ multiple 31,189 mul(25) meta/text internet ✓ ✗ chef ✓ multiple 10,000 chinese meta/text internet ✓ ✓ ## background: dataset comparisons in this section, we reviewed the existing factchecking dataset as summarized in ## dataset construction chef is constructed in four stages: data collection, claim labeling, evidence retrieval and data validation. data collection selects sources, crawls claims and associated metadata. claim labeling identifies claims from fact-checking articles and website domain url total piyao multiple www.piyao.org.cn 3,741 tfc multiple tfc-taiwan.org.tw 1,759 mygopen multiple www.mygopen.com 1,654 jiaozhen multiple vp.fact.qq.com 157 cnews multiple m.chinanews.com 2,689 total multiple -10,000 ## baseline systems unlike previous natural datasets, chef requires the system to first retrieve the evidence sentences from the documents, then predict the veracity based on the evidence. therefore, we design two types of baselines: pipeline and joint systems. ## conclusion we constructed the first chinese dataset for evidence-based fact-checking. further, we have discussed the annotation methods and shared some of the insights obtained that will be useful to other non-english annotation efforts. to evaluate the challenge chef presents, we have developed established baselines and conducted extensive experiments. we show that the task is challenging yet feasible. we believe that chef will provide a stimulating challenge for automatic fact-checking.