P ars FEVER : a Dataset for F arsi Fact Extraction and Verification (2021)

Abstract: Training and evaluation of automatic fact extraction and verification techniques require large amounts of annotated data which might not be available for low-resource languages. This paper presents ParsFEVER: the first publicly available Farsi dataset for fact extraction and verification. We adopt the construction procedure of the standard English dataset for the task, i.e., FEVER, and improve it for the case of low-resource languages. Specifically, claims are extracted from sentences that are carefully selected to be more informative. The dataset comprises nearly 23K manually-annotated claims. Over 65% of the claims in ParsFEVER are many-hop (require evidence from multiple sources), making the dataset a challenging benchmark (only 13% of the claims in FEVER are many-hop). Also, despite having a smaller training set (around one-ninth of that in Fever), a model trained on ParsFEVER attains similar downstream performance, indicating the quality of the dataset. We release the dataset and the annotation guidelines at https://github.com/Zarharan/ParsFEVER .

## introduction the spread of false information can lead to severe social and political problems @xcite . it would be extremely difficult to detect and track false information manually, given that the abundance of available technology has made it possible for these to be produced at scale and disseminated rapidly. therefore, there has been a lot of interest in developing natural language technologies for fact-checking @xcite . unfortunately, similarly to many other fields of nlp that rely on manually curated datasets, fact-checking has remained restricted to a few high-resource languages for which large-scale annotated datasets are available. in this paper, we present parsfever, the first farsi fact extraction and verification dataset. the dataset opens room for research in fact-checking and verification on low-resourced languages. pars-fever is constructed based on fever @xcite , the most widely used dataset for factchecking and fake news detection in english. we collected 22,906 claims by altering sentences extracted from introductory sections of 358 popular articles from farsi wikipedia. annotators manually classified these claims into supported, refuted, or notenoughinfo based on the provided reference pages. in addition, the annotators tagged those sentences which they used as evidence for this classification. therefore, the dataset can be used for both fact-checking (a 3-class classification task) and evidence retrieval (which is a necessary step for the classification). the quality of the dataset was evaluated using three different validation checks: (1) 5-way interannotator agreement, (2) agreement against superannotators foot_0 , and (3) manual validation by the authors. we also report experimental results for when parsfever was used as a benchmark for the factchecking task. in this task, given an input claim the model is expected to support or refute it and provide the corresponding evidence for this decision. if no enough evidence is found, notenoughinfo is returned. we evaluated the baseline system provided for fever on our dataset. the results indicate the more challenging nature of parsfever: 50.0% (vs. 52.1% in fever) accuracy on a heldout test set on claim classification, and 28.1% (vs. 32.6% in fever) for evidence retrieval. finally, we release parsfever and related tools to allow further research on low-resource fact-checking, particularly in farsi. ## related work the only related datasets in farsi are those of @xcite and @xcite . the former is a dataset for farsi stance detection containing hundreds of instances in the news domain. unlike parsfever, the dataset does not provide any evidence for the claims; hence, it can only be used for a constrained fact-checking evaluation setting where evidences are already extracted for verifying stance. also, the dataset of @xcite is targeted towards rumor detection in farsi tweets, which mostly relies on twitterspecific features such as user profile information and response/retweet structure. in contrast, our dataset mostly focuses on lexical features. parsfever is mainly based on fever, a dataset widely used for fact extraction and verification in english. the dataset consists of around 185k claims generated by modifying sentences extracted from wikipedia. the claims are classified as supported, refuted, and notenoghinfo. despite being based on fever, our dataset has some fundamental differences that aim at making a more challenging benchmark for low-resourced languages. in the following section, we elaborate on the construction procedure of our dataset and the differences it has to that used for fever. other related datasets include hover @xcite and liar @xcite . hover is a dataset for many-hop fact extraction and claim verification. unlike our dataset, which consists of single sentence claims, hover includes claims from one sentence up to one paragraph. it consists of 26k claims with supported or notsupported labels. liar was instead derived from the short statements extracted from politifact.com for fake news detection. this dataset contains 12.8k human-labeled instances. other related datasets in the social media domain include pheme @xcite and ru-moureval @xcite . pheme consists of 5,802 comment threads collected from twitter, with approximately 103k tweets. this dataset has 1,972 and 3,830 threads labeled as rumour and nonrumour, respectively, resulting in an imbalanced dataset. rumoureval was released as part of the semeval-2017 task 8 @xcite . the dataset contains 330 rumour threads (4,842 tweets) from twitter, annotated for both stance and veracity. ## experiments following @xcite , we implemented a full pipeline system for fact verification and extraction with the following three modules: 1. a document retrieval component @xcite to find the most relevant page to a specific claim. split sup ref nei training 6,253 4,008 5,685 dev 841 824 861 test 853 833 863 total 7,947 5,665 7,409 these models are based on mlp @xcite , with a single hidden layer that benefits term frequencies and tf-idf cosine similarity between the claim and evidence, and decomposable attention @xcite . given that note-noughinfo instances are not associated with any evidence, they cannot be used for training the rte models. to address this issue, @xcite proposed two alternatives solutions: sampling a sentence (as evidence) from the nearest page to the claim (np) or using the document retrieval component to uniformly select a random sentence (as evidence) from wikipedia (rs). ## conclusion we presented parsfever, a novel and publicly available dataset for farsi fact extraction and verification. we elaborated the construction procedure for this dataset, which focuses on having a rich dataset suitable for low-resource languages. although this work uses wikipedia as its source, other textual structures and corpora can also be used for fact extraction in this framework. we evaluated the baseline system proposed for fever on our dataset. however, there have been recent developments in the field of fact-checking with models such as qabriefs @xcite ). an immediate future work would be to take pars-fever as a more challenging benchmark (than fever) with significant many-hop operations as a benchmark for evaluating and analyzing existing fact-checking models. this analysis can also shed light on the ability of these models to go beyond the english languages and in low-resource settings.