D an FEVER : claim verification dataset for D anish (2021)

Abstract: We present a dataset, DanFEVER, intended for multilingual misinformation research. The dataset is in Danish and has the same format as the well-known English FEVER dataset. It can be used for testing methods in multilingual settings, as well as for creating models in production for the Danish language.

## introduction the internet is rife with false and misleading information. detection of misinformation and fact checking therefore presents a considerable task, spread over many languages @xcite @xcite . one approach to this task is to break down information content into verifiable claims, which can subsequently be fact-checked by automated systems. automated fact checking can be framed as a machine learning task, where a model is trained to verify a claim. applying machine learning requires training and validation data that is representative of the task and is annotated for the desired behaviour. a model should then attempt to generalise over the labeled data. one dataset supporting automatic verification is the fact extraction and verification dataset (fever) in english @xcite , which supports the fever task @xcite . the dataset is aimed both at claim detection and verification. while the misinformation problem spans both geography and language, much work in the field has focused on english. there have been suggestions on strategies for alleviating the misinformation problem @xcite . it is however evident that multilingual models are essential if automation is to assist in multilingual regions like europe. a possible approach for multilingual verification is to use translation systems for existing methods @xcite , but relevant datasets in more languages are necessary for testing multilingual models' performance within each language, and ideally also for training. this paper presents danfever, a dataset and baseline for the fever task in danish, a language with shortage of resources @xcite . while danfever enables improved automatic verification for danish, an important task @xcite , it is also, to our knowledge, the first non-english dataset on the fever task, and so paves the way for multilingual fact verification systems. danfever is openly available at https: //figshare.com/articles/dataset/ danfever_claim_verification_ dataset_for_danish/14380970 2 english fever the fact extraction and verification dataset and task (fever) is aimed at automatic claim verification in english @xcite . when comparing we will stylize the original fever dataset enfever to avoid confusion. the dataset was created by first sampling sentences from approximately 50,000 popular english wikipedia pages. human annotators were asked to generate sets of claims based on these sentences. claims focus on the same entity as the sentence, but may not be contradictory to or not verifiable by the sentence. a second round of annotators labelled these claims, producing the labels seen in the enfever guidelines state that claims labelled notenoughinfo could possibly be verified using other publicly available information, which was not considered in the annotation. in the fever task @xcite , automatic verification is commonly framed as a two-step process: given a claim, relevant evidence must first be collected, and secondly be assessed as supporting or refuting the claim, or not providing enough information. enfever contains data for training models for both steps. we tasked annotators to create claims for dan-fever based on the same guidelines and without regulation of class-distribution. the classdistribution of danfever is therefore a bit different that that of enfever; there is about the same ratio of supported claims, but more refuted and less notenoughinfo claims in danfever that in enfever. ## method a fever task instance consists of a claim, zero or more pieces of evidence, and a label. the labels take one of the following values: supported claims that can be supported by evidence from the textual data refuted claims that can be refuted by evidence from the textual data notenoughinfo claims that can neither be supported or refuted based on the textual data the claims were created based on data from danish wikipedia and den store danske (a privately-developed, non-profit, online encyclopedia based in denmark and financed through foundations and universities). both sites are generally considered high quality and trustworthy. along with the claims, danfever supplies the wikipedia dump used for creating the claims as well as the content of the articles used from den store danske. the remaining articles from den store danske are not included (due to rights), and all articles should be considered to be iid.for modelling. the format of the dataset can be found in appendix a.1. ## dataset details & analysis danfever consists of 6407 claims. we have included one example from each class in tables 2a , 2b and 2c, and shown the label distribution in ## baseline: recognizing textual entailment the fever task consists of verifying claims based on a text corpus. one common strategy is to split the task into three components (as in the original work @xcite ) to provide baseline performance for future research to benchmark against, we trained a baseline model on the final task; recognizing textual entailment. since there are no evidence extracts for the notverifiable samples, we apply the random-sampling method from the original en-fever paper, where evidence is randomly assigned from the data to each of these samples. we trained classifiers on the resulting 3-class problem. for comparison the multi-layer perceptron and decomposable attention models from the en-fever paper @xcite an f1 score of respectively 73% and 88% on the verification subtask. the comparable performance indicates that pretrained, multilingual, language models are useful for the task, especially considering that danfever is small relative to en-fever. we show the collective test-set confusion matrix of xlm-roberta-large in ## conclusion we have presented a human-annotated dataset, danfever, for claim verification in a new language; danish. danfever can be used for building danish claim verification systems and for researching & building multilingual claim verification systems. to our knowledge danfever is the first non-english fever dataset, and it is openly accessible foot_1 . baseline results are presented over four models for the textual-entailment part of the fever-task.