The Fact Extraction and VER ification Over Unstructured and Structured information ( FEVEROUS ) Shared Task (2021)

Abstract: The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are Supported or Refuted based on evidence retrieved from Wikipedia (or NotEnoughInfo if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018, FEVEROUS requires partial evidence to be returned for NotEnoughInfo claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.

## introduction automated fact verification has become an important field of research, as fact-checkers and journalists are facing an even-increasing volume of claims to verify @xcite . this task has been explored by the nlp community through forums and shared tasks such as clef checkthat! @xcite , @xcite and fever @xcite , as well as a number of datasets aimed at modelling parts of the task @xcite @xcite @xcite . while these previous works focus on claims that are verified against a single type of evidence, such as text or structured information, the new fever-ous dataset @xcite we study in this shared task requires the models to reason about both types of evidence. this helps better approximate real-world fact checking, where both the claims and the sources of evidence are more complex in nature. feverous models the task of fact extraction and verification over unstructured and structured information, overcoming some limitations of the previous fever dataset @xcite , which only considers text as evidence, and improving the quality of the annotations as well as removing known biases from the dataset. feverous contains 87,026 new claims which are more complex (25.3 word/claim on average compared to 9.4 for fever), and a larger pool of evidence (tables, lists, and sentences from the entirety of wikipedia), bringing us closer to real-world scenarios, while maintaining the experimental control of an artificially designed dataset. this paper presents a short description of the task and dataset, the final test phase leaderboard, and a summary of the submissions with a comparison to previous fever shared tasks, an analysis of current challenges and a discussion around interesting research directions for this task. the shared task received 13 entries in total, with the winning team, "bust a move!", achieving a score of 27%, 9 percentage points higher than the baseline system we released. while considerable progress was made by the participants of the task, there are still plenty of opportunities for systems to improve. we will leave the scoring system open to allow future work to build upon the advances made in this shared task. figure 1 : feverous sample instances. evidence in tables is highlighted in red. each piece of evidence e i has associated context, i.e. page, section title(s) and the closest row/column headers (highlighted in dark gray). left: evidence consists of two table cells refuting the claim. right: evidence consists of two table cells and one sentence from two different pages, supporting the claim. ## task description given a human-authored claim, systems had to first retrieve evidence from wikipedia in the form of sentences and table cells, each accompanied by the page/section titles and column headers they were found under respectively. they then had to classify whether the claim is supported or refuted based on the evidence, or notenoughinfo if the claim cannot be verified. system responses would be scored both on the evidence retrieval and the label classification. note that unlike in the original fever shared task, (partial) evidence needs to be provided for the notenoughinfo claims. each claim in the feverous dataset could have multiple ways of being verified, which is represented in the different evidence sets -each with potentially multiple pieces of evidence. the participating systems only had to provide one complete evidence set for their response to be considered correct. ## results the results for all submissions to the shared task are in page retrieval page retrieval was mostly kept simple relying on term-matching for efficiency bust a move!, ncu and metuis used bm25 (the latter two using the implementation of anserini @xcite ), while martin funkquist used vanilla tf-idf matching. papelo and albatross, following the baseline, combined tf-idf matching with entity matching, and eurecom_fever reranked its results with a bert model pre-trained on ms-marco @xcite . overall, focusing on the entities in the claim for page retrieval was found to be beneficial by the participants. ## analysis the feverous baseline system is stronger than the one proposed for fever relatively to the respective submitted systems, achieving a higher score than more than half of participating teams, while the fever baseline only performed better than around a fifth of all systems. generally, feverous scores are generally much lower than fever scores, with retrieval scores being particularly low for feverous. while this is likely partly due to the neis being easier to predict and more numerous in fever (by only predicting nei, a system would already get a score of 0.33), it might also be a result of artefacts in the original fever dataset, as a claim-only baseline was able to get an accuracy score of about 62% @xcite , compared to a majority-class baseline of 33%. in contrast, the claim-only baseline on feverous achieves a score of 58% against a majority-class baseline of 56%. yet, peculiarities of the feverous dataset observed by participants is a higher number of redundant evidence pieces than in fever @xcite (likely a result of the higher complexity of evidence annotation), as well as a considerable number of claims that are refuted/nei due to a single piece of information being incorrect/missing in an otherwise supported claim @xcite . since claims are much longer in feverous than fever, such cases are much harder to identify. similar to the fever 2.0 challenge @xcite where in a build-it, break-it, fix-it competition teams created adversarial attacks (breakers) against systems that were trained on the fever dataset (builders), to identify biases and weaknesses and address them (fixers), such a challenge might provide highly valuable insights to the feverous dataset to foster further research on this task.