The Automated Verification of Textual Claims ( AV eri T e C ) Shared Task (2024)

Abstract: The Automated Verification of Textual Claims (AVeriTeC) shared task asks participants to retrieve evidence and predict veracity for real-world claims checked by fact-checkers. Evidence can be found either via a search engine, or via a knowledge store provided by the organisers. Submissions are evaluated using the AVeriTeC score, which considers a claim to be accurately verified if and only if both the verdict is correct and retrieved evidence is considered to meet a certain quality threshold. The shared task received 21 submissions, 18 of which surpassed our baseline. The winning team was TUDA_MAI with an AVeriTeC score of 63%. In this paper we describe the shared task, present the full results, and highlight key takeaways from the shared task.

## introduction automated fact-checking (afc) has been proposed as an assistive tool for beleaguered factcheckers @xcite , whose work is crucial for limiting misinformation @xcite . this has inspired applications in journalism @xcite @xcite and other domains, e.g. science @xcite . substantial progress has been made on common benchmarks, such as fever @xcite and @xcite . nevertheless, existing resources have recently come under criticism. many datasets (for example, @xcite ; @xcite ; @xcite ) contain purpose-made claims derived e.g. from wikipedia, and are thus not representative of real-world use cases. datasets that do contain real-world claims either lack evidence annotation @xcite , or suffer issues resulting from superficial automated evidence annotation @xcite . claim: the usa has succeeded in reducing greenhouse emissions in previous years. date: 2020.11.2 speaker: morgan griffith verdict: conflicting evidence/cherrypicking. q1: what were the total gross u.s. greenhouse gas emissions in 2007? a1: in 2007, total gross u.s. greenhouse gas emissions were 7,371 mmt. q2: when did greenhouse gas emissions drop in us? a2: in 2017, total gross u.s. greenhouse gas emissions were 6,472.3 mmt, or million metric tons, carbon dioxide. q3: did the total gross u.s. greenhouse gas emissions rise after 2017? a3: yes. after 3 years of decline, us co2 emissions rose sharply last year. based on preliminary power generation, natural gas, and oil consumption data, we estimate emissions increased by 3.4% in 2018. figure 1 : example instance from averitec. given a claim and associated metadata, participating systems must first retrieve appropriate evidence. then, they must output a verdict for the claim given that evidence. the averitec dataset was constructed to overcome these limitations @xcite . averitec combines real-world claims with evidence from the web. the process of evidence retrieval is broken down into question generation and answering, providing a structured representation of the evidential reasoning process. the annotation process for averitec was designed to ensure (1) that claims are understandable independently of the fact-checking articles they were sourced from, (2) that the evidence given is sufficient to support the verdicts, and (3) that all evidence used would have been available on the web before the claim was made. this avoids common problems found in previous datasets @xcite . averitec consists originally of 4,568 examples, collected from 50 fact-checking organizations using the google factcheck claim search api foot_0 ; itself based on claimreview foot_1 . to ensure that systems are evaluated on unseen data, we expanded the (hidden) test set with a further 1,215 claims for the shared task, bringing the total dataset size to 5,783. we furthermore released a "knowledge store" containing, for each claim in the training, development, and test splits, documents which can be used as evidence for that claim. this was done to prevent participants from being limited by the prohibitive cost of the search api we used for evidence retrieval in the original paper @xcite . we also developed an updated version of the baseline for the shared task, which uses the knowledge store. participants in the shared task were allowed to use evidence from the knowledge store, use a search engine on their own, or combine the two options. our dataset and baseline are available under a cc-by-nc-4.0 license at https://fever.ai/dataset/averitec.html . this paper presents a description of the task and dataset, the final test phase leaderboard. we also summarise the submitted system description papers, drawing out commonalities, differences, and lessons. we furthermore carry out additional analysis of the shared task results, including human evaluation. finally, we reflect on the task, deriving lessons for future work -and further shared tasks -on automated fact-checking. the shared task received 21 submissions. the winning team, tuda_mai, achieved a score of 63%, a very significant improvement on the 11% achieved by the baseline system. nevertheless, there are still plenty of opportunities for further improvement. during the process, we identified an issue with the evidence set provided for participants, which for some claims in the second half of the dataset contained fact-checking articles written by humans about those claims. we release an updated knowledge store at https://fever.ai/dataset/averitec. html , where these articles have been removed. we leave open an evaluation page corresponding to the new knowledge store foot_2 so that future work can build upon the advances made in this shared task. participants are given claims and associated metadata, such as the publication date (see figure 1 ). based on this, they must retrieve evidence for or against the claims. in the gold annotation, this evidence is broken down into question-answer pairs, naturally enabling multi-hop reasoning. we do not restrict participants to providing evidence in this format, although given the meteor-based evaluation setup most participants found it beneficial to follow it. when submitting test set predictions, we also required participants to include a url to an external website for each piece of evidence, corresponding to a webpage providing backing. finally, based on the evidence, participants must predict whether a veracity label from the set supported, refuted, not enough evidence, or conflicting evidence/cherrypicking. unlike the original averitec dataset, we did not require participants to submit a justification for the verdict. ## results the overall results for the shared task can be seen in one issue identified by several participants was the scraper we used for the knowledge store, based on trafilatura (barbaresi, 2021). papelo identified how, in 297 out of 500 development examples, at least one gold document was not correctly scraped. dunamu-ml similarly discussed how the scraper did not correctly handle evidence from pdfs and videos. in their submission, dunamu-ml extended the scraper to extract text and transcripts from pdfs and youtube videos, and noted that this helped performance. when constructing averitec, our annotators filtered out claims requiring multimodal reasoning; all claims in the dataset are textual and can be verified through exclusively textual evidence. nevertheless, the helpfulness of video transcripts suggests that multimodal evidence can be useful even for that scenario. question generation & retrieval most systems employed an llm-based question generation strategy. that is, they generated questions or queries, and then retrieved evidence based on those questions. generating questions, rather than simply searching for the claim, was noted by many topscoring systems to be essential for good retrieval performance. this supports our hypothesis from @xcite that question generation (or query expansion @xcite ) is a key avenue for further gains in retrieval. team name evidence qg retrieval qa veracity tuda_mai ks gpt-4o gte_base_en_v1.5 gpt-4o gpt-4o humane ks llama-3-8b bm25 sfr-embedding-2 llama-3.1-70b -llama-3.1-70b ctu aic ks gpt-4o mxbai-large-v1 gpt-4o gpt-4o dunamu-ml ks gpt-4 bm25 gpt-4 gpt-4 papelo google t5-large gpt-4o -gpt-4o gpt-4o uhh ks gpt-4o-mini bm25 gte_base_en_v1.5 gpt-4o-mini mixtral-8x7b synapse google gpt-4o all-minilm-l6-v2 gpt-4o gpt-4o gpt-3.5 mistral-7b aioriaveri ks gpt-4o stella_en_400m_v5 gpt-4o gpt-4o data-wizards ks phi-3-medium stella_en_1.5b_v5 mixtral-8x22b mixtral-8x22b ma-bros-h ks llama-3-70b bm25 llama-3-70b llama-3-70b sk_du ks gpt-4o bm25 ms-marco-minilm-l-12-v2 -deberta-v3-base ups ks t5-large bm25 bert -bert fzi-wim ks llama-3-70b ms-marco-minilm-l-12-v2 llama-3-70b bart-large-mnli llama-3-70b knowcomp google llama-3-8b -llama-3-8b llama-3-8b ikr3-unimib ks -bm25 colbert gpt-3.5 bert question generation was typically implemented using large-scale llms, such as gpt-4o or llama-3.1-70b. some systems based on smaller model -humane with llama-3-8b, uhh with gpt-4o-mini, data-wizards with phi-3-medium, and papelo with t5 (for the first question only) -also achieved a high question-only score. this suggests that smaller models can be competitive on search query generation. several teams -papelo, synapse, and ikr3mentioned that they saw benefits from modeling the retrieval task as multi-hop retrieval. that is, instead of retrieving all documents at once, their systems used multiple rounds of retrieval with each round conditional on previous rounds. the benefits of this strategy were also documented in previous fever shared tasks, e.g., @xcite . team papelo further expanded on this strategy, showing that the use of different components at different retrieval steps -t5 for the first question and gpt-4o for subsequent questions -yielded higher performance than using a single-question generation model. as can be seen in we tested this, and observed baseline performance increase by 2 points qa score and 0.5 points averitec score when including two additional duplicates of each question. there are two reasons this might happen. first, some generated qa pairs may be the best match for multiple gold qa pairs (i.e. because they are very long, or because other qa pairs are irrelevant to the claim). duplicating qa pairs means the generated pair can be matched to multiple gold pairs when computing the hungarian algorithm, marginally increasing overall performance. second, hungarian meteor is computed by averaging over gold question-answer pairs. if there are more gold pairs than generated pairs, some gold pairs will be unmatched. these will receive a score of 0, as the "matched" evidence is the empty string, dragging down the average. effectively, systems are heavily penalised for generating too few questions, and may benefit slightly from generating too many. for evidence retrieval, vector-based dense retrieval systems @xcite were common, along with bm25 @xcite . several teams -humane, uhh, sk_du -proposed hybrid systems where coarse retrieval with bm25 was followed by reranking with a vector-based approach. for vector-based retrievers, the gte @xcite family of models were popular, and noted by participants to perform well on the task; this includes stella 6 , an mrl @xcite approach based on gte. several teams noted that their gte-or stella-based retrievers were chosen as they, at the time of the competition, were top performers on the mteb @xcite leaderboard. the overall best performing retrieval system was dunamu-ml, closely followed by humane. in in we also compute results separated by gold verdict (@xmath0, @xmath1, @xmath2, ce/@xmath3. finally, we report the average number of evidence documents submitted per claim. we note that if a team submitted more than 10 documents for a claim, only the first 10 were used to compute retrieval scores for evaluation. veracity prediction veracity prediction was also dominated by llm-based approaches, including gpt-4o, llama 3.1, and mixtral. teams humane and synapse note that some fine-tuning was necessary for good performance on veracity prediction. various teams saw improvements both from full fine-tuning of all the weights, and from fine-tuning with lora @xcite . interestingly, one team -papelo -chose to prevent their veracity prediction system from predicting not enough evidence and conflicting evidence, arguing that their prompting-based model too frequently chose these rarer labels. this may explain why calibration was especially helpful for this task. we note that top-scoring systems tended to use very large models for veracity prediction, such as gpt-4o, llama-3.1-70b, or mixtral-8x7b. the superior reasoning capabilities of these cutting-edge models appear especially critical to this stage of the pipeline, unlike for question generation. types & verdicts in systems performed slightly better on quote verification, slightly worse on position statements, and approximately equally well on other claims. this is interesting, as quote verification and position statements are relatively similar tasks. in the former, systems must verify if a person has uttered a quote verbatim; in the latter, systems must verify if a person or organisation holds a specific position (e.g., supporting a policy), but not necessarily verbatim. verifying position statements often required abductive reasoning, which llms are known to struggle with @xcite . among the top performing systems, performance is frequently lower on numerical statements (along with position statements) compared to other claims. this suggests that the gap is smaller for numerical reasoning than other forms of reasoning. as top performers often use very large llms, that is suggestive of the type of reasoning gains accomplished by scaling up these models. in terms of performance across the different labels, there is significant variation. first, systems often have different calibration to predict supported versus refuted claims. as refuted claims dominate (making up approximately two-thirds of the dataset), this yields a significant advantage for some participants. we note that a common strategy among participants was to ignore the rarer veracity labels -not enough evidence, and conflicting evidence. as mentioned e.g. by team papelo in their system description paper, large language models tend to overpredict these rarer classes. nevertheless, many top performers, including the winning system, made significant gains on these classes. ## human evaluation of evidence following the approach taken in the first fever shared task @xcite , we conducted human evaluation of the evidence retrieved by the systems participating in the shared task, motivated by two concerns. first, the incompleteness of the gold evidence annotation, since it is often the case that adequate evidence to determine the verdict for a claim can be found in multiple webpages, as shown in the inter-annotation agreement study of @xcite . second, the inaccuracies of automatic evaluation metrics of textual evaluation, especially in the case of token-matching metrics such as meteor (banerjee and lavie, 2005) used here, but also of more recent neural ones such as factscore @xcite . thus we can gain a deeper understanding of the quality of the retrieved evidence, and assess how well the averitec scores assigned to the retrieved evidence aligns with human judgements. ## lessons learned providing a knowledge store rather than requiring participants to rely on a search engine api made the task more accessible. given the cost of api access, this allowed substantial analysis and work by participants on retrieval. we note that most submissions -13 of 16 system description papers -used the knowledge store. nevertheless, because of the size of the knowledge store and the inclusion of distractor documents, the knowledge store did not trivialise the task, and systems relying on search remain competitive and provide unique advantages. several participants, such as team fzi-wim, commented on how the two are complementary, and suggested hybrid systems using both as a potentially fruitful extension of their systems. averitec presupposes a strong focus on evidence retrieval. the overall score, as in fever @xcite , is determined both by retrieval performance and by veracity prediction performance. in the averitec shared task, participant systems innovated across the pipeline, and all of the top-scoring systems suggest improvements to multiple subtasks of fact-checking. when submitting test set predictions, we required participants to include a field ("scraped_text") for each piece of evidence in their submission, corresponding to the webpage providing backing for that piece of evidence. this enabled us to carry out manual and automatic quality control evaluation verifying that systems do indeed ground their evidence in external sources (see section 3). this enabled us to detect, for example, if some systems were hallucinating evidence; we did not see any evidence of hallucinated evidence, but we consider guardrails against this crucial. unfortunately, the inclusion of this field made some submissions substantial in size, as entire webpages were included -up to 2.3gb for the largest submission. our submission portal, eval.ai, was not able to handle these large files, blocking the portal for all participants during the last few days of the competition. we extended the deadline to compensate. the scraper we used for the knowledge store (same as in @xcite ) to retrieve evidence turned out to be a significant weakness. as some participants noticed, many knowledge store documents are empty. the submission with the best retrieval performance, dunamu-ml, used a custom scraper, and may have derived significant gains from that choice. we suggest that this may be an interesting area for further research. during the competition, we identified an issue with the knowledge store data for the last 1215 test examples. due to an error with date formats, for some claims, web pages published after the claim were included in the knowledge store. this included fact-checking articles, as also mentioned by ctu aic in their system description paper. as the first 1000 examples were not affected, we computed performance on the first 1000 and last 1215 test examples separately -see as can be seen, the ranking of participants on the two splits is roughly the same -and, indeed, roughly the same as for the entire test set. the second half was easier, and many systems perform slightly better there. somewhat surprisingly, some systems which relied on google search -specifically, synapse -also saw a performance gain when measured only on the second split. as such, we do not believe this issue majorly impacted any subset of participants, such as those not relying on the knowledge store. we release an updated knowledge store along with our shared task paper, accessible at https://fever.ai/dataset/ averitec.html . we have re-compiled the knowledge store with the correct date cutoff, and removed any fact-checking articles that snuck through from the evidence base. ## conclusions & future work the averitec shared task attracted submissions from 21 teams, 18 of which outperformed our baseline. the leaderboard was dominated by systems relying on large language models, especially gpt-4o; nevertheless, especially for question generation and retrieval, smaller models -such as llama-3-8b -also achieved top performance. the winner of the shared task was team tuda_mai, which achieved an averitec-score of 63%. in this paper we have analysed the shared task, highlighting aspects of the 16 submitted system description papers, as well as key takeaways from the shared task itself. the strong performance of the participating teams establishes a firm foundation for automating aspects of real-world fact-checking. the results furthermore indicate clear directions for future work. first, most participating systems -especially for veracity prediction -relied on very large models, such as gpt-4. further, many of these are blackbox models. these models may be prohibitively expensive for some real-world use cases, e.g., assisting smaller fact-checking organisations @xcite . given that, we suggest that getting smaller, more efficient models to reach the performance of their larger counterparts may be a fruitful direction for further research. similarly, we note that performance for most top-scoring systems was much higher on supported and refuted claims, compared to conflicting evidence and not enough evidence. we suggest that leveling this gap is another clear avenue for future improvements.