H er O at AV eri T e C : The Herd of Open Large Language Models for Verifying Real-World Claims (2024)

Abstract: To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Her d of O pen LLMs for verifying real-world claims ( HerO ). HerO employs multiple LLMs for each step of automated fact-checking. For evidence retrieval, a language model is used to enhance a query by generating hypothetical documents that check the veracity of a claim. We fine-tune LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at https://github.com/ssu-humane/HerO .

## introduction automated fact-checking is a task that predicts a claim's veracity by referring to pieces of evidence @xcite . claim verification requires the retrieval of relevant information from a reliable document collection and the decision on whether the claim is supported by the known relevant information. early research attempted to automate the fact-checking process by generating synthetic claims based on wikipedia documents @xcite or collecting manually verified claims by human experts @xcite . however, most datasets suffer from critical issues such as context dependence, evidence insufficiency, and temporal leaks; these limitations made the resulting systems less applicable to the verification of real-world claims. in light of this, a recent study proposed a dataset called averitec @xcite . they addressed the limitations by conducting fine-grained crowdsourced annotations for the fact-checking process. this paper describes our system for the averitec shared task hosted by the fever-24 workshop @xcite . motivated by the recent advancements in large language models, we introduce a fact-checking system that utilizes llms for each step of evidence-based fact verification: evidence retrieval, question generation, and veracity prediction. our system, the herd of open llms for verifying real-world claims (hero), employs publicly available llms without using proprietary llms, to ensure the transparency of the system. hero achieved 2nd place in the shared task with an averitec score of 0.57. given that the winning system used gpt-4o @xcite , hero's competitive performance imply the potential of open llms for verifying real-world claims. ## related work llms have achieved remarkable success in natural language understanding and generation @xcite @xcite . while major tech companies primarily drove the initial success, they only provided limited access to the model through an api. on the other hand, some research groups have attempted to develop open llms to facilitate open research. while the performance of the initial models was unsatisfactory @xcite , recent models are on par with closed models and even outperform them in certain categories @xcite . ## task definition the averitec shared task aims to develop a factchecking system that verifies real-world claims by retrieving evidence from the web. to verify a given claim, the system first needs to retrieve relevant information from the web documents (evidence retrieval). for each of the collected evidence, the system evidence retrieval question generation reranking veracity prediction query model baseline claim bm25 bloom-7b bert-base bert-base hero hyde-fc (llama-3.1-70b) bm25 w/ sfr-embedding-2 llama-3-8b -llama-3.1-70b ## evaluation experiments in this section, we present experimental results to decide the system configuration. ## conclusion to tackle the averitec shared task hosted by the fever-24, we developed hero, a fact-checking system that employs publicly available large language models for each step of automated factchecking: evidence retrieval, question generation, and veracity prediction. our system achieved 2nd place in the shared task, supporting the effectiveness of open llms for verifying real-world claims. we release our code publicly for future research.