Document-level Claim Extraction and Decontextualisation for Fact-Checking (2024)

Abstract: Selecting which claims to check is a time-consuming task for human fact-checkers, especially from documents consisting of multiple sentences and containing multiple claims. However, existing claim extraction approaches focus more on identifying and extracting claims from individual sentences, e.g., identifying whether a sentence contains a claim or the exact boundaries of the claim within a sentence. In this paper, we propose a method for document-level claim extraction for fact-checking, which aims to extract check-worthy claims from documents and decontextualise them so that they can be understood out of context. Specifically, we first recast claim extraction as extractive summarization in order to identify central sentences from documents, then rewrite them to include necessary context from the originating document through sentence decontextualisation. Evaluation with both automatic metrics and a fact-checking professional shows that our method is able to extract check-worthy claims from documents at a higher rate than previous work, while also improving evidence retrieval.

## introduction human fact-checkers typically select a claim in the beginning of their day to work on for the rest of it. claim extraction (ce) is an important part of their work, as the overwhelming volume of claims in circulation means the choice of what to fact-check greatly affects the fact-checkers' impact @xcite . automated approaches to this task have been proposed to assist them in selecting check-worthy claims, i.e., claims that the public has an interest in knowing the truth @xcite . existing ce methods mainly focus on detecting whether a sentence contains a claim @xcite or the boundaries of the claim within a sentence @xcite . in real-world scenarios though, claims often need to be extracted from documents consisting of multiple sentences and containing multiple claims, not all of which are relevant to the central idea of the document, and verifying all claims manually or even automatically would be inefficient. moving from sentence-level ce to documentlevel ce is challenging; we illustrate this with the example in figure 1 . sentences in orange are claims selected by a popular sentence-level ce method, claimbuster @xcite , that are worth checking in principle but do not always relate to the central idea of the document, and multiple sentences with similar claims are selected, which would not all need to be fact-checked (e.g., sentences 1 and 6). claims extracted for fact-checking are expected to be unambiguous @xcite , which means that they cannot be misinterpreted or misunderstood when they are considered outside the context of the document they were extracted from, consequently allowing them to be fact-checked more easily @xcite . figure 1 shows an example of claim decontextualisation, where the claim "bird is scrapping thousands of e-scooters in the middle east ...... " requires coreference resolution to be understood out of context, e.g., "bird" refers to "california scooter sharing start-up bird". however, existing ce methods primarily focus on extracting sentence-level claims (i.e., extracting sentences that contain a claim) from the original document @xcite and ignore their decontextualisation, resulting in claims that are not unambiguously understood and verified. to address these issues, we propose a novel method for document-level claim extraction and decontexualisation for fact-checking, aiming to extract salient check-worthy claims from documents that can be understood outside the context of the document. specifically, assuming that salient document (cnbc news) @xcite between 8,000 and 10,000 e-scooters are being destroyed in the middle east by california scooter sharing start-up bird, according to sources. [2] they belong to circ, an e-scooter company that was acquired by bird in january. @xcite bird shut down its entire middle east operation as a result of bird is scrapping thousands of e-scooters in the middle east and shutting down its operations in the majority of the region as a result of the coronavirus pandemic, according to five people familiar with the matter. @xcite the e-scooters being scrapped belong to circ, which was acquired by bird for an undisclosed sum in january. @xcite there are between 8,000 and 10,000 circ scooters across cities in qatar, bahrain and united arab emirates, according to one former employee and one company source who asked to be kept anonymous as they've signed a confidentiality agreement. ……. [7] but there have been questions about the longevity of their vehicles, with reports suggesting some bird e-scooters have a life span of just a few months. [8] last week, it emerged that uber is scrapping thousands of e-bikes and e-scooters worth millions of dollars after selling its jump unit to mobility start-up lime. ……. ## method as illustrated in figure 2 , our proposed documentlevel claim extraction framework consists of four components: i) sentence extraction ( §3.1); extracts the sentences related to the central idea of the document as candidate claim sentences; ii) context generation ( §3.2), extracts context from the doc-ument for each candidate sentence; iii) sentence decontextualisation ( §3.3), rewrites each sentence with its corresponding context to be understandable out of context; iv) check-worthiness estimation ( §3.4), selects the final check-worthy claims from candidate decontextualised sentences. ## dataset we convert averitec, a recently proposed dataset for real-world claim extraction and verification @xcite , into averitec-dce, a dataset for the document-level ce task. averitec is collected from 50 different fact-checking organizations and contains 4568 real-world claims. each claim is associated with attributes such as its type, source and date. in this work, we mainly focus on the following attributes relevant to claim extraction: i) claim, the claim as extracted by the fact-checkers and decontextualised by annotators, and ii) source url: the url linking to the original web article of the claim. the task of this work is to extract the salient check-worthy claims from the source url. we also consider whether claims need to be decontextualised when extracting them from documents, as this will directly affect the subsequent evidence retrieval and claim verification. to extract claim-document pairs from averitec that can be used for document-level ce, we perform the following filtering steps: 1) since we focus on the extraction of textual claims, we do not include source urls containing images, video or audio; 2) to extract the sentences containing the claims from source urls, we build a web scraper to extract text data in the source url as the document. we found that the attribute source url is not always available in samples, thus we only select those samples where the web scraper can return text data from source urls. we obtain a dataset averitec-dce, containing 1231 available samples, for document-level ce. we do not divide the dataset into train, dev and test sets, as all models we rely on are pre-trained models (e.g., bert-sum) and approaches that do not require training (e.g., bm25). statistics for averitec-dce are described in ## experiments our approach consists of four components: sentence extraction, context generation, sentence decontextualisation and check-worthiness estimation. as such, we conduct separate experiments to evaluate them, as well as an overall evaluation for document-level ce. ## conclusions and future work this paper presented a document-level claim extraction framework for fact-checking, aiming to extract salient check-worthy claims from documents that can be understood out of context. to extract salient claims from documents, we recast the claim extraction task as the extractive summarization task to select candidate claim sentences. to make sentences understandable out of context, we introduce a qa-based decontextualisation model to enrich them with the necessary context. the experimental results show the superiority of our method over previous methods, including document-level claim extraction and evidence retrieval, as indicated by human evaluation and automatic evaluation. in future work, we plan to extend our document-level claim extraction method to extract salient checkworthy claims from multimodal web articles.