N ews C laims: A New Benchmark for Claim Detection from News with Attribute Knowledge (2022)

Abstract: Claim detection and verification are crucial for news understanding and have emerged as promising technologies for mitigating misinformation and disinformation in the news. However, most existing work has focused on claim sentence analysis while overlooking additional crucial attributes (e.g., the claimer and the main object associated with the claim).In this work, we present NewsClaims, a new benchmark for attribute-aware claim detection in the news domain. We extend the claim detection problem to include extraction of additional attributes related to each claim and release 889 claims annotated over 143 news articles. NewsClaims aims to benchmark claim detection systems in emerging scenarios, comprising unseen topics with little or no training data. To this end, we see that zero-shot and prompt-based baselines show promising performance on this benchmark, while still considerably behind human performance.

## introduction the internet era has ushered in an explosion of online content creation, resulting in increased concerns regarding misinformation in news, online debates, and social media. a key element of identifying misinformation is detecting the claims and the arguments that have been presented. in this regard, news articles are particularly interesting as they contain claims in various formats: from arguments by journalists to reported statements by prominent public figures. check-worthiness estimation aims to decide if a piece of text is worth fact-checking, i.e., whether it contains an important verifiable factual claim @xcite . most current approaches @xcite largely ignore relevant attributes of the claim (e.g., the claimer and the primary object associated with the claim). 1 the code and data have been made publicly available here: https://github.com/blender-nlp/newsclaims moreover, current claim detection tasks mainly identify claims in debates @xcite , speeches @xcite , and social media @xcite , where the claim source (i.e., the claimer) is known. news articles, on the other hand, have more complex arguments, requiring a deeper understanding of what each claim is about and identifying where it comes from. thus, here we introduce the notion of claim object, which we define as an entity that identifies what is being claimed with respect to the topic of the claim. figure 1 shows a claim about the origin of covid-19, suggesting that the virus came from space, which is the claim object. we further identify the claimer, which could be useful for fact-checking organizations to examine how current claims compare to previous ones by the same person/organization. in this regard, we extend the claim detection task to ask for the extraction of more attributes related to the claim. specifically, given a news article, we aim to extract all claims pertaining to a set of topics along with the corresponding claim span, the claimer, the claimer's stance, and the claim object for each claim. the claim attributes enable comparing claims at a more fine-grained level: claims with the same topic, object and stance can be considered equivalent whereas those with similar claim objects but opposing stance could be contradicting. we note that while identifying the claim span and stance have been explored independently in prior work @xcite , we bring them into the purview of a unified claim detection task. to promote research in this direction, we release newsclaims, a new evaluation benchmark for claim detection. we consider this in an evaluation setting since claims about new topics can emerge rapidly foot_0 , requiring systems that are effective under zero/few-shot settings. newsclaims aims to study how existing nlp techniques can be leveraged to tackle claim detection in emerging scenarios and regarding previously unseen topics. we explore multiple zero/few-shot strategies for our subtasks including topic classification, stance detection, and claim object detection. this is in line with recent progress in using pre-trained language models in zero/few-shot settings @xcite . such approaches can be adapted to new use cases and problems as they arise without the need for large additional training data. in our benchmark, all news articles are related to the covid-19 pandemic, motivated by multiple considerations. first, covid-19 has gained extensive media coverage, with the world health organization coining the term infodemic foot_1 to refer to disinformation related to covid-19 (naeem and bhatti, 2020) and suggesting that "fake news spreads faster and more easily than this virus". second, this is an emerging scenario with limited previous data related to the virus, making it a suitable candidate for evaluating claim detection in a low-resource setting. newsclaims covers claims about four covid-19 topics, namely the origin of the virus, possible cure for the virus, the transmission of the virus, and protecting against the virus. our contributions include (i) extending the claim detection task to include more attributes (claimer and object of the claim), (ii) releasing a manually annotated evaluation benchmark for this new task, newsclaims, which covers multiple topics related to covid-19 and is the first dataset with such extensive annotations for claim detection in the news, with 889 claims from 143 news articles, and (iii) demonstrating promising performance of various zero-shot and prompt-based few-shot approaches for the claim detection task. argumentation mining @xcite @xcite includes context-dependent claim detection @xcite , which entails detecting claims specifically relevant to a predefined topic. however, claims in the context of argumentation are neither necessarily factual nor verifiable. moreover, prior work on both check-worthiness estimation and argumentation mining did not deal with identifying additional claim attributes, such as the claimer, or the source of the claim, and the claim object. the claimer detection subtask is related to attribution in the news. current attribution methods are mainly sentence-level @xcite or only involve direct quotations @xcite . in contrast, we require cross-sentence reasoning for identifying the claimer as it may not be present in the claim sentence (see figure 1 ). there has been recent work addressing claims related to covid-19. @xcite proposed a new fever-like @xcite dataset, where given a claim, the task is to identify relevant evidence and to verify whether it refutes or supports the claim; however, this does not tackle identifying the claims or the claimer. there has also been work on identifying the check-worthiness of tweets related to covid-19 @xcite ; however, unlike news articles, tweets do not require attribution for claimer identification. ## proposed claim detection task our task is to identify claims related to a set of topics in a news article along with corresponding attributes such as the claimer, the claim object, and the claim span and stance, as shown in figure 2 . claim sentence detection: given a news article, the first subtask is to extract claim sentences relevant to a set of pre-defined topics. this involves first identifying sentences that contain factually verifiable claims, similar to prior work on checkworthiness estimation, and then selecting those that are related to the target topics. to address misinformation in an emerging real-world setting, we consider the following topics related to covid-19: origin of the virus: claims related to the origin of the virus (i.e., location of first detection, zoonosis, 'lab leak' theories); transmission of the virus: claims related to who/what can transmit the virus or conditions favorable for viral transmission; cure for the virus: claims related to curing the virus, (e.g., via medical intervention after infection); and protection from the virus: claims related to precautions against viral infection. claimer detection: claims within a news article can come from various types of sources such as an entity (e.g., person, organization) or published artifact (e.g., study, report, investigation). in such cases, the claimer identity can usually be extracted from the news article itself. however, if the claim is asserted by the article author or if no attribution is specified or inferrable, then the article author, i.e. the journalist, is considered to be the claimer. the claimer detection subtask involves identifying whether the claim is made by a journalist or whether it is reported in the news article, in which case the source is also extracted. moreover, sources of such reported claims need not be within the claim sentence. in our datatset newsclaims, the claimer span was extracted from outside of the claim sentence for about 47% of the claims. thus, the claimer detection subtask in our benchmark requires considerable document-level reasoning, thus making it harder than existing attribution tasks @xcite , which require only sentence-level reasoning. claim object detection: the claim object relates to what is being claimed in the claim sentence with respect to the topic. for example, in a claim regarding the virus origin, the claim object could be the species of origin in zoonosis claims, or who created the virus in bioengineering claims. ## the newsclaims dataset in this work, we build newsclaims, a new benchmark dataset for evaluating the performance of models on different components of our claim detection task. specifically, we release an evaluation set based on news articles about covid-19, which can be used to benchmark systems on detecting claim sentences and associated attributes including claim objects, claim span, claimer, and claimer stance. newsclaims uses news articles from the ldc corpus ldc2021e11, from which we selected those related to covid-19. we describe below the annotation process (section 4.1) and provide statistics about newsclaims (section 4.2). ## baselines in this section, we describe various zero-shot and prompt-based few-shot learning baselines for the claim detection subtasks outlined in section 3. we describe a diverse set of baselines with each chosen to be relevant in an evaluation-only setting. ## experiments in this section, we evaluate various zero-shot and few-shot approaches for the subtasks of our claim detection task. to estimate the upper bounds, we also report the human performance for each subtask computed over ten random news articles. ## conclusion and future work we proposed a new benchmark, newsclaims, which extends the current claim detection task to extract more attributes related to each claim. our benchmark comprehensively evaluates multiple aspects of claim detection such as identifying the topics, the stance, the claim span, the claim object, and the claimer in news articles from emerging scenarios such as the covid-19 pandemic. we showed that zero-shot and prompt-based few-shot approaches can achieve promising performance in such low-resource scenarios, but still lag behind human performance, which presents opportunities for further research. in future work, we plan to explore extending this to build claim networks by identifying relations between the claims, including temporal connections. another direction is build a unified framework that can extract claims and corresponding attributes together, without the need for separate components for each attribute.