What to Fact-Check: Guiding Check-Worthy Information Detection in News Articles through Argumentative Discourse Structure (2021)

Abstract: Most existing methods for automatic fact-checking start with a precompiled list of claims to verify. We investigate the understudied problem of determining what statements in news articles are worthy to fact-check. We annotate the argument structure of 95 news articles in the climate change domain that are fact-checked by climate scientists at climatefeedback.org. We release the first multi-layer annotated corpus for both argumentative discourse structure (argument types and relations) and for fact-checked statements in news articles. We discuss the connection between argument structure and check-worthy statements and develop several baseline models for detecting check-worthy statements in the climate change domain. Our preliminary results show that using information about argumentative discourse structure shows slight but statistically significant improvement over a baseline of local discourse structure.

## introduction the proliferation of misinformation in online portals is increasing at a scale that calls for the automation of the slow and labor-intensive manual fact-checking process @xcite . the need for automation is even bigger in highly controversial topics such as climate change. an endto-end automatic fact-checking system needs to accomplish three main tasks: 1) find claims that are worth fact-checking, 2) retrieve relevant evidence from credible sources, and 3) determine the veracity of that claim given the retrieved evidence. most previous attempts at automating factchecking focus on the latter two steps by comparing a manually prepared list of claims against automatically-or manually-retrieved evidences from (trusted) sources such as wikipedia or news articles from credible publishers @xcite @xcite . however, less attention is given to automatically compiling a list of check-worthy statements that can then be inspected and fact-checked by a human fact-checker (or by a fact-checking system). a small number of previous studies developed datasets and models for identifying checkworthy statements in political news and debates @xcite @xcite . we look at the problem of deciding what sentences to fact-check in news articles and in particular in the climate change domain. we hypothesize that selecting segments for fact-checking in news articles, particularly for controversial topics, is related to the overall argumentative structure of the article, more specifically to the argument component type (e.g., claim, premise) and to the incoming and outgoing argumentative relations (e.g., support, attack) from or to the argument components. by looking at some of the fact-checked articles, we notice that the segments selected for fact-checking by climate scientists sometimes contain a claim, a premise, or a combination of both a claim and a premise. when we look at the context around the fact-checked segments, we notice patterns related to the argumentative structure. for example, human fact-checkers tend to fact-check a claim when it is not supported by an evidence (premise) or only supported by another claim, and fact-check a premise when it is used to support an claim (e.g., to challenge the relevance of that evidence in support for the claim). not all fact-checked segments are chosen on a basis related to the argumentative structure as we show in our analysis, however, having annotations of both fact-checked segments and argument component types allow us to understand and model this relation. figure 1 shows an excerpt from one article in our dataset with its argument and fact-checked segments annotations. our contributions in this paper are as follows foot_0 : ## related work previous work on fact-checking has focused on different steps of the fact-checking pipeline @xcite , the majority of which is work on predicting the veracity of claims either by comparing them against evidence from wikipedia @xcite , trusted news outlets @xcite , discussion forums @xcite , or debate websites @xcite , or by analyzing the linguistic properties of false and true claim @xcite in addition to the speaker's history @xcite . other work focuses on estimating the credibility of sources by using an external list of bias per publisher @xcite or by modeling conflicting reports on a claim from different sources @xcite . however, all of these methods either report bias at the publisher level or start with a list of claims to fact-check. previous work on detecting check-worthy claims focus on text from the political domain. the two main existing systems for check-worthy claim detection are claimbuster @xcite and claimrank @xcite . claimbuster is trained on sentences from political debates and uses sentence level features such as tf-idf weights and sentiment. claimrank extends this to arabic (in addition to english) and uses a richer feature set that includes the context. other more recent work include datasets that are bigger in size and across longer time spans @xcite or in other languages such as dutch @xcite . covering multiple domains (political speeches, tweets, wikipedia) and task formulations (check-worthiness, rumor detection, and citation detection), wright and augenstein (2020) use positive unlabelled learning @xcite to perform a comparison of datasets across domains where the notion of check-worthiness vary greatly. over the past three years, the clef check-that lab introduced tasks for detecting check-worthy political claims from debates and social media @xcite @xcite , where the best teams in the 2019 task @xcite uses syntactic features and word embeddings in an lstm model. more recently on the same datasets, kartal et al. (2020) introduce a logistic regression model using bert-based features, presence of comparative and superlative adjectives, augmented with data from controversial topics. finally, @xcite use adversarial training on transformer neural network models for detecting check-worthy statements. however, all of these models are trained on political text from debates, speeches and tweets, or lists of claims previously checked by various factchecking agencies such as factcheck.org. we on the other hand work on a dataset from a different genre: news articles, and from a different domain: climate change, and investigate the question whether argumentative discourse structure helps in detecting check-worthy statements. argument mining is a field concerned with finding argument structure in text from argument components (claim, premises) to relations (support, attack) as covered extensively by @xcite . several argumentation corpora are available on texts from multiple genres such as student essays @xcite , and social-media threads @xcite , which have been used in applications such as writing assistance @xcite and essay scoring @xcite . @xcite has argued that statements have different types which affects the type of evidence they need or lack thereof. this was empirically explored by works that attempted to identify the appropriate type of support for statements in user comments @xcite and controversial topics in the social media @xcite . in this work, we provide a resource and a model that aims to deepen our understanding of the relations between argumentative discourse structure and check-worthiness. ## multi-layer annotated corpus we describe below the dataset, its fact-checked segment annotation by climate scientists, and our argumentative discourse structures annotation on the same dataset. ## analysis of argumentation in fact-checked segments to further understand the relation between argumentative discourse structure and fact-checked segments, we analyze the argument components types and relations of the fact-checked segments in the training data. to see the effect of our strategy in selecting gold argumentative spans and relations on the overlap with fact-checked segments, we do our analysis using the annotations of the best annotator for each article (overall highest in pairwise agreement with other annotators), and the gold annotations. we look at the original fact-checked segments before they are split to sentences as described in section 3.1. this results in 589 factchecked segments that mostly consist of multiple sentences (splitting them to sentences increases the number to 824 fact-checked sentences). ac total relation frequency type rel. type claim 1 sup --→ claim 12 1 sup --→ major-claim 11 premise 1 sup --→ claim 54 1 sup --→ premise 4 @xmath0: relation types count in gold argument component types. we first look at the best annotator's coding. out of the 589 factchecked segments, 430 map to argument components in the articles. out of argumentative factchecked segments, 53% consist of a single argument component: 95 are claims, 82 are premises and 17 are major-claims, while the remaining consist of two (25%), three (10%), or four or more argument components (12%). when we use the gold annotations, the number of annotated segments in most articles decreases due to only including segments that are annotated by two or more annotators. this reduces the argumentative fact-checked segments from 430 to 307 out of the 589 total fact-checked segments. this reductions cascades to the frequency of argument component types ( argumentative relations. when we look at the relations from and to argument components that are fact-checked (as annotated by the best annotator), we notice that a premise is fact-checked when it has one relation (mostly an outgoing sup-port relation) and a claim is fact-checked when it has many relations (up to four) with mixed directions (incoming, outgoing) and types (support, attack). this essentially maps to fact-checking a premise when it is used as a supportive evidence and fact-checking a claim when it is central to the overall argument of the article. also, claims and major-claims are fact-checked when they are only supported by other claims (which could signal that the author is not providing an evidence, thus showcasing an "evading the burden of proof " fallacy). the most frequent relation counts of fact-checked segments are shown in the general patterns found in the annotations of the best annotator still hold for the gold annotations. the only exception in the gold annotations is that a major-claim is fact-checked more often that segments consisting of two premises or two claims, which is mainly due to the smaller count of argument component (and relations) in the gold annotations. more detailed counts are shown in appendix b. ## experimental setup we use the climate scientists' decision to factcheck a sentence as our gold labels for checkworthiness. in order to understand the capability of machine learning models to decide whether a sentence should be fact-checked, we introduce an experimental setup as follows. in line with previous work, we formulate this problem in two ways: a) sentence classification task, i.e. determining whether a given sentence should be factchecked or not, and b) sentence ranking by checkworthiness. for the sentence classification task, we use macro f1 scores as our evaluation metric, while for ranking we use mean average precision (map). we experiment with fine-tuning bert @xcite using the transformers library by huggingface @xcite with and without argumentation-based selection of context as described below. baselines. we fine-tune bert for 3 epochs (bertbase-uncased, max sequence length 256, batch size 16, learning rate 2e-5) using three different inputs to establish a baseline for this task. the first baseline is fine-tuning using only the target sentence for classification as the input (sent). the other two configurations utilize the capability of bert to handle two inputs. therefore, we experiment with passing the target sentence with its previous sentence as input (prev+sent) and with its next sentence (sent+next). these two configurations essentially provide local discourse context following the natural order of sentences in the article. argumentation context. one simple way to test our hypothesis on the relation between argumentation and check-worthiness is by selecting a context for the target sentence using the argumentative discourse structure. we refer to such context as the argumentation context in our discussion. if the target sentence is argumentative, we look at its outgoing and incoming argumentative relations. if the sentence has an incoming relation, then the source of that relation is passed as the first input of bert and the target sentence is passed as the second input. if the relation is outgoing from the target sentence, then the target sentence is passed as the first input and the target of the relation is passed as the second. as a single sentence could consist of more than one argument component, which in turn could have many relations, this creates many pairs for the target sentence. we explore three configurations for using the argument structure to select context. first, we keep all pairs for each target sentences, thus increasing the number of instances in the data and maintaining the same gold label for each repeated target sentence in the training data that is matched with a different argumentation context. we denote such configuration as ac(all) in our discussion. the final label during inference time can be determined in two ways: via majority label of predictions for each target sentence, and via favoring the minority class, i.e., if one prediction is to fact-check then we consider that as the final label. second, we select some of the argumentation context by keeping the most frequent relations in fact-checked segments seen in training as discussed in section 4. if the target sentence has a claim or major-claim, then we only keep incoming support relations from other claims or major-claims. however, if the target sentence has a premise, we keep outgoing relations to claims or major-claims. we also limit the total number by either 3 (ac(3)) or 1 (ac(1)) selecting at random if the remaining relations exceed the limit. in case the target sentence is not argumentative, we revert to the discourse context by selecting the previous sentence. third, we experiment with prepending argument component type of the target sentence and its context to the input text (e.g., if the sentence has a ## results and discussion however, providing more than one sentence does not improve the results in the ac(3) and ac(all) experiments as shown in ## conclusion we introduced a corpus of news articles with multilayer annotations of check-worthiness and argumentative discourse structure to further our understanding of the relation between argumentation and fact-checking. we approached the task of determining what sentences to fact-check in a news articles formulated as a sentence classification task and as a sentence ranking task. we showed that providing an argumentative discourse context along with the target sentence when fine-tuning bert improves over baselines of the target sentence alone or with its local discourse context, especially on the argumentative part of the articles. in future work, we want to compare using the gold annotations of argument structure with predicted argument components and relations by training another model that generate argumentation fea-tures to be used for the main task as done in previous work @xcite . also, we want to explore the use of other linguistic features tested in previous work and other variations of argumentation context and features such as counts of relations for the target argumentative segment. bert is pretrained on the next sentence prediction task, which makes an out-of-order argumentation context to be further away from the distribution of the pretraining data. to remedy this, we plan to adaptively pretrain bert on more argumentation context extracted from multiple argumentation corpora. finally, we want to study the relation of check-worthiness to intrinsic clause types such as facts and testimony, and to argument fallacies not related to the argument structure.