The Role of Context in Detecting Previously Fact-Checked Claims
Abstract
Recent years have seen the proliferation of disinformation and fake news online.
Traditional proposals to mitigate these problems are manual and automatic fact-checking.
Recently, another approach has emerged: checking whether the input claim has previously been fact-checked, which can be done automatically, and thus fast, while also offering credibility and explainability, thanks to the human factchecking and explanations in the associated fact-checking article.
Here we focus on claims made in a political debate, where context really matters.
We study the impact of modeling the context of the claim: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking explanation document.
We do this by modeling the local context, the global context, as well as by means of coreference resolution, and multi-hop reasoning over the sentences of the document describing the fact-checked claim.
The experimental results show that each of these represents a valuable information source, but that modeling the source-side context is more important, and can yield 10+ points of absolute improvement over a state-of-the-art model.
Introduction
The fight against dis/mis-information has become an urgent social and political issue.
Online media have been widely used not only for social good but also to mislead entire communities.
Many factchecking organizations, such as FactCheck.org, Snopes, 2 PolitiFact, 3 and FullFact, 4 as well as some broader international initiatives such as the Credibility Coalition 5 and Eufactcheck , 6 have emerged to address the issue (Stencel, 2019).
At the same time, there have been efforts to develop automatic systems to detect and to flag such content (Vo and Lee, 2018; Shu et al., 2017; Thorne and Vlachos, 2018; Li et al., 2016; Lazer et al., 2018; Vosoughi et al., 2018a), including the development of datasets (Hassan et al., 2015; Augenstein et al., 2019), systems, and evaluation campaigns (Barrón-Cedeño et al., 2020; Nakov et al., 2021a,b; Shaar et al., 2021; Nakov et al., 2022).
An important issue with automatic systems is that journalists and fact-checkers often question their credibility for reasons such as (perceived) insufficient accuracy given the state of present technology, but also due to the lack of explanation about how the system has made its decision.
On the other hand, manual fact-checking is time-consuming as it requires to go through several manual steps. It has been reported in the literature that fake news travels faster than real news (Vosoughi et al., 2018b), and that 50% of the spread of the viral claims happens within first ten minutes (Zaman et al., 2014). Such findings show the importance of real-time detection of the factuality of the claims, which can make it possible to take timely action.
As both manual and automatic systems have their limitations, there have been also proposals of human-in-the-loop settings, aiming to bring the best of both worlds.
In order to enable such an approach, one question that arises is how to facilitate fact-checkers and journalists with automated systems.
An immediate interesting problem is to know whether a given input claim has been previously fact-checked by a reputable fact-checking organization.
This would give them a credible reference and could save them significant amount of time, as manually fact-checking a single non-trivial claim may take from 1-2 days to 1-2 weeks.
While earlier studies have suggested that such a mechanism should be part of an end-to-end automated system, there has been limited work in this direction (Shaar et al., 2020;
Vo and Lee, 2020). Looking from a different perspective, at the time of COVID-19, we see the same false claims and conspiracy theories coming over and over again (e.g., about garlic water as a cure, about holding your breath for 10 seconds as a way to test for COVID-19, etc.).
That is why fact-checking makes sense: to debunk such frequent claims.
The problem is that next time they come in a slightly different form (although having the same meaning), it is important to be able to recognize them quickly and possibly to post a reply in social media with a link to a fact-checking article.
If we consider a scenario in which a politician is being interviewed or is taking part in a debate, a quick response would make it possible to put him/her on the spot.
However, the problem in such a real-time scenario is that, unlike written text, interviews, debates, and speeches are more spontaneous, and the claims that are being made are often not clearly formulated in a single sentence.
This is illustrated in Figure 1, where we can see a fragment from a
Democratic debate for the 2016 US Presidential election, where Hillary Clinton said: “ I waited until it had actually been negotiated because I did want to give the benefit of the doubt to the administration. ”
Understanding this claim requires pronominal co-reference resolution (e.g., what does it refer to, is it CAFTA or is it TPP , as both are mentioned in the previous sentences), more general co-reference (e.g., that the administration being discusses is the Obama administration), as well as a general understanding of the conversation so far, and possibly general world knowledge about US politics at the time of the debate (e.g., that Hillary
Clinton was Secretary of State when TPP was being discussed).
Moreover, previous research has shown that it is beneficial to match the input claim not only against the canonical verified claim that fact-checkers worked with, but against the entire article that they wrote explaining why the claim was ultimately judged to be true/false (Shaar et al., 2020;
Vo and
Lee, 2020).
This is because, in the fact-checking article, the claim is likely to be mentioned in different ways, and also background information and related terms would be used, which can facilitate claim matching, and thus improve recall.
This means that we need to exploit global contextual information contained within whole fact-checking articles or at least previous and following context of the claim (i.e., local context).
Similarly, for the FEVER fact-checking task against Wikipedia, it has been shown that multi-hop reasoning (Transformer-XH) over the sentences of the target article can help (Zhao et al., 2019), an observation that was further confirmed in the context of fact-checking political claims (Ostrowski et al., 2021).
TransformerXH uses a novel attention mechanism that naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence.
As claims and their reasonings are manifested across documents, this hop-based attention mechanism constructs global contextualized representation to provide better joint multi-evidence reasoning.
We rely on Transformer-XH to extract and use global contextual information.
Based on the above considerations, we propose a framework that focuses on modeling the coreference, local context (features from neighboring sentences, see Section 4.2.2) and global context (features using Transformer-XH, see Section 4.2.3), both on the source and on the target side, while also using multi-hop reasoning over the target side.
Our contributions can be summarized as follows:
• We perform careful manual analysis to understand what makes detecting previously factchecked claims a hard problem, and we categorize the claims by type.
We release these annotations to enable further research.
• Unlike previous work, we focus on modeling the context both on the source side and on the target side, both local and global, using co-reference resolution and reasoning with Transformer-XH, which yields sizable improvements over state-of-the-art models of over 10 MAP points absolute.
• We propose a realistic and challenging, timesensitive and document-aware, data split compared to previous work, which we also release.
The rest of the paper is organized as follows. Section 2 provides a brief overview of previous work. Sections 3 introduce the the dataset development process. Section 4 presents the experiments and 5 discusses the evaluation results.
Finally, Section 6 concludes the paper with lessons learned and points to possible directions for future work.
Related Work 2. Check-Worthiness Estimation
Notable work in this direction includes contextaware approaches to detect check-worthy claims in political debates (Gencheva et al., 2017), using various patterns to find factual claims (Ennals et al., 2010), multitask learning (Vasileva et al., 2019), and a variety of other approaches used by the participants of the CLEF CheckThat!
labs’ shared tasks on checkworthiness (Nakov et al., 2018;
Elsayed et al., 2019b,a; Vasileva et al., 2019).
2. Previously Fact-Checked Claims
While there is a surge in research to develop systems for automatic fact-checking, such systems suffer from credibility issues, e.g., in the eyes of journalists, and manual efforts are still the norm.
Thus, it is important to reduce such manual effort by detecting when a claim has already been factchecked.
Work in this direction includes (Shaar et al., 2020) and (Vo and Lee, 2020): the former developed a dataset for the task and proposed a ranking model, while the latter proposed a neural ranking model using textual and visual modalities.
A recent work by Sheng et al.
(2021) highlights the importance of lexical, semantic, and patternbased information and proposes a re-ranker based on memory-enhanced transformers for matching (MTM) to detect and rank previously fact-checked claims.
2. Semantic Matching and Ranking
Here we focus on the textual formulation of the problem, as defined by Shaar et al.
(2020): given an input claim, we want to detect potentially matching previously fact-checked claims and to rank them accordingly.
A related research area is semantic matching and ranking, as matching some InputClaim – VerClaim pairs might require BERT-based sentence embeddings, natural language inference, and coreference resolution.
An example of such a difficult pair is shown in Table 1, line 607.
Recent relevant work in this direction uses neural approaches.
Nie et al.
(2019) proposed a semantic matching method that combines document retrieval, sentence selection, and claim verification neural models to extract facts and to verify them.
Thorne et al.
(2018) proposed a very simple model, where pieces of evidence are concatenated together and then fed into a Natural Language Inference (NLI) model.
Yoneda et al.
(2018) used a fourstage approach that combines document and sentence retrieval with NLI.
Hanselowski et al.
(2018) introduced Enhanced Sequential Inference Model (BiLSTM based)
(Chen et al., 2017) methods to rank candidate facts and to classify a claim based on the selected facts.
Several studies used model combination (i.e., document retrieval, sentence retrieval, and NLI for classifying the retrieved sentences) with joint learning (Yoneda et al., 2018; Hidey and Diab, 2018; Luken et al., 2018).
2. Context Modeling for Factuality
Fact-checking is a complex problem.
It requires retrieving pieces of evidence, which are often scattered in the document.
Once they have been retrieved, they can be used to verify the claim.
The evidence with contextual information can play a great role for fact verification and retrieval.
Previous work has shown that the relation between the target statement and a context in the document (e.g., debate), the interaction between speakers, and the reaction of the moderator and the public can significantly help to find check-worthy claims (Gencheva et al., 2017).
Liu et al.
(2020) proposed a graph-based approach, a Kernel Graph Attention Network, to use evidence as context for fact verification.
Similarly, Zhou et al.
(2019) used a fully connected evidence graph with multi-evidence information for fact verification.
Zhong et al.
(2020) used different pre-trained Transformer models and a graph-based approach (i.e., graph convolutional network and graph attention network) for fact verification.
Zhao et al.
(2019) introduced extra hop attention to incorporate contextual information, while maintaining the Transformer capabilities.
The extra hop attention enables it to learn a global representation of the different pieces of evidence and to jointly reason over the evidence graph.
It is a promising approach that uses contextual information as a graph representation and Transformer capabilities in the same model.
One of the limitations is the need for human-labeled evidence in relation to the input claims in existing fact-verification datasets.
The study by Ostrowski et al.
(2021) addressed this limitation by developing a dataset of annotated pieces of evidence associated with input claims and explored multihop attention mechanism, proposed in (Zhao et al., 2019), to make prtediction on the factuality of a claim.
Unlike the above work, here we target a different task: detecting previously fact-checked claims as opposed to performing fact-checking per se.
Moreover, while the above work was limited to the target, we also model the source context (which turns out to be much more important).
Dataset
Here, we focus on the problem of detecting previously fact-checked claims, using the task formulation and also analysis and adaptation of the data from (Shaar et al., 2020).
They had two datasets: one on matching tweets against Snopes claims, and another one on matching claims in the context of a political debate to PolitiFact claims.
Here, we focus on the latter,
and we perform a close analysis of the claims and what makes them easy/hard to match.
The dataset was collected from the US political fact-checking organization PolitiFact.
After a US political debate, speech, or interview, fact-checking journalists would select few claims made in the event and would verify them either from scratch or by linking them to a previously fact-checked claim.
Each previously fact-checked claim has an associated article stating its truthfulness along with a justification.
The dataset has two parts: ( i ) verified claims {normalized VerClaim , article title , and article text }, ( ii ) transcripts of the political events (e.g., debates).
They annotated the data by linking sentences from the transcript ( InputClaim ) to one or more verified claim (out of 16,636 claims).
To further analyze the dataset, we looked at the
InputClaim – VerClaim pairs, and we manually categorized them into one of the following categories:
clean : A clean pair is a self-contained InputClaim with a VerClaim that directly verifies it (see line 255 in Table 1 for an example).
clean-hard : A clean-hard pair is a selfcontained InputClaim with a VerClaim that indirectly verifies it (see line 688 in Table 1).
part-of : A part-of ’s pair InputClaim is not self-contained and requires the addition of other sentences from the transcript to fully form a single claim.
context-dep : A context-dep pair is similar to clean and clean-hard ; however, the InputClaim is not self-contained and needs coreference.
These categories include all types of pairs we have seen.
Moreover, since the dataset is constructed from speeches, debates, and interviews, the structure of the InputClaim – VerClaim pairs differs.
For example, in debates, we see more part-of examples, as there are multiple questions– answers claims and back-and-forth arguments splitting the claims into multiple sentences.
The annotations were performed by three annotators who are experts in fact-checking (and coauthors of this paper), using the above definitions for the categories.
We consolidated their annotations using majority voting, and they had a consolidation discussion for cases with no majority.
The
Fleiss Kappa inter-annotator agreement was 0.5, which corresponds to moderate agreement, which is reasonable for such a complex annotation task.
[Table] [Table]
Table 1 shows examples of InputClaim –
VerClaim pairs that illustrate four categories.
We can see that, due to the presence of cases like line 607 and 695–699, the task goes beyond simple textual similarity and natural language inference.
Recognizing the context-dep pairs requires understanding the InputClaim ’s local context, and recognizing the clean-hard pairs requires analysis of the overall global context of the VerClaim .
While annotating the data into the four categories described in this section, we found out that a few InputClaimVerClaim pairs in (Shaar et al., 2020) were false matches (which happened, as they did the matching automatically, without manually double-checking every single example) and we removed them. [Table]
Thus, our reported number of pairs is slightly lower, but it is also more accurate.
Table 2 shows statistics about the distribution the four categories of claims in the dataset.
We can see that clean and clean-hard are the most frequent categories, while part-of is the least frequent one.
We also investigated previous work and observed that they dealt with each InputClaim independently, i.e., at the sentence level.
That means two claims from the same debate can end up being in the training set and test set.
This is problematic because if we have pairs that are categorized as part-of , we could end up splitting them and putting them in different sets, i.e., train and test.
Moreover, splitting the dataset in this manner has another implication: the discussed topics in the input claim can fall into both training and test sets.
To avoid such issues we can split the data in different settings that reflects various scenarios:
• Debate-Level Chrono :
We split the data chronologically.
We use the first 50 debates for training and the last 20 for testing.
Specifically, we have 554 pairs for training, and 141 pairs for testing.
This is a more realistic scenario, where we would only have access to earlier debates, and we can use them to make decisions about claims made in future debates.
The complexity of this setting is also reflected in the MAP score as shown in Table 3.
We see that this score is lower than the best model in the previous work (last row).
This is because this setting is complex as we use a model trained on debates and speeches from 2012-2018, and we test on debates from 2019.
Across those different time frames, different politicians discuss different topics.
• Debate-Level Semi-Chrono : We split the data per year, e.g., for year 2018, we divide the transcripts into train and test with 80/20 splits, and then we train and evaluate using the same reranking model.
In Table 3, we can see an improvement with this setting compared to the Debate Level Chrono setting.
This might be because the same politicians discuss same/similar issues throughout the same year.
• Debate-Level Random : We randomly choose 80% of the debates for training and the remaining ones for testing.
This is a comparatively easier setting as the data is randomly distributed in training and testing.
This is also reflected in the results in Table 3.
The reason could be that politicians repeat themselves a lot, especially in two consecutive political events, and the random split can lead to having two similar debates/speeches in two splits.
• Sentence Level Random :
This is the setting used in (Shaar et al., 2020), where sentences from the debates are randomly divided into train and test set with 80% and 20% proportion, respectively.
This is the most unrealistic split.
In the rest of the experiments we choose to use the more realistic setup Debate Level Chrono , which means that our baseline MAP score (which is in fact the state-of-the-art from previous work) goes down from 0.602 to 0.429.
Experimental Setup 4. Baseline
From our analysis of the dataset (described in Section 3), we conclude that ( i ) we need to resolve the references in the InputClaim , ( ii ) to capture the local context of the InputClaim , and ( iii ) to encapsulate the global context of the VerClaim .
For the baseline, we use the same setup as in the state-of-the-art model of Shaar et al.
We use the claim as a query against the full text of the documents using BM25 (a hard-to-beat model from information retrieval).
We then train a reranker on the top-100 results returned by BM25 using rankSVM (Herbrich et al., 1999) with an RBF kernel.
The reranker uses nine similarity measures that compare the InputClaim to the VerClaim , as well as the respective reciprocal ranks.
In particular, we compute the BM25 score for InputClaim vs. VerClaim , title , text , VerClaim + title + text .
We also compute the cosine using sentence-BERT embeddings for InputClaim vs. VerClaim , title , and the top-4 sentences from text .
Using these scores, we create a vector representation of the InputClaim – VerClaim pair with dimensionality R 18 .
We then scale the vectors of all InputClaim –
VerClaim pairs in [ − 1; 1] and we train a rankSVM with the default parameters ( KernelDegree = 3 , γ = 1 /num _ features , ϵ = 0 . 001 ).
4. Proposed Models
As shown in Figure 1, our model uses co-reference resolution on the source and on the target side, the local context (i.e., neighboring sentences as context), and the global context (Transformer-XH) as discussed below.
It is still a pairwise reranker, but with a richer context representation.
4.2. Co-reference Resolution
We manually inspected the training transcripts and the associated verified claims, and we realized that there were many co-reference dependencies.
Thus, resolving them can help to obtain more representative textual and contextual similarity scores.
As for the verified claims, we noticed that not all VerClaim were self-contained, and that some understanding of the context was needed 8 from the article’s text that explains the verdict provided by the
PolitiFact journalists.
Therefore, our hypothesis is that resolving such co-references should improve the downstream matching scores.
For the same reason, we also performed coreference resolution on the PolitiFact articles when they were used to compute the BM25 scores.
We explored different co-reference models such as NeuralCoref , 9 e2e-coref 10 and SpanBERT 11 .
We found that NeuralCoref model performed best on the transcripts, while e2e-coref was best on the
VerClaim s. Hence, in the rest of the experiments, we show results using NeuralCoref for the source side, and e2e-coref for the target side.
We resolved the co-reference in the InputClaim by performing co-reference resolution on the entire input transcript (as was suggested in the literature); we will refer to this approach as srccoref .
As for the verified claims, we aimed to resolve the co-references in both the VerClaim and the text of the PolitiFact articles.
We also aimed to ensure that the dependencies from the text can be used for the VerClaim .
Therefore, we concatenated both the text and VerClaim (in the same order), and we applied the co-reference model on the concatenated text.
We choose this order of concatenation because the published text reserves the last paragraph to rephrase the VerClaim and to provide a summary of the justification; hence, there is a higher probability to resolve the co-references correctly.
4.2. Local Context
Resolving the pronominal co-references allows us to obtain the correct objects and the names the InputClaim is referring to.
However, in the process of analyzing the dataset, we noticed that different VerClaim s, although having similar structure, could be talking about different things, depending on the article text and also on the surrounding context.
Therefore, it is important to understand the context of an InputClaim .
In particular, we achieve this by performing a feature-level concatenation of the neighboring sentences in the transcript, i.e., we take the eighteen features ( R 18 , as discussed in Section 4.1 above) for the neighboring sentences, and we concatenate them to the similarity score for the InputClaim .
We then use the resulting represenattion as a feature vector to be fed into our reranker.
For example, if we take three sentences before the InputClaim and one sentence after, we denote this as FC (3, 1).
Let S i be our InputClaim , which is the i ’th sentence in the transcript.
We compute the similarity measures and the reciprocal rank (as described in
Section 4.1) to obtain the vector representation S i,v for S i .
With k = 3 previous and l = 1 following neighbouring sentences our final feature vector is
[Equation]
where ++ represents concatenation.
After the concatenation, the resulting dimension of the feature vector for FC(3, 1) is 18 × (3 + 1 + 1) = 90 .
4.2. Global Context
The similarity scores leveraging the local context are obtained from the textual content of the InputClaim and the VerClaim ( i ) using BM25, ( ii ) cosine similarity between the Sentence-BERT embeddings of InputClaim vs. the top-4 sentences of the VerClaim .
This might miss relevant information further away from the InputClaim in the input document and further away from VerClaim in the document accompanying the VerClaim .
We refer to such scattered information as global context .
To capture it, we adapt a graph-based Transformer,
Transformer-XH (Zhao et al., 2019).
In particular, we use a Transformer-XH model pretrained on the
FEVER (Fact Extraction and VERification) dataset, which is trained to predict whether a given input claim is supported/refuted by a set of target sentences (from Wikipedia), represented as a graph, or there is no enough information.
We used the model that is publicly made available by (Zhao et al., 2019).
For a given InputClaim , we generate a graph for each of the top-
VerClaim s retrieved from the BM25 algorithm using the normalized claim, the title and the top-3 sentences from the text as nodes.
Using the Transformer-XH model on the graph, we obtain three additional scores that correspond to the posterior probability that VerClaim supports or refutes the InputClaim , or there is no enough information.
4. Hyper-Parameter Values
For the baseline, we use the best values of the hyperparameters as found in (Shaar et al., 2020).
For our context-aware models, we select the values of the hyper-parameters by splitting the training dataset into train-train (debates from 2012-2017) and train-dev (debates from 2018), then training on train-train, and testing on train-dev. [Table]
4. Evaluation Measures
As we have a ranking task, we use mean average precision (MAP).
It is a suitable score as some InputClaim s have more than one VerClaim paired to them.
This is why we opted for not using mean reciprocal rank (MRR), which would only pay attention to the rank of the highest-ranked match [Table]
Results 5. Source-Side Experiments
For the source side experiments, we used coreference resolution on transcripts and variations of the local context by varying k and l in Eq. 1.
When we inspected the transcripts, we found that co-references tend to be resolved by a few sentences before the InputClaim ; therefore, we tried
FC (1, 1), FC (3, 1), FC (3, 3), and FC (5, 1).
We obtained the best results (on cross-validation) using FC (3, 1), which we use in this study.
As shown in Table 4, local context (Line 2) has improved over the baseline (Line 1) by 8 MAP points absolute.
We then experiment using co-reference resolution with the NeuralCoref model.
Compared to the baseline, we have a sizable improvement using co-reference resolution as shown in line 3, in
Table 4.
Specifically, in part-of and context-dep , because those pairs have many co-references that confuses the InputClaim .
After combining both methods, i.e., src-coref and FC(3,1)
(Line 4), we achieved the highest MAP score of 0.532.
As expected, we always see an increase in the performance for the clean category as the resolved
InputClaim can match the article text better.
5. Target-Side Experiments
For the target side experiments, we investigate using co-reference resolution (on the source and on the target side) in the VerClaim and their documents, as well as modeling the global context with (Transformer-XH) .
Compared to the baseline, we see a sizable improvement (from 0.365 to 0. MAP points) in clean-hard as can be seen in lineof Table 4.
This outcome is expected as the pair does not exhibit much semantic similarity, and we need to build our own understanding of the text of the VerClaim in order to capture the contextual similarity in the pair.
We also experiment with co-reference resolution on the VerClaim and the text of the VerClaim and also see some improvement.
Combining tgt-coref and (Transformer-XH) (line 7) improved the performance over tgt-coref alone, but it underperforms (Transformer-XH) alone.
The combination outperforms other target-side experiments on clean type.
5. Source-Side & Target-Side Experiments
Eventually, we tried to combine modeling the source and the target side.
Line 8 in Table 4 shows the evaluation result when we use co-reference resolution both on the source and on the target side.
We can see that this yields better overall MAP score of 0.487, compared to using source-side (MAP of 0.479; line 3) or target-side only (MAP of 0.443; line 6).
Moreover, co-reference resolution on both the source and target improves clean-hard and partof pairs (compared to using co-reference on one side only) as they require better local and global context, respectively.
We further tried putting it all together, and the result is shown in line 9.
12 While this yielded better results for clean , it was slightly worse compared to the source-side context modeling combination, in line 4.
This is probably due to source-side context models being generally stronger than target-side ones (compare lines 2–3 to lines 5–6).
We can conclude that modeling the context on the source side is much more important than on the target side.
This is expected for political debates, which are conversational in nature.
In contrast, the target side is a well-written journalistic article, where sentences are much more self-contained.
Thus, features from the source side (i.e., from the debate) are more useful as can be seen in Table 4.
Comparison to Previous Work As mentioned
above, our baseline is a reimplementation of the best system of Shaar et al.
(2020), and our context modeling adds additional components on top of it.
Note, however, that our results are not directly comparable to their work, as we use a more realistic and also a much harder setup, where the data is split by entire debates and also chronologically, i.e., training on the data from 2012 to 2018 and testing on 2019 (while they split all debates into sentences and randomly distribute them to training/testing).
Conclusion and Future Work
We have presented our work on the problem of detecting previously fact-checked claims in political debates.
In particular, we studied the impact of modeling the context of the claim: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking explanation document.
We did this by modeling the local context, the global context, as well as by means of co-reference resolution, and reasoning over the target text using Transformer-XH.
The experimental results have shown that each of these represents a valuable information source, however, modeling the source-side context is more important, and can yield 10+ points of absolute improvement.
In future work, we plan to experiment with other language models, and also to apply our approach to other domains and languages, and tasks.
Acknowledgments
This research is part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of “fake news,” propaganda, and media bias by making users aware of what they are reading.
Ethics and Broader Impact Biases
We note that there might be some biases in the data we use, as well as in some judgments for claim matching.
These biases, in turn, will likely be exacerbated by the unsupervised models trained on them.
This is beyond our control, as the potential biases in pre-trained large-scale transformers such as BERT and RoBERTa, which we use in our experiments.
Intended Use and Misuse Potential
Our models can make it possible to put politicians on the spot in real time, e.g., during an interview or a political debate, by providing journalists with tools to do trustable fact-checking in real time.
They can also save a lot of time to fact-checkers for unnecessary double-checking something that was already factchecked.
However, these models could also be misused by malicious actors.
We, therefore, ask researchers to exercise caution.
Environmental Impact
We would also like to warn that the use of large-scale Transformers requires a lot of computations and the use of
GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019).
This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we fine-tune them on relatively small datasets.
Moreover, running on a
CPU for inference, once the model is fine-tuned, is perfectly feasible, and CPUs contribute much less to global warming.