Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims (2021)

Abstract: False claims that have been previously fact-checked can still spread on social media. To mitigate their continual spread, detecting previously fact-checked claims is indispensable. Given a claim, existing works focus on providing evidence for detection by reranking candidate fact-checking articles (FC-articles) retrieved by BM25. However, these performances may be limited because they ignore the following characteristics of FC-articles: (1) claims are often quoted to describe the checked events, providing lexical information besides semantics; (2) sentence templates to introduce or debunk claims are common across articles, providing pattern information. Models that ignore the two aspects only leverage semantic relevance and may be misled by sentences that describe similar but irrelevant events. In this paper, we propose a novel reranker, MTM (Memory-enhanced Transformers for Matching) to rank FC-articles using key sentences selected with event (lexical and semantic) and pattern information. For event information, we propose a ROUGE-guided Transformer which is finetuned with regression of ROUGE. For pattern information, we generate pattern vectors for matching with sentences. By fusing event and pattern information, we select key sentences to represent an article and then predict if the article fact-checks the given claim using the claim, key sentences, and patterns. Experiments on two real-world datasets show that MTM outperforms existing methods. Human evaluation proves that MTM can capture key sentences for explanations.

## introduction social media posts with false claims have led to real-world threats on many aspects such as politics @xcite , social order @xcite , and personal health @xcite . to tackle this issue, over 300 fact-checking projects have been launched, such as snopes foot_0 and jiaozhen foot_1 (duke reporters' lab, 2020). meanwhile, automatic systems have been developed for detecting suspicious claims on social media @xcite . this is however not the end. a considerable amount of false claims continually spread, even though they are already proved false. according to a recent report @xcite , around 12% of false claims published on chinese social media, are actually "old", as they have been debunked previously. hence, detecting previously fact-checked claims is an important 5469 task. according to the seminal work by @xcite , the task is tackled by a two-stage information retrieval approach. its typical workflow is illustrated in figure 1 @xcite . given a claim as a query, in the first stage a basic searcher (e.g., @xcite searches for candidate articles from a collection of fact-checking articles (fc-articles). in the second stage, a more powerful model (e.g., @xcite reranks the candidates to provide evidence for manual or automatic detection. existing works focus on the reranking stage: vo and lee (2020) model the interactions between a claim and the whole candidate articles, while @xcite extract several semantically similar sentences from fc-articles as a proxy. nevertheless, these methods treat fcarticles as general documents and ignore characteristics of fc-articles. figure 1(b) shows three sentences from candidate articles for the given claim. among them, s1 is more friendly to semantic matching than s2 and s3 because the whole s1 focuses on describing its topic and does not contain tokens irrelevant to the given claim, e.g., "has spread over years" in s2. thus, a semantic-based model does not require to have strong filtering capability. if we use only general methods on this task, the relevant s2 and s3 may be neglected while irrelevant s1 is focused. to let the model focus on key sentences (i.e., sentences as a good proxy of article-level relevance) like s2 and s3, we need to consider two characteristics of fc-articles besides semantics: c1. claims are often quoted to describe the checked events (e.g., the underlined text in s2); c2. event-irrelevant patterns to introduce or debunk claims are common in fc-articles (e.g., bold texts in s2 and s3 ). based on the observations, we propose a novel reranker, mtm (memory-enhanced transformers for matching). the reranker identifies key sentences per article using claim-and pattern-sentence relevance, and then integrates information from the claim, key sentences, and patterns for article-level relevance prediction. in particular, regarding c1, we propose rouge-guided transformer (rot) to score claim-sentence relevance literally and semantically. as for c2, we obtain the pattern vectors by clustering the difference of sentence and claim vectors for scoring pattern-sentence relevance and store them in the pattern memory bank (pmb). the joint use of rot and pmb allows us to iden-tify key sentences that reflect the two characteristics of fc-articles. subsequently, fine-grained interactions among claims and key sentences are modeled by the multi-layer transformer and aggregated with patterns to obtain an article-level feature representation. the article feature is fed into a multi-layer perceptron (mlp) to predict the claim-article relevance. to validate the effectiveness of our method, we built the first chinese dataset for this task with 11,934 claims collected from chinese weibo foot_2 and 27,505 fact-checking articles from multiple sources. 39,178 claim-article pairs are annotated as relevant. experiments on the english dataset and the newly built chinese dataset show that mtm outperforms existing methods. further human evaluation and case studies prove that mtm finds key sentences as explanations. our main contributions are as follows: ## related work to defend against false information, researchers are mainly devoted to two threads: (1) automatic fact-checking methods mainly retrieve relevant factual information from designated sources and judge the claim's veracity. thorne et al. (2018) use wikipedia as a fact tank and build a shared task for automatic fact-checking, while popat et al. (2018b) and wang et al. (2018) retrieve webpages as evidence and use their stances on claims for veracity prediction. (2) fake news detection methods often use non-factual signals, such as styles (przybyla, 2020; qi et al., 2019), emotions (ajao et @xcite , source credibility @xcite , user response @xcite and diffusion network @xcite . however, these methods mainly aim at newly emerged claims and do not address those claims that have been fact-checked but continually spread. our work is in a new thread, detecting previously fact-checked claims. vo and lee (2020) models interaction between claims and fc-articles by combining glove @xcite and elmo embeddings @xcite . @xcite train a ranksvm with scores from bm25 and sentence-bert for relevance prediction. these methods ignore the characteristics of fc-articles, which limits the ranking performance and explainability. ## proposed method given a claim q and a candidate set of k 1 fcarticles d obtained by a standard full-text retrieval model (bm25), we aim to rerank fc-articles truly relevant w.r.t. q at the top by modeling fine-grained relevance between q and each article @xmath0. this is accomplished by memory-enhanced transformers for matching (mtm), which conceptually has two steps, (1) key sentence identification and (2) article relevance prediction, see figure 2 . for an article of l sentences, let @xmath1, ..., s l } be its sentence set. in step (1), for each sentence, we derive claim-sentence relevance score from rougeguided transformer (rot) and pattern-sentence relevance score from pattern memory bank (pmb). the scores indicate how similar the sentence is to the claim and pattern vectors, i.e., how possible to be a key sentence. top k 2 sentences are selected for more complicated interactions and aggregation with the claim and pattern vectors in step (2). the aggregated vector is used for the final prediction. we detail the components and then summarize the training procedure below. ## conclusions we propose mtm to select from fact-checked articles key sentences that introduce or debunk claims. these auto-selected sentences are exploited in an end-to-end network for estimating the relevance of the fact-checked articles w.r.t. a given claim. experiments on the public twitter dataset and the private weibo dataset show that mtm outperforms the state of the art. moreover, human evaluation and case studies demonstrate that the selected sentences provide helpful explanations of the results.