E arly Rumor Detection Using Neural H awkes Process with a New Benchmark Dataset (2022)

Abstract: Little attention has been paid on EArly Rumor Detection (EARD), and EARD performance was evaluated inappropriately on a few datasets where the actual early-stage information is largely missing. To reverse such situation, we construct BEARD, a new Benchmark dataset for EARD, based on claims from fact-checking websites by trying to gather as many early relevant posts as possible. We also propose HEARD, a novel model based on neural Hawkes process for EARD, which can guide a generic rumor detection model to make timely, accurate and stable predictions. Experiments show that HEARD achieves effective EARD performance on two commonly used general rumor detection datasets and our BEARD dataset.

## introduction the proliferation of online rumors has aroused widespread concerns. many studies have been conducted for automatic rumor detection in social media and achieved high detection accuracy @xcite @xcite @xcite @xcite . however, these generic detection methods lack indepth modeling of temporality, which can cause tremendous delay in detection given the instantaneous and sporadic nature of rumor propagation. while a few early rumor detection (eard) models have been proposed @xcite @xcite , they have been designed with oversimplifcation and evaluated inappropriately using the datasets constructed for generic rumor detection. widely used rumor detection datasets, such as twitter @xcite and pheme @xcite , are generally limited in covering relevant posts in the early stage as there was no mechanism ensuring to gather information that is further away from the offcial debunking time of a rumor. for this reason, the generalizability of eard cannot be it is observed that the original posts in both datasets are clearly delayed as compared to our recollected posts. also, the rumor indicative patterns in the recollected posts unfold differently, where dissenting voices, a common indicator of rumor, appear much earlier and may last for many hours or even years, evolving from vaguely opposing the claim (e.g., 'like bubbles', 'someone') to frmly refuting it with evidence (e.g., 'photoshop contest'). the original "early" posts in pheme clearly fail to cover such useful patterns refecting the early dynamics, and the posts in twitter do not cover any early indicative signals before the rumor was offcially debunked by snopes. given the unavailability of eard-specifc dataset, it is necessary to construct a benchmark dataset for eard (beard) consid-ering the earliness of relevant posts to gather. meanwhile, eard methods have not been well studied either in the literature. prior works claimed as being able to do early detection can be divided into two categories, both of which are sub-optimal: 1) methods that are unable to automatically determine a time point for confrming the detection @xcite @xcite @xcite . typically, such methods apply a generic rumor detection model to report a decision of classifcation (e.g., rumor or non-rumor) at each of the pre-determined checkpoints while leaving the determination of the best detection point to human judge. this is subject to a delayed decision as the results at the later checkpoints also need to be examined. 2) methods that are trained to automatically determine an early detection point, but cannot guarantee the stability of decision @xcite . for example, ced @xcite decides an early detection point using a fxed probability threshold to assess if the current prediction is credible or not. however, prediction probability does not really refect model's confdence @xcite , and such decision without properly modeling the uncertainty beyond the decision point may fail to give a timely and reliable detection because the prediction could fip over and over again afterwards with new posts fow in. in this work, we propose a new method called hawkes early rumor detection (heard) to model the stabilization process of rumor detection based on a neural hawkes process (nhp) (mei @xcite , which can automatically determine when to make a timely and stable decision of detection. the basic idea is to construct a detection stability distribution over the expected future predictions based on a sequence of prior and current predictions, such that an optimal time point can be fxed without any delay for awaiting and checking the upcoming data beyond that point. our main contributions can be summarized as follows 1 : ## beard corpus construction we scrape the text of title, claim, debunking time and veracity label in the articles on the factchecking website snopes.com. our goals are two-fold: 1) the collected posts are not only relevant to the claim but can diversely cover copious variations of relevant text expressions; 2) the collection can cover posts of early arrival, possibly ahead of the pertinent news exposure on the mainstream media. to this end, we frstly construct high-quality search queries for twitter search. an original query is formed from the title and claim of each article, with stop words removed. since the lengthy query might harm the diversity of search results, we utilize some heuristics to obtain a substantial set of variants of each query potentially with better result coverage in twitter search: i) we preform synonym replacement to create a set of variants of the query; ii) we shorten each variant by removing its words one by one with carefully crafted rules to maintain useful information, e.g., named entities, for good search quality, while keeping the remaining words after each removal as a new variant. as a result, we obtain a substantial set of variants of the original query and merge the twitter search results of each query and all its variants. to cover early posts, each twitter search is performed in an iterative fashion. to avoid groundtruth leakage, we frst obtain the possible earliest offcial debunking time of the given claim by cross-checking its similar claims in a range of factchecking websites (see appendix a.1). from the earliest debunking time, we search backward for the relevant posts within m days prior to debunking, and then push back further n days earlier than before in each iteration until the number of newly gathered posts in an iteration becomes less than 1% of the posts obtained from the previous iteration. finally, for each retrieved post, we use its conversation id to fnd the root post of the conversation it is engaged in. we utilize sentence-bert (reimers and gurevych, 2019) to retain those root posts with cosine similarities to the claim being higher than an empirical threshold. thus far, we have obtained a set of conversation ids for each claim which are led by different root posts (see appendix a.4 for postprocessing). then we fetch from twitter all the posts in the detected conversations along with the root posts into our fnal collection as an instance, and label the conversation as rumor if the corresponding claim is from the "fact checks" category on the snopes or non-rumor if it is from the "news" category. due to space limit, we provide the details of search queries construction in appendix a.2, and the settings of iterative twitter search in appendix a.3. ## problem defnition let @xmath1, where each @xmath2, s} consists of the ground-truth label @xmath3, 1} and a set of relevant posts in chronological order each tuple (m i , τ i ) ∈ s includes the text content m i and the timestamp τ i of the i-th post, where τ i is defned as the time difference between the frst and the i-th post, such that τ 1 = 0 and τ i-1 ≤ τ i for @xmath4. in other words, τ i can be regarded as the elapsed time relative to the earliest post so that the timelines of different instances are aligned. detection update inverse count continuous-time lstm (ctlstm) ( i(𝑡 % ) 𝑡 ∈ (0, 𝑡 !"@xmath0 , 𝑡 ! ] ( . 𝐡 t evolve) no output " 𝑦 ! & stop yes recur to i+1 prediction input posts 1) time t; 2) inverse count up to t inverse count prediction stabilization process modeling standard lstm current observation historical observations lstm figure 1 : the architecture of heard. when the current observation arrives at time t i , rumor detection predictions from lstm ŷi and ŷi-1 are used to update pi count i(t i ). then the intensity function λ(t) is computed by ctlstm for @xmath5. heard will determine t i as the earliest time point with stable prediction ŷi and stop if the stability prediction î(t ∞ ) equals to i(t i ), or continue otherwise. following the pre-processing method in most prior studies @xcite @xcite , we divide each posts sequence into a sequence of intervals to avoid excessively long sequence. we chop a sequence s into intervals based on three strategies: 1) fixed posts number, 2) fixed time length and 3) variable length in each interval @xcite . hence s is converted to @xmath6, t i )} we define the eard task as automatically determining the earliest time @xmath7, such that the prediction ŷ ∈ {0, 1} at t for a given claim is accurate and remains unchanged afterwards with time goes by. it is worthwhile to mention that since t i relates to the granularity of intervals, it might affect the precision of a decision point based on the formed intervals. in practice, however, we will try to make the intervals small for keeping such impact marginal. ## heard model figure 1 shows the architecture of heard, which contains two components: 1) the rumor detection component predicts rumor/non-rumor label at each time step/interval; 2) the stabilization component models the prediction stabilization process and determines when to stop at the earliest detection point. we will describe them with detail in this section.