Detecting Rumor Veracity with Only Textual Information by Double-Channel Structure (2022)

Abstract: Kyle (1985) proposes two types of rumors: informed rumors which are based on some private information and uninformed rumors which are not based on any information (i.e. bluffing). Also, prior studies find that when people have credible source of information, they are likely to use a more confident textual tone in their spreading of rumors. Motivated by these theoretical findings, we propose a double-channel structure to determine the ex-ante veracity of rumors on social media. Our ultimate goal is to classify each rumor into true, false, or unverifiable category. We first assign each text into either certain (informed rumor) or uncertain (uninformed rumor) category. Then, we apply lie detection algorithm to informed rumors and thread-reply agreement detection algorithm to uninformed rumors. Using the dataset of SemEval 2019 Task 7, which requires ex-ante threefold classification (true, false, or unverifiable) of social media rumors, our model yields a macro-F1 score of 0.4027, outperforming all the baseline models and the second-place winner (Gorrell et al., 2019). Furthermore, we empirically validate that the double-channel structure outperforms single-channel structures which use either lie detection or agreement detection algorithm to all posts.

## introduction detecting the veracity of rumors spreading out on various social media platforms has been of great importance. indeed, several studies find that online rumors can affect human behaviors @xcite . however, detecting the veracity of rumors is not a simple task. unlike news articles which are considered ex-post, rumors are ex-ante @xcite . at the time when a rumor originates, the information user is not able to determine its veracity by checking whether the event has happened or not. instead, the user can make his best guess based on the information set that he has been exposed to. in contrast, we can check the veracity of a news article immediately by comparing it with the event that the article is referring to @xcite . there can be diverse definitions of rumors, but in our study we define the rumors as "information that cannot be verified at the time of origination @xcite ". 2 therefore, whether a rumor is false or not can only be determined afterward when the user can objectively observe the event @xcite . in our research, we use only the textual features of the posts and their corresponding replies, mitigating the concern that our results are driven by external information that was not readily available to the general public at the early stage of rumor origination. also, our model shows that textual features embedded in social media posts can reasonably predict the ex-ante veracity of rumors. @xcite provides a theoretical model that explains the motivation of spreading rumors. the model includes two types of rumor spreading: (i) rumors based on private information and (ii) rumors not based on any information (i.e. bluffing). spreaders with private information can either deliver the correct information that they have or intentionally distort the information. on the other hand, there can be spreaders without private information. they take advantage of their social influence and spread some made-up rumors in favor of their benefits @xcite . refer to figure 1 for the visual representation of rumor classification. studies on linguistics find that the perceived credibility of information source affects the tone of rumors on social media @xcite @xcite . the more credible the information source is, the more confident the textual tone is. for instance, rumors based on concrete source of information are likely to include a reference link or refer to specific identities. in contrast, bluffing is less likely to encompass the source of information. combining these two lines of literature, rumors based on private information and rumors not based on private information are systematically and linguistically different. however, prior studies that intent to identify the "ex-ante" veracity of social media rumors (e.g. @xcite @xcite treat every rumor equally. in other words, they apply the same logic or algorithm to both types of rumors. to tackle this issue, we conjecture that dividing the sample into "informed rumors" (rumors that are based on private information) and "uninformed rumors" (rumors that do not have any information background) and applying different algorithms to the two subgroups can improve the performance of veracity detection. motivated by the linguistic differences between the two rumor types, we first divide the sample based on the textual confidence of rumor texts. this algorithm classifies each rumor into certain (informed rumors) or uncertain (uninformed rumors) category. as in @xcite , informed spreaders can strategically choose whether or not to truthfully report the private information that they have. if they choose to distort the information, the spreaders are intentionally lying. in contrast, they might opt for truth-telling. therefore, we apply the lie detection algorithm to informed rumors to determine their ex-ante veracity. on the other hand, for uninformed rumors, the spreaders are not intentionally lying nor are they truthfully reporting. therefore, we do not expect lie detection algorithm to function properly. instead, we rely on the agreement detector algorithm @xcite . prior literature finds that when primary replies are generally in accordance with the original thread, the thread is likely to be true ex-post, and vice-versa @xcite . in our model, we use primary replies and calculate their agreement scores with the main thread. the logic beyond this algorithm is that the wisdom of the crowd plays a role in social media platforms to provide accurate information @xcite . we leave the mathematical details for sections 3.1 and 3.2. in our study, we further validate this theorymotivated double-channel approach by showing that our model outperforms the single channel structures (applying lie detection algorithm or agreement detection algorithm to both channels). section 4.1 outlines the relative performance of doublechannel model compared with other structures and with other competing models of semeval 2019. specifically, our model achieves a macro-f1 score of 0.4027, which is approximately 12% points higher than that of the second-place winner. figure 2 provides an example of the classification results of our model. the uninformed thread does not refer to any source information while the informed one does so. lie detection algorithm correctly classifies the veracity of the informed rumor. on the other hand, agreement detector captures whether each primary reply is in accordance with the main thread. the algorithm correctly classifies the thread to be false. our research contributes to the existing line of literature for at least two reasons. first, we are the first to employ a double-channel model to detect the veracity of rumors. this approach reflects the rumor classification (informed and uninformed) proposed by the linguistics literature. we show that the lie detection algorithm is relatively more appropriate for classifying informed rumors and that the agreement detection is more accurate when classifying uninformed rumors. after employing a bert-based certainty classifier to divide the samples into two subgroups, we find a significant increase in our classification accuracy. second, we also use minimal information to obtain our results. our f1 score falls behind the winner of semeval 2019 task 7, primarily due to the scope of the information that we use. the winner exploits a variety of peripheral information such as the account credibility or the number of followers @xcite , which explains a great portion of their results. however, such a model cannot be applied to anonymous rumors or rumors posted by relatively "new" users. in contrast, our model operates even without considering the peripheral or user-specific information, allowing it be applied to even anonymous rumors in social media. also, since the second-place winner primarily focuses on the textual dimension of twitter posts, we find the second-place winner more comparable to our assumptions and experiments. ## results we present our results in ## conclusion perfectly determining the veracity of rumors at the time of their origination is impossible. nonetheless, an increasing number of rumors are spreading out via social media, and people are affected by those rumors. therefore, sorting out the "likelyfraudulent" rumors at their early stage is of great importance to information users. our model takes minimal textual information and achieves a reasonable prediction accuracy in the semeval 2019 task 7 dataset. this dataset contains only 365 train samples and 81 test samples, but requires three-way classification. we achieve the macro-f1 score of 0.4027 in this task, which is approximately 12% points higher than that of the second-place winner which also focuses on the textual features of posts. instead of integrating a wide variety of userspecific information, our model shows that textual features have sufficient predictive power in determining the veracity of rumors. more importantly, we demonstrate that applying a uniform classifier to all twitter and reddit posts can harm the model's performance. instead, we apply a double-channel approach in rumor veracity detection. we divide the sample into two subgroups depending on the textual certainty and apply two different classifiers to each subgroup. also, by using only textual features of a post and its primary replies, this study responds to @xcite 's call for research that enables the early detection of rumor veracity. our research can be successfully implemented in the real world setting. our model, which does not rely on user-specific information (e.g. the number of followers, the number of previous posts, etc.), can even be implemented to determine the veracity of anonymous rumors. the model produces a rapid veracity prediction. that is, we can produce the results almost immediately for informed rumors and within several days for uninformed rumors. ul-timately, providing users with predicted veracity information can help their potential decision making.