STANKER : Stacking Network based on Level-grained Attention-masked BERT for Rumor Detection on Social Media (2021)

Abstract: Rumor detection on social media puts pre-trained language models (LMs), such as BERT, and auxiliary features, such as comments, into use. However, on the one hand, rumor detection datasets in Chinese companies with comments are rare; on the other hand, intensive interaction of attention on Transformer-based models like BERT may hinder performance improvement. To alleviate these problems, we build a new Chinese microblog dataset named Weibo20 by collecting posts and associated comments from Sina Weibo and propose a new ensemble named STANKER (Stacking neTwork bAsed-on atteNtion-masKed BERT). STANKER adopts two level-grained attention-masked BERT (LGAM-BERT) models as base encoders. Unlike the original BERT, our new LGAM-BERT model takes comments as important auxiliary features and masks co-attention between posts and comments on lower-layers. Experiments on Weibo20 and three existing social media datasets showed that STANKER outperformed all compared models, especially beating the old state-of-the-art on Weibo dataset.

## introduction social media like sina weibo is an indispensable part of life, while rumors have severe consequences in political decision-making or manipulating public opinions @xcite . therefore, evil-doers can create and spread rumors on social media conveniently on a massive scale at a low cost @xcite , which provokes a text classification task called rumor detection @xcite . for text classification tasks, including rumor detection, transformer-based models like bidirectional encoder representations from transform-ers (bert) @xcite , which achieved impressive results, have a significant performance variation when fine-tuned on small datasets @xcite . thus researchers proposed ensembles of multiple bert models @xcite @xcite to provide more robust predictions, but a big ensemble size makes the fine-tuning computationally expensive, for the training time and the inference time increase linearly with the ensemble size. moreover, the attention mechanism, which is a key of transformer @xcite , makes the computational complexity scale quadratically with the input sequence length. it facilitates complex models to learn the contextual representation of a word via attending to other words. encouragingly, a few studies indicated that not all attention is necessary @xcite : partial attention can be pruned @xcite or masked @xcite depending on specific tasks, because bert learns different features at different levels. apart from the computation cost, the input length limitation is another obstacle in using pretrained language models (lms) to detect rumors. the content of social media posts is text shorter than 140 words with rich auxiliary features, e.g., comments and user profiles. among these features, comments are semantically relevant to a source post and support or deny the original claim @xcite . however, social media posts often have comments whose total length exceeds the input-length limitation of lms, demanding pre-processing like truncation. unfortunately, as the classical pre-processing for inputting long texts into lms, truncation discards valuable information in the truncated part 2 . meanwhile, although longformer @xcite was proposed recently to tackle long input sequences, excessive attention interactions may degrade the overall performance. to alleviate these problems, we propose the stanker (stacking network based-on attention-masked bert), which adopts two level-grained attention-masked bert models as base encoders, stacked with a final dense prediction layer with softmax activation that maps the 768-dimensional vectors to two outputs for this binary classification. contributions of this paper: lgam-bert), which masks insignificant co-attention between source posts and comments on lower-layers. ## problem statement let @xmath0, s 2 , ..., s |s| } be a set of source posts. each s @xmath1, w i 2 , ......w i l i >, given l i as the length of s i . each s @xmath2. replies) like s i , each c i @xmath3. each s i is also associated with a binary label y @xmath4, 1} to represent its truthfulness, where y @xmath5. suppose the dataset is symbolized as @xmath6, d 2 , ..., d |d| } where each d @xmath7, c i , y i }. given d i , our goal is to predict the truthfulness y i of source post s i , i.e., binary classification. due to the nature of social media, we regard s i as primary data and c i as auxiliary data.