Cross-Topic Rumor Detection using Topic-Mixtures (2021)

Abstract: There has been much interest in rumor detection using deep learning models in recent years. A well-known limitation of deep learning models is that they tend to learn superficial patterns, which restricts their generalization ability. We find that this is also true for cross-topic rumor detection. In this paper, we propose a method inspired by the “mixture of experts” paradigm. We assume that the prediction of the rumor class label given an instance is dependent on the topic distribution of the instance. After deriving a vector representation for each topic, given an instance, we derive a “topic mixture” vector for the instance based on its topic distribution. This topic mixture is combined with the vector representation of the instance itself to make rumor predictions. Our experiments show that our proposed method can outperform two baseline debiasing methods in a cross-topic setting. In a synthetic setting when we removed topic-specific words, our method also works better than the baselines, showing that our method does not rely on superficial features.

## introduction recently, there has been much interest in detecting online false information such as rumors and fake news. existing work has explored different features including network structures @xcite , propagation paths @xcite , user credibility @xcite and the fusion of heterogeneous data such as image and text @xcite . however, these proposed algorithms still cannot be easily deployed for real-world applications, and one of the key reasons is that, just like many other nlp problems, rumor or fake news detection models may easily overfit the training data and thus cannot perform well on new data. the * corresponding author problem can be more serious with deep learning solutions, because deep neural networks tend to learn superficial patterns that are specific to the training data but do not always generalize well @xcite . in this work, we study the task of rumor detection and focus on the problem of adapting a rumor detection model trained on a set of source topics to a target topic, which we refer to as cross-topic rumor detection. in a recent study by @xcite , the authors compared the performance of rumor detection in an in-topic setting and an outof-topic setting. they found that their model could achieve 77.4% macro f-score on the in-topic testing data but the performance of the same classifier dropped to 39.5% when applied to out-of-topic testing data, which describe events different from the training events. in this paper, we propose a method inspired by the "mixture of experts" paradigm, abbreviated as "moe". understanding that the rumor prediction model may work differently for different topics, we assume that the prediction result on an instance is dependent on the topic distribution of that instance. while a standard method is to train topic-specific classifiers and then use the topic distribution to combine these topic-specific classifiers, we propose a different approach where the topic distribution is used to linearly combine a set of vectors representing different topics. this gives us a "topic-mixture" given an example. this topic-mixture vector of the example is concatenated with the vector representation of the example itself and used as the input to a neural network model for rumor label prediction. we implement our method on top of a state-ofthe-art sta-hitplan model and conduct experiments using the pheme dataset. compared with two baseline methods that also perform debiasing, we find that our method can achieve clearly better cross-topic performance. we also experiment with modified within-topic data where we intentionally remove topic-specific words. this creates a setting where it is hard for models to rely on topic-specific words to make rumor predictions. we find that our method can also outperform the baselines substantially. ## performance degradation in cross-topic rumur detection in this section, we present a case study on the pheme dataset to quantify the degree of overfitting of an existing model by analyzing the influence of topic-specific words. concretely, we use the pheme dataset, which has five topics. we use four topics during training and the remaining one for out-of-domain testing. after obtaining a trained hierarchical transformer model @xcite , we perform post-hoc testing by applying it to different topics, with k topic-specific words masked to examine the performance drop. here the topic-specific words are identified based on log-odds ratio with dirichlet prior @xcite , and we regard these topic-specific words as possible spurious patterns. it is a common way to identify words that are statistically over-represented in a particular population compared to others. for the in-domain testing, we split the data as 7:2:1 for training, testing and validation. experiments are performed using @xmath0, 50, 100, 200}. results: the partial results are shown in ## conclusion and future work in this work, we propose a new cross-topic rumor detection task base on mixture of experts, which can reinforce the generalization capacity of a model when adapting to new topics. we suggest that: 1) instead of training an unstable adversarial component or removing bias directly from semantic contents, the mixture of experts provides us with another way to increase generalization ability. 2) in this work, we use feature concatenation and train one classifier rather than several expert classifiers, and utilize a fixed confidence score. in the future, we can learn adaptive weights to make the model more flexible. for example, we could use variational inference methods to dynamically learn the best mixture of topics for a given held-out topic.