Stanceosaurus: Classifying Stance Towards Multicultural Misinformation (2022)

Abstract: We present Stanceosaurus, a new corpus of 28,033 tweets in English, Hindi and Arabic annotated with stance towards 250 misinformation claims. As far as we are aware, it is the largest corpus annotated with stance towards misinformation claims. The claims in Stanceosaurus originate from 15 fact-checking sources that cover diverse geographical regions and cultures. Unlike existing stance datasets, we introduce a more fine-grained 5-class labeling strategy with additional subcategories to distinguish implicit stance. Pre-trained transformer-based stance classifiers that are fine-tuned on our corpus show good generalization on unseen claims and regional claims from countries outside the training data. Cross-lingual experiments demonstrate Stanceosaurus’ capability of training multilingual models, achieving 53.1 F1 on Hindi and 50.4 F1 on Arabic without any target-language fine-tuning. Finally, we show how a domain adaptation method can be used to improve performance on Stanceosaurus using additional RumourEval-2019 data. We will make Stanceosaurus publicly available to the research community upon publication and hope it will encourage further work on misinformation identification across languages and cultures.

## introduction the prevalence of misinformation on online social media has become an increasingly severe societal problem. a key language technology, which has the potential to help content moderators identify rapidly-spreading misinformation, is the automatic identification of both affective and epistemic stance @xcite towards false claims. progress on the problem of stance identification has largely been driven by the availability of annotated corpora, such as rumoureval @xcite . however, 1 our code and data are available at https://tinyurl. com/stanceosaurus translation: out of 16 priests of tirupati temple, 128 kg gold, 150 crores cash, 77 crores diamonds were found from income tax raid at the house of one priest. translation: these are the priests of our country who have immense wealth. existing corpora mostly focus on misinformation spreading within western countries. in this paper, we present stanceosaurus, a diverse and high-quality corpus that builds on the best design choices made in previous misinformation corpora, including rumoureval-2019 and covidlies. stanceosaurus covers more diverse topics, geographic regions, and cultures than prior work. it includes 28,033 tweets in english, hindi, and arabic that are manually annotated for stance (see figure 1 ) towards 251 misinformation claims, collected from 15 independent fact-checking websites that cover india, singapore, australia, new zealand, canada, the united states, europe, and the arab world (the levantine, gulf, northwest african regions, and egypt). to the best of our knowledge, stanceosaurus is the largest and most diverse annotated stance dataset to date. through extensive experiments, we demonstrate that stanceosaurus can support the fine-grained dataset target number/range of topics semeval-2016 @xcite subject 6 political topics (e.g., atheism, feminist movement) srq @xcite subject 4 political topics & events (e.g., general terms, student marches) catalonia @xcite subject 1 topic (i.e., catalonia independence) covid @xcite subject 4 topic related to covid-19 (e.g., stay at home orders) multi-target @xcite entity 3 pairs of candidates in 2016 us election wtwt @xcite event 5 merger and acquisition events rumoureval @xcite tweet 8 news events + rumors about natural disasters rumor-has-it @xcite claim 5 rumors (e.g., sarah palin getting divorced?) covidlies @xcite claim 86 pieces of covid-19 misinformation stanceosaurus (this work) claim 251 claims over a diverse set of global and regional topics ## the stanceosaurus corpus our corpus consists of social media posts manually annotated for stance toward claims from multiple fact-checking websites across the world. we carefully designed the data collection and annotation scheme to ensure better quality and coverage, improving upon prior work. ## automatic stance detection we design multiple automatic stance identification experiments to test the generalization capabilities of models trained on stanceosaurus. first, we establish the baseline performance of predicting stance towards unseen claim using fine-tuned transformer models in §5.1 and experiment with the class-balanced focal loss for addressing the imbalanced class distribution. we present zeroshort cross-lingual experiments in §5.2, where multilingual models are trained on english tweets and evaluated on the hindi and arabic tweets. furthermore, we demonstrate that a simple domain adaptation method can help improve performance on stanceosaurus using additional rumoureval data in §5.3. finally, we show that models trained only on international claims subset can extrapolate well to regional claims from individual countries in §5.4. ## experiments and results we report average results over five random seeds primarily by macro f1, which has been used as the standard metric in stance classification since the arguably more important stances (i.e., refuting and supporting) only consist of a small portion of data. ## conclusion we introduce stanceosaurus, a new corpus of 28,033 social media messages annotated with stance towards 251 misinformation claims originating from 15 multicultural fact-checking sources. to the best of our knowledge, stanceosaurus is the largest stance dataset yet. stanceosaurus contains consistent annotations across claims and languages, and stance classifier models trained on our dataset can perform well on unseen claims and languages. our experiments demonstrate that class-balanced focal loss consistently improves upon cross-entropy loss in addressing the stance label-imbalance issue. furthermore, the domain adaptation experiments with easyadapt show it is possible to utilize ru-moureval data to achieve even better performance on stanceosaurus despite significant differences in their data collection strategies. our work represents a step towards the development of accurate models that can track the spread of misinformation online across diverse languages and cultures.