A ra S tance: A Multi-Country and Multi-Domain Dataset of A rabic Stance Detection for Fact Checking (2021)

Abstract: With the continuing spread of misinformation and disinformation online, it is of increasing importance to develop combating mechanisms at scale in the form of automated systems that support multiple languages. One task of interest is claim veracity prediction, which can be addressed using stance detection with respect to relevant documents retrieved online. To this end, we present our new Arabic Stance Detection dataset (AraStance) of 4,063 claim–article pairs from a diverse set of sources comprising three fact-checking websites and one news website. AraStance covers false and true claims from multiple domains (e.g., politics, sports, health) and several Arab countries, and it is well-balanced between related and unrelated documents with respect to the claims. We benchmark AraStance, along with two other stance detection datasets, using a number of BERT-based models. Our best model achieves an accuracy of 85% and a macro F1 score of 78%, which leaves room for improvement and reflects the challenging nature of AraStance and the task of stance detection in general.

## introduction the proliferation of social media has made it possible for individuals and groups to share information quickly. while this is useful in many situations such as emergencies, where disaster management efforts can make use of shared information to allocate resources, this evolution can also be dangerous, e.g., when the news shared is not precise or is even intentionally misleading. polarization in different communities further aggravates the problem, causing individuals and groups to believe and to disseminate information without necessarily verifying its veracity (misinformation) or even making up stories that support their world views (disinformation). these circumstances motivate a need to develop tools for detecting fake news online, including for a region with opposing forces and ongoing conflicts such as the arab world. our work here contributes to these efforts a new dataset and baseline results on it. in particular, we create a new dataset for stance detection of claims collected from a number of websites covering different domains such as politics, health, and economics. the websites cover several arab countries, which enables wider applicability of our dataset. this compares favorably to previous work for arabic stance detection such as the work of @xcite , who focused on a single country. we use the websites as our source to collect true and false claims, and we carefully crawl web articles related to these claims. using the claim-article pairs, we then manually assign stance labels to the articles. by stance we mean whether an article agrees, disagrees, discusses a claim or it is just unrelated. this allows us to exploit the resulting dataset to build models that automatically identify the stance with respect to a given claim, which is an important component of fact-checking and fake news detection systems. to develop these models, we resort to transfer learning by fine-tuning language models on our labeled dataset. we also benchmark our models on two existing datasets for arabic stance detection. finally, we make our dataset publicly available. 1 our contributions can be summarized as follows: ## related work stance detection started as a standalone task, unrelated to fact-checking @xcite . one type of stance models the relation (e.g., for, against, neutral) of a text segment towards a topic, usually a controversial one such as abortion or gun control @xcite . another one models the relation (e.g., agree, disagree, discuss, unrelated) between two pieces of text @xcite . the latter definition is used in automatic fact-checking, fake news detection, and rumour verification @xcite . there are several english datasets that model fact-checking as a stance detection task on text from multiple genres such as wikipedia @xcite , news articles @xcite , and social media @xcite . most related to our work here is the fake news challenge, or fnc, @xcite , which is built by randomly matching claim-article pairs from the emergent dataset @xcite , which itself pairs 300 claims to 2,500 articles. in fnc, this pairing is done at random, and it yielded a large number of unrelated claimarticle pairs. there are several approaches attempting to predict the stance on the fnc dataset using lstms, memory networks, and transformers @xcite @xcite @xcite . there are two datasets for arabic stance detection with respect to claims. the first one collected their false claims from a single political source @xcite ), while we cover three sources from multiple countries and topics. they retrieved relevant documents and annotated the claim-article pairs using the four labels listed earlier (i.e., agree, disagree, discuss, unrelated). they also annotated "rationales," which are segments in the articles where the stance is most strongly expressed. the other arabic dataset by @xcite uses headlines from news sources and generated true and false claims by modifying the headlines. they used a three-class labeling scheme of stance by merging the discuss and the unrelated classes in one class called other. our work is also related to detecting machinegenerated and manipulated text @xcite . ## arastance construction we constructed our arastance dataset similarly to the way this was done for the english fake news challenge (fnc) dataset @xcite and for the arabic dataset of @xcite . our dataset contains true and false claims, where each claim is paired with one or more documents. each claim-article pair has a stance label: agree, disagree, discuss, or unrelated. below, we decribe the three steps of building arastance: (i) claim collection and pre-processing, (ii) relevant document retrieval, and (iii) stance annotations. ## results the evaluation results are shown in tables 5 and 6 for the development and for the test sets, respectively. we use accuracy and macro-f1 to account for the different class distributions; we also report per-class f1 scores. note that khouja (2020) uses three labels rather than four, merging discuss and unrelated into other. their label distribution has a majority of disagree, followed by agree, and very few instances of other, which is different from our dataset and from @xcite 's. we can see that arbert yields the best overall and per-class performance on dev for the khouja (2020) dataset and arastance. it also generalizes very well to the test sets, where it even achieved a higher macro-f1 score for the khouja (2020) dataset. the performance of the other three models (mbert, arabicbert, and marbert) drops slightly on the test set compared to dev for both arastance and the khouja (2020) dataset. this might be due to arbert being pre-trained on more suitable data, which includes books, gigaword and common crawl data primarily from msa, but also a small amount of egyptian arabic. since half of our data comes from an egyptian website (dabegad), this could be helpful. indeed, while arabicbert is pretrained on slightly more data than arbert, it was almost exclusively pretrained on msa, without dialectal data, and arastance it performs worse. about the other models: the datasets on which arabicbert was trained have duplicates, which could explain the model being outperformed. for marbert, it is pretrained on tweets that have both msa and dialectal arabic. marbert's data come from social media, which is different from the news articles or titles from which all the experimental downstream three datasets are derived. also, it seems that arbert and marbert are better than the other two models at predicting the stance between a pair of sentences, as it is the case with the khouja (2020) dataset. this could be due to the diversity of their pretraining data, which improves the model's ability to capture inter-sentence relations such as paraphrases and contradictions. another factor that could explain arbert's better performace compared to marbert is that the latter is trained with a masking objective only, while arbert is trained with both a masking objective and a next sentence prediction objective. the use of the latter objective by arbert could explain its ability to capture information in our claim-stance pairs, although these pairs are different from other types of pairs such as in the question and answer task, where the pair occurs in an extended piece of text. on the other hand, there is no consistently best model for the @xcite dataset. this could be due to a number of reasons. first, that dataset has a severe class imbalance, as we have explained in section 4. second, the dataset (especially the false claims) is derived from one particular domain, i.e., the syrian war, which might not be well represented in the pretraining data. therefore, additional modeling considerations such as adaptive pretraining on a relevant unlabelled corpus before fine-tuning on the target labeled data could help. surprisingly, arabicbert and arbert perform much better on the test set than on the development set of the @xcite dataset for the disagree class, which has the lowest frequency: from 0.14 f1 to 0.29-0.35 f1. since the number of disagree instances is very low (25 documents for 10-12 unique claims), it is possible that the claims in the test set happen to be more similar to the ones in the training data than it is for development. this is plausible because we did our train-dev-test split based on the five-folds prepared by the authors as explained in section 4. it is worth noting that the multilingual model (mbert) has the highest overall accuracy and f1 score for the unrelated class of the @xcite dataset. multilingual text representations such as mbert might over-predict from the majority class, and thus would perform poorly on the two low-frequency classes; indeed, mbert has an f1-score of 0 for disagree, and no more than 0.12 for discuss on development and testing. finally, we observe very high performance for all models for the unrelated class of arastance. this could be an indication of strong signals that differentiate the related and the unrelated classes, whereas the discuss class is the most challenging one in arastance, due to its strong resemblance to agree in some examples such as the one shown in ## conclusion and future work we presented arastance, a new multi-topic arabic stance detection dataset with claims extracted from multiple fact-checking sources across three countries and one news source. we discussed the process of data collection and approaches to overcome challenges in related document retrieval for claims with low online presence, e.g., due to topic or country specificity. we further experimented with four bert-based models and two additional arabic stance detection datasets. in future work, we want to further investigate the differences between the three arabic stance detection datasets and to make attempts to mitigate the impact of class imbalance, e.g., by training with weighted loss, by upsampling or downsampling the classes, etc. we further want to examine the discuss class across datasets and to compare the choice of annotation scheme -three-way vs. four-way-on this task. moreover, we plan to enrich arastance by collecting more true claims from other websites, thus creating a dataset that would be more evenly distributed across the claim veracity labels. furthermore, we would like to investigate approaches for improving stance detection by extracting the parts of the documents that contain the main stance rather than truncating the documents after the first 512 tokens. finally, we plan to experiment with cross-domain @xcite and crosslanguage approaches @xcite .