Stance Prediction and Claim Verification: An A rabic Perspective (2020)

Abstract: This work explores the application of textual entailment in news claim verification and stance prediction using a new corpus in Arabic. The publicly available corpus comes in two perspectives: a version consisting of 4,547 true and false claims and a version consisting of 3,786 pairs (claim, evidence). We describe the methodology for creating the corpus and the annotation process. Using the introduced corpus, we also develop two machine learning baselines for two proposed tasks: claim verification and stance prediction. Our best model utilizes pretraining (BERT) and achieves 76.7 F1 on the stance prediction task and 64.3 F1 on the claim verification task. Our preliminary experiments shed some light on the limits of automatic claim verification that relies on claims text only. Results hint that while the linguistic features and world knowledge learned during pretraining are useful for stance prediction, such learned representations from pretraining are insufficient for verifying claims without access to context or evidence.

## introduction although fake news is not an emerging phenomenon and has been documented throughout history, the prevalence and wide spread of misinformation over the internet has captured significant proportion of public attention in recent years. this is in part linked to the low barrier for content generation through the advent of the internet and social media @xcite and the fact that false news spread faster than true news @xcite rendering it increasingly dangerous to public discourse. the widespread exposure in the u.s. for example has been reported by researchers who found that the average american encountered between one and three stories from known publishers of fake news during the month before the 2016 election @xcite . since manual fact-checking by human experts does not scale well with the amount of information shared on the web, there is a growing body of work in recent years aimed at developing automatic tools to target fake news, misinformation and credibility of content on social media in general @xcite @xcite @xcite . several datasets were developed to further aid research on this topic foot_0 @xcite @xcite . we refer readers to @xcite for a more comprehensive overview of recent research on fake news, propaganda and misinformation. despite the increased attention, most of the work has been focusing on the english language. tools, resources and datasets available in arabic are limited @xcite @xcite . as such, this work contributes to recent efforts targeting arabic by introducing a new publicly available corpus in arabic that is suitable to study claim verification and semantic entailment @xcite . ## related work in recent years, there has been rapid progress in developing systems and tools for automatic fact checking and claim verification. various approaches were developed which relied on a diverse set of methods and information to verify claims. most relevant to this work are approaches that used content such as textual information in the title and/or body of the claims to predict their veracity. among this direction of research those that considered a machine learning approach @xcite given a news title, write two news titles that: a-paraphrase the original title: has same meaning but is worded differently by rephrasing and changing syntax, using verb synonyms, using different words to describe the same information such as locations, counts and dates. b-contradict the original title: looks similar to the original title but has contradicting meaning (both cannot be true in the same context) by reversing meaning without negating main verb, using antonym of main verb with rephrasing, changing key information using world knowledge such as locations, counts and dates. datasets: there are limited but growing datasets related to claim verification @xcite @xcite @xcite @xcite . however, datasets focusing on arabic remain scarce @xcite @xcite . recently, work on the application of textual entailment for claim verfication has been explored and new datasets combining stance prediction and claim verfication were introduced @xcite . this work is most in line with that direction. we developed a new corpus in arabic that can be used jointly for claim verification and textual entailment recognition. however, our new corpus differs from the aforementioned datasets in that it is at the sentence level, hence, we are disentangling the tasks of claim verification and textual entailment from the task of evidence extraction (information retrieval) and focusing on the former. we also start from real news titles and generate true/false claims from them. our aim is to mitigate one type of bias that results from starting with fake news collected in the wild: bias in the distribution of topics among the true/false claims. while some forms of biases about the world are useful in determining the veracity of a claim, some can be problematic. we can imagine a dataset that contains more positive 2 news in the "fake" class than in the "true" class. 2 positive here refers to sentiment a system trained on such data could predict the class "fake" with higher confidence for any claim that has a positive tone compared to one that has a negative or neutral tone. such surface level biases in topics and linguistic styles could arguably result in models that do not generalize well. ## the corpus in this part, we describe our arabic news stance (ans) corpus. 3 we derived two perspectives of the corpus suitable for claim verification and stance classification. please refer to appendix a to read our data statement about the corpus. ## experiments in this section, to demonstrate the utility of the corpus, we derive two tasks useful for evaluating news veracity and stance prediction and develop two baselines to evaluate on the proposed tasks. we describe the proposed tasks and details of the baselines in this section and the results in section 5. ## results for the recurrent perspective models, we trained all models for 100 epochs using adam optimizer (kingma and ba, 2014) with 0.001 learning rate. we conducted hyper-parameter tuning on the de- prediction label translation arabic fake fake historic agreement between europe and japan to support trump fake true historic agreement between europe and japan to confront trump true fake first women's interest channel in gaza soon to see the light true true first women's interest channel in gaza faces uncertain fate fake true ethiopia assures egypt of its nile share fake fake ethiopia apathetic about egypt's right of the nile water results for the pretraining experiments (shown in limits of claim only verification: we briefly mentioned in section 4.1 the limited scope of claim verification in a setting were the decision about the veracity of a claim can be made using only the text of the claim. we hypothesize that the task might not be learnable through a direct mapping from the claim text to the veracity space. given that the initial results of the fine-tuned bert model supported this intuition, we elected to manually inspect a sample of the predictions and noticed that in many cases the model was predicting the same label for claims that look similar but are semantically different. we share a sample of these cases in these initial experiments suggest that discriminate models trained using claim only information might rely on biases in the topics, linguistic styles, tones and implicit world-knowledge learned from training data to make predictions. results of the performance of such models could, therefore, be inflated if the training data is not uniformly distributed across languages, topics, writing styles, political ideologies etc. while we believe that our dataset collection process which yields classes that share the same distribution of topics and news sources mitigated these types of biases, we also note that the annotation process and human factor introduced other types of biases that could be present in the data. ## conclusion in this work we presented a new publicly available corpus for textual entailment and its use in studying misinformation in the arabic language. we shared some insights about the creation of the corpus and the baselines developed to evaluate the corpus. we further explored the use of pretraining @xcite and developed a strong baseline for our tasks. our experiments additionally shed light on the limits of "claim-only" misinformation detection methods that rely solely on the stated claims without use of accompanying evidence. we hope to explore this further in future work. as we plan to also explore the use of generated data in studying the robustness of misinformation detection methods against adversarial data with varying linguistic styles, political ideologies and world-knowledge. news sources (bbc, al arabiya, cnn, sky news, france24) and 6 categories (culture, economy, international news, middle east, sport, technology) collected from february 2018 to october 2018. data also includes rewritten versions of the news titles by the annotators following the provided guidelines (see