Mawqif: A Multi-label A rabic Dataset for Target-specific Stance Detection (2022)

Abstract: Social media platforms are becoming inherent parts of people’s daily life to express opinions and stances toward topics of varying polarities. Stance detection determines the viewpoint expressed in a text toward a target. While communication on social media (e.g., Twitter) takes place in more than 40 languages, the majority of stance detection research has been focused on English. Although some efforts have recently been made to develop stance detection datasets in other languages, no similar efforts seem to have considered the Arabic language. In this paper, we present Mawqif, the first Arabic dataset for target-specific stance detection, composed of 4,121 tweets annotated with stance, sentiment, and sarcasm polarities. Mawqif, as a multi-label dataset, can provide more opportunities for studying the interaction between different opinion dimensions and evaluating a multi-task model. We provide a detailed description of the dataset, present an analysis of the produced annotation, and evaluate four BERT-based models on it. Our best model achieves a macro-F1 of 78.89%, which shows that there is ample room for improvement on this challenging task. We publicly release our dataset, the annotation guidelines, and the code of the experiments.

## introduction currently, online forums and social media platforms are being inherent parts of people's daily life as a media of expressing their stances toward different targets (e.g., events, politics, services, or controversial news). consequently, the demand for automatic solutions for stance detection significantly increases as the volume of unstructured data does. stance detection is the task of predicting whether the author of a written text is in favor of, against, or neutral toward a subject of interest (i.e., target), in which the stance is explicitly or implicitly stated in the text @xcite . automatic and high-performance solutions for stance detection can play a valuable role in decision-making for politicians, businesses, and authorities. the input to the stance detector is usually a pair of written text and a target. however, other inputs can be used to boost the model performance such as the user's social activity on the social media platforms (e.g., retweets and likes). existing stance detection datasets can be categorized based on the target dependency into targetspecific, cross-target, and target-independent. in target-specific stance detection, a specific target (e.g., donald trump or brexit referendum) has to be given along with the user's text, and sometimes the user's information, in order to detect the stance toward the predefined target. in cross-target stance detection, the objective is to build a classifier that can transfer the learned knowledge between targets using a large dataset that comprise a wider range of different targets. in the target-specific and cross-target tasks, the target of the stance is an explicit entity (e.g., person, event, or controversial issue), whereas the target in target-independent tasks is a claim or a piece of fake news and the objective is to detect whether the comments are confirming the claim/news or denying its veracity. a significant number of stance detection techniques have been proposed in the literature. however, most of these studies used an old public dataset, @xcite , including those published recently @xcite @xcite @xcite . we believe that more benchmarked stance detection datasets should be released under a common open license for public usage. non-english data, multilingual data, and annotations of other opinion dimensions (e.g., sarcasm and emotions) should all be considered for establishing new stance detection datasets. we aim to facilitate the research on targetspecific stance detection of arabic micro-blogs. to our knowledge, this problem has not been studied for the arabic language and there is no publicly available dataset for arabic that can be used for target-specific stance detection. arabic is a challenging language for most natural language processing (nlp) applications due to its unique nature in the variety of dialectics and its rich and complex morphology @xcite . furthermore, different from media that use modern standard arabic (msa) with formal linguistic criteria, social media texts represent dialectal arabic and contain an informal writing style (e.g., spelling errors, abbreviations, irregular grammar, emojis, and symbols). thus, automatically detecting the user's stance on social media, specifically in arabic, is a worthwhile and challenging task. in addition, the increase of arabic content on social media, and the mobilized masses for political and economic changes in the middle east have motivated us to search in this direction. in this paper, we release mawqif, the first arabic dataset that can be used for target-specific stance detection. this dataset consists of 4,121 tweets in multi-dialectal arabic. each tweet is annotated with a stance toward one of three targets: "covid-19 vaccine," "digital transformation," and "women empowerment." in addition, this is a multilabel dataset where each data point is annotated for stance, sentiment, and sarcasm, which will provide a benchmark for the three tasks. it will also help in analyzing the interaction between the different opinion dimensions (i.e., stance, sentiment, and sarcasm). our contributions in this paper can, therefore, be summarized as follows. 1) we construct and release mawqif, the first multi-label arabic dataset for stance detection. the proposed dataset consists of 4,121 tweets covering three topics (i.e., targets) that are controversial in the middle east. we also provide a detailed description of the dataset and an analysis of the produced annotation; 2) the proposed dataset is annotated for stance, sentiment, and sarcasm. this provides more opportunities for studying the interaction between different opinion dimensions, and evaluating a model trained on different opinion dimensions in a multi-task paradigm to boost the performance of stance detection; 3) we benchmark the proposed dataset on the stance detection task and evaluate the performance of four bert-based models. ## related work stance detection is a relatively new field of study; however, considerable effort has been devoted into building datasets for stance detection tasks. from the definitions of the three stance detection tasks (presented in section 1); the structure of the datasets used for target-independent tasks is different than the datasets used for target-specific or cross-target tasks. in target-independent stance detection, each input entry is usually in the form of a pair of textual claims and responses. examples of target-independent datasets are: emergent @xcite , ibm debater @xcite , pheme @xcite , rumoureval-17 @xcite , fnc-1 @xcite , args.me @xcite , perspectrum @xcite , rumoureval-19 @xcite , arabic news stance @xcite , and @xcite . meanwhile, the input entry for target-specific and cross-target stance detection systems usually consists of a text and target pair. several datasets have been proposed for targetspecific and cross-target stance detection. these datasets have been collected from different platforms such as social media @xcite @xcite @xcite @xcite @xcite @xcite , debate websites @xcite @xcite , and news commentaries @xcite . with regard to language orientation, most of the available stance detection datasets are monolingual where their data are available in one language. the majority of these monolingual datasets are in english language @xcite @xcite @xcite @xcite . for italian, @xcite and @xcite collected tweets targeting the italian constitutional reform and the sardines movement, respectively. similarly, @xcite collected turkish tweets targeting football clubs. furthermore, a dataset for chinese language is presented in @xcite , and a czech stance detection dataset is presented in @xcite . however, few datasets are multilingual where more than one language is considered in collecting the data. vamvas and sennrich (2020) proposed a multilingual dataset with french, german, and italian languages. two other datasets considered catalan and spanish languages in one dataset @xcite . in our dataset, we attempt to address two gaps; the language and the annotation of other opinion dimensions. despite the growing interest in studying stance detection, no study, as far as we know, considered arabic language for target-specific stance detection. in this paper, we release the first arabic target-specific stance detection dataset. it is worthwhile noting that there are two stance detection datasets that target arabic language @xcite . however, these two datasets are dedicated to study claim verification, as they consist of claim/reference pairs to predict the stance of a claim toward the reference sentence. thus, they cannot be used for building a targetspecific stance detection model. in addition, the two datasets are comprising texts in modern standard arabic, which is not the language used in social media debates where dialectal arabic is quite prevalent. moreover, most of the existing datasets annotated each text with stance labels (favor, against, none). other studies considered the sentiment polarity during data annotation. the aim of involving sentiment annotation was to analyze the interaction between stance and sentiment in order to boost the performance of stance detection @xcite . however, there is no study to the best of our knowledge has considered sarcasm features for stance detection. according to the findings of a comparative empirical study by @xcite , the main source of misclassification in stance detection is texts with sarcastic content. therefore, studying sarcasm could be beneficial for improving the performance of stance detection models. we thus proposed to annotate our dataset with sarcasm in addition to stance and sentiment polarities. our dataset is established in order to create a novel arabic linguistic resource for stance, sentiment, and sarcasm. ## mawqif dataset in this section, we explain the procedure followed to collect a set of opinions (texts) toward selected targets for stance detection. we also present the crowdsourcing setup used for stance annotation and discuss the statistics of the proposed dataset. ## benchmark experiments in this section, we present benchmarking experiments performed on the target-specific stance detection task. as mentioned earlier, the main purpose of mawqif dataset is stance detection. therefore, we considered only the stance detection task for the benchmark experiments. however, the sentiment and sarcasm annotations could be used in further experiments (i.e, future studies) to analyze the interaction between the three dimensions. models bert-based models have been shown to be effective in a variety of text classification tasks @xcite , including dialectical arabic text @xcite . thus, we chose to develop a bert-based classifier that we fine-tuned for targetspecific stance detection. specifically, we finetuned the following four bert-based models for stance detection: 1. camelbert-da, is a bert-based model trained on 5.8 billion tokens from the dialectal arabic (da) dataset @xcite . 2. marbert, is a bert-based model trained on 15.6 billion tokens from 1 billion arabic tweets (abdul-mageed et al., 2020). 3. arabert, is trained on 8.6 billion tokens from five datasets consisting of modern standard arabic (msa) text (antoun et al., 2020). 4. arabert-twitter, is trained by extending the training of arabert (v0.2) on 60 million arabic tweets (antoun et al., 2020 ). we fine-tuned the four pre-trained models and built a standard pipeline under the pytorch lightning framework. the fine-tuning code is available online along with our dataset. the proposed system starts by preprocessing the arabic texts by removing diacritics, tatweel, non-arabic letters, and repeated characters. then, a wordpiece @xcite tokenizer is used to split the input text into tokens compatible with bert-based models. for classification, the hidden representation of the @xcite token is fed into a feed-forward layer along with a softmax function. we set the maximum sequence length to 128 tokens, and the batch size to 32. each of the four models is fine-tuned for 20 epochs; adamw optimizer @xcite is used with a learning rate of 2e-5. the hyper-parameters used in these experiments have been selected empirically. evaluation metrics we evaluated our baseline models using f avg2 and f avg3 scores. f avg2 is the macro-average f1 over the "favor" and "against" stance labels (the "none" class was ignored since it was scarcely in the data). this score is computed as follows: we selected f avg2 metric to align with other stance detection datasets that report their results using f avg2 metric @xcite . we are also reporting our results using f avg3 that considers all stances and it is computed as follows: results tables 4 and 5 present the obtained results of the proposed models with the development and test sets, respectively. the development set was obtained by dividing the training set into 5folds and training the model with cross-validation. as shown in tables 4 and 5 , arabert-twitter model yields the best overall and per-target performance. this can be attributed to the type of the train data (i.e, dialectical arabic tweets) that were used to train arabert-twitter model, which is similar to the type of arabic tweets used in mawqif dataset. furthermore, we can observe that the best performed model (i.e. arabert-twitter) and the other three models (camelbert-da, marbert, and arabert) generalized quite well to the test data, even achieving higher accuracies and macro-f 1 scores. although marbert was trained on dialectical arabic tweets, its performance is low compared to arabert-twitter. this may be explained by the fact that marbert was trained with maskedlanguage modeling (mlm) objective only, whereas arabert was trained with both mlm and the next sentence prediction (nsp) objectives. while mlm aims to capture the relationship between words, nsp aims to understand longer-term dependencies between sentences. thus, nsp objective could improve the ability to capture more information in the sentence-stance pairs that appear in our training dataset. camelbert-da was trained on dialectical arabic data collected from social media sites and other resources. however, camelbert-da has a lower performance due to the smaller size of its training data compared to the data used to train araberttwitter. camelbert-da was trained on 5.8 billion words with a vocabulary size of 30k, while arabert-twitter was trained on 8.6 billion words with a vocabulary size of 60k in addition to 60m multi-dialect tweets. it is also noticeable in the obtained results that the performance of all models in detecting the none stance is low compared with other stances. this can be attributed to the small number of tweets with none stance used in model training. however, none is a class that is not of interest as the ultimate goal is to infer if the author of a written text is in favor of or against a specific target. on other hand, the obtained results with the favor stance were high compared with the against stance in all experimented models. this indicates that there is room for improvement in all models, where a model can benefit from the techniques that mitigate the impact of class imbalance. furthermore, we can observe from ## conclusion we introduced mawqif, the first multi-label arabic dataset for target-specific stance detection. the proposed dataset consists of 4,121 multi-dialectal arabic tweets targeting three topics that are controversial in the middle east. mawqif is not limited to stance annotation, it is further annotated with sentiment and sarcasm polarity. thus, mawqif can serve as a new benchmark for three tasks: stance detection, sentiment analysis, and sarcasm detection. in addition, it can enable future research in studying the interaction between different opinion dimensions, and evaluating multi-task models. we also presented a detailed description of the dataset and an analysis of the produced annotation. lastly, we experimented on the target-specific stance detection task and establish strong baselines based on four bert-based models. future work may improve upon the reported results by minimizing the effects of class imbalance, which can be accomplished by oversampling or undersampling techniques, or by training with weighted loss. another interesting direction for further research is developing a joint neural archi-tecture based on a multi-task learning paradigm that jointly models sentiment and sarcasm to boost the performance of stance detection. to facilitate future research, we publicly release our dataset, the annotation guidelines, and the code that can be used to reproduce the presented evaluation results.