ISHFMG _ TUN at S tance E val: Ensemble Method for A rabic Stance Evaluation System (2024)

Abstract: It is essential to understand the attitude of individuals towards specific topics in Arabic language for tasks like sentiment analysis, opinion mining, and social media monitoring. However, the diversity of the linguistic characteristics of the Arabic language presents several challenges to accurately evaluate the stance. In this study, we suggest ensemble approach to tackle these challenges. Our method combines different classifiers using the voting method. Through multiple experiments, we prove the effectiveness of our method achieving significant F1-score value equal to 0.7027. Our findings contribute to promoting NLP and offer treasured enlightenment for applications like sentiment analysis, opinion mining, and social media monitoring.

## introduction nowadays, almost all people use social media for different purposes such as communication, business, and advertisement. social media platforms have revolutionized the way people interact, share information, and conduct business, creating a global community that is more interconnected than ever before. this digital transformation has led to an immense amount of user-generated content being produced daily, encompassing various languages and dialects. a large number of these users are arabic people. in fact, arabic is among the most commonly worldwide spoken languages, with over 300 million native speakers across more than 20 countries. this widespread use underscores the significance of arabic in the digital sphere and highlights the need for effective natural language processing (nlp) tools tailored to this language. arabic represents an important target for nlp due to its linguistic richness and diverse user base. however, arabic presents different characteristics making it challenging for nlp tasks. for instance, arabic script is written without spaces between letters, which poses difficulties in tokenization and word segmentation. additionally, there are several different orthographic conventions for representing short vowels, long vowels, and other vowel sounds, which add to the complexity of processing arabic text @xcite . these linguistic features necessitate the development of specialized algorithms and models that can accurately interpret and analyze arabic text. many researches have been conducted on this topic, addressing various aspects of arabic nlp, including text classification, named entity recognition, and machine translation. among these research efforts, stance detection has emerged as a crucial task. stance detection involves determining the position or attitude expressed in a text towards a particular target, which is essential for applications like opinion mining, sentiment analysis, and social media monitoring. this paper describes our participation in such a task: arabic stance evaluation @xcite . our work focuses on developing and refining techniques to accurately assess stances expressed in arabic texts, leveraging the latest advancements in nlp and machine learning. the implications of effective stance detection in arabic are far-reaching. it can enhance the capabilities of automated systems to gauge public opinion, monitor trends, and identify emerging issues in real-time, providing valuable insights for businesses, policymakers, and researchers. this work can be exploited in tasks like opinion mining, sentiment analysis, and social media monitoring, enabling more nuanced and culturally aware analyses of arabic content. this paper is organized into four sections: section 2 describes the dataset, detailing the sources, preprocessing steps, and characteristics of the data used in our experiments. section 3 presents the methods employed, including the models and techniques applied to tackle the stance detection task. section 4 is reserved for experimental results and discussions, where we analyze the performance of our approach and compare it with existing methods. finally, section 5 provides perspectives on future work, outlining potential improvements and directions for further research in arabic nlp. ## data in the experimentation of this work, we used mawqif dataset @xcite , which is the first arabic dataset for target-specific stance detection, composed of 4,121 tweets annotated about three targets : "covid-19 vaccine" (1,373 tweets), "digital transformation" (1,348 tweets) and "women empowerment" (1,400 tweets) as a shown in the mawqif is a multi-label dataset that provides annotations for stance,sentiment, and sarcasm as a shown in the in preparing our data and through extensive experiments, we opted for applying series of cleaning up steps to each tweet. first, we removed punctuation marks,numbers, non arabic words, and duplicated characters. second, we eliminated emoticons. third, we dropped stop words and arabic diacritics. finally we normalized the dataset. @xcite . furthermore, to address class imbalance in our training data, we implemented the synthetic minority oversampling technique (smote) @xcite . smote operates by identifying neighboring instances in the feature space, drawing a line between them, and generating new samples along that line. ## system our model is constructed upon a conventional machine learning approach. for model construction and training, we utilize the featureunion module in scikit-learn @xcite , which facilitates the seamless combination of various n-gram representations at both the word and character levels, as illustrated in figure 1 . during model training, we concatenate three vectors containing the following features weighted using term frequency-inverse document frequency (tf-idf): word n-grams (1 to 5-grams), character n-grams (1 to 4-grams), and character n-grams (1 to 5-grams) with word boundaries explicitly delineated by spaces. we employ series of classifiers organized through a voting technique utilizing the voting classifiers stochastic gradient descent(sgd) @xcite , linear support vector classification(linearsvc) @xcite , multinomial naive bayes (mnb) @xcite , ridge classifier @xcite , random forest (rf) @xcite . this approach constructs an ensemble voting classifier employing hard voting, which aggregates predicted class labels based on majority rule. the ensemble comprises the following classifiers: note that our model draws inspiration from a system designed for arabic dialect classification (ghoul and lejeune, 2020). ## results the arabic stance evaluation system using the ensemble approach showed promising outcomes. the f1-scores for digital transformation, covid vaccine, and women empowerment are shown in the the f1-score for women empowerment was significantly higher than for the other themes, indicating that the system performed a particularly good job of assessing positions on this particular topic. this could be due to a number of things, including the obvious sentiments that are common in nevertheless, the f1-score for digital transformation was marginally lower. this suggests that it would be difficult to assess stances on this issue. to overcome this issue more investigation is necessary. to sum up, these outcomes show that the used approach is valuable for stance evaluation with performance varying depending on the topic. ## discussion the application of the ensemble method in the arabic stance evaluation process has proved the importance of using advanced methods to overcome the challenges that arise in arabic nlp. ensemble methods, which combine multiple models to ameliorate prediction accuracy, have shown promise in dealing with the complexity of the arabic language such as its rich morphology, diverse dialects, orthographic variations, and complex tokenization task. arabic is renowned for its rich morphology, characterized by a complex system of word formation and inflection. arabic verbs and nouns undergo extensive inflectional changes to indicate tense, aspect, mood, person, number, and gender. for example, the verb root (to vaccinate) can be conjugated as (he vaccinated), (he vaccinates), (they vaccinate). in addition, arabic is known for its diverse dialects, which vary significantly across regions where arabic is spo- target trained on tested on f1-score women empowerment train set dev set 0.7927 train+dev test set 0.7393 covid vaccine train set dev set 0.7548 train+dev test set 0.70193 digital transformation train set dev set 0.6981 train+dev test set 0.6670 overall f1-score train set dev set 0.7485 train+dev test set 0.7027 other noticed complexity is the tokenization task. it can be challenging due to the complex nature of the language, especially with regards to morphology and to the different forms that words can take based on their context. for example the tweet (they monitor the negative effects) , the tokenization challenge is to discover the root ( ) of the conjugated verb and discover the definite article attached to the nouns ( and ). these methods use the strengths of certain models and reduce their weaknesses resulting in more reliable performance in stance detection. differences in f1-scores between subjects arise interesting questions about the impact of subject specificity on the accuracy of the stance detection task. for instance, the ambiguity that comes from idiomatic expressions can impact the effectiveness of nlp models. the significantly higher f1-scores for the women empowerment target raise the possibility that some topics have unique linguistic characteristics that facilitate correct classification. moreover, the relatively low f1-scores of digital transformation highlight the importance of continuous research programs to solve problems related unique to specific areas. in figure 2 , the confusion matrix of the test set indicates that predicting the "against" label was more challenging, with 118 true positives (tp), 66 false positives(fp), and 62 false negatives (fn). this difficulty might be attributed to the use of indirect language and cultural context. similarly, the confusion matrix highlights the difficulty in accurately predicting the "none" label, with only 10 tp, 24 fp, and 50 fn. these prediction challenges stem from the imbalanced nature of the data. to overcome these shortcomings, other techniques could be investigated: ## conclusion in summary, this study utilized an ensemble approach to examine arabic stance evaluation using the mawqif dataset. our system demonstrated a notable degree of success in detecting stance from arabic tweets, showcasing the potential of ensemble methods in this domain. despite these promising results, there remains room for improvement through advanced modeling techniques, such as transformer-based architectures and fine-tuning pre-trained language models for arabic. the availability of annotated datasets like mawqif is crucial for advancing arabic nlp research, providing a solid foundation for innovative applications. future research should focus on expanding dataset diversity and exploring transfer learning to adapt models trained on high-resource languages to arabic. our findings contribute to the growing body of work in arabic nlp, highlighting the potential of ensemble approaches. by continuing to innovate and build on this research, we aim to develop more accurate, efficient, and contextually aware systems for arabic text analysis, fostering better communication and understanding in the arabic-speaking world.