ANLP RG at S tance E val2024: Comparative Evaluation of Stance, Sentiment and Sarcasm Detection (2024)

Abstract: As part of our study, we worked on three tasks:stance detection, sarcasm detection and senti-ment analysis using fine-tuning techniques onBERT-based models. Fine-tuning parameterswere carefully adjusted over multiple iterationsto maximize model performance. The threetasks are essential in the field of natural lan-guage processing (NLP) and present uniquechallenges. Stance detection is a critical taskaimed at identifying a writer’s stances or view-points in relation to a topic. Sarcasm detectionseeks to spot sarcastic expressions, while senti-ment analysis determines the attitude expressedin a text. After numerous experiments, we iden-tified Arabert-twitter as the model offering thebest performance for all three tasks. In particu-lar, it achieves a macro F-score of 78.08% forstance detection, a macro F1-score of 59.51%for sarcasm detection and a macro F1-score of64.57% for sentiment detection. .Our source code is available at https://github.com/MezghaniAmal/Mawqif

## introduction in a world where opinions abound and are constantly confronted, the ability to detect stances becomes crucial. whether in the political arena, social debates or academic discussions, knowing how to identify stances becomes an essential tool for understanding underlying intentions and arguments. detailed and nuanced understanding of natural language represents a major challenge for contemporary artificial intelligence, as highlighted by @xcite it is important to recognize that decoding the subtleties of human language confronts us with the cultural and linguistic diversity that characterizes human communication. researchers are thus stimulated to constantly push the limits of algorithmic understanding, by exploring the nuances of language in its various contexts. every obstacle encountered, whether it is the variability of language, the ambiguity of expressions or the crucial need for context, represents an opportunity for learning and growth. significant progress has been made in the field of stance detection. recent work by @xcite introduced new datasets like mt-csd and proposed innovative methods such as the global-local attention network (glan) , achieving a precision of only 50.47% in the stance detection task. most stance detection research focuses on the english language, with relatively little research conducted on arabic. arabic is known for its syntactic complexity and rich vocabulary. @xcite focused on adapting models for better understanding and analyzing this language. they presented their new arabic stance detection dataset (arastance). they then evaluated this dataset, along with two other stance detection datasets, using several bert-based models. their best model achieved an accuracy of 85% and a macro f1 score of 78%. recent work by alrowais and alsaeed, 2023 has focused on stance detection in arabic tweets by comparing different stance detection models using four transformers: araelectra, marbert, arabert, and qarib. their results show that the arabert model performed better than the other three models, with an f1 score of 70%. jaziriyan et al., 2021 are working on target-based stance detection, which involves identifying a stance towards a specific target. their approach resulted in a new corpus called exaasc for the arabic language. they used bert to evaluate their corpus and achieved a macro f1 score of 70.69%.in addition to stance detection, understanding sarcasm represents a major challenge for sentiment analysis systems. researchers like a. @xcite have worked on creating sarcastic corpora for arabic and proposed hybrid deep learning approaches to improve sarcasm detection. several works have also been carried out on the detection of sarcasm and sentiment, thus enriching our understanding of these complex aspects of natural language. the shared task stanceeval alturayeif et al., 2024 provides a platform to explore and perfect stance, sarcasm and sentiment detection techniques. it focuses on classifying stances in texts using different natural language processing (nlp) techniques to capture the subtlety of points of view expressed in various contexts. in this article, we look at the challenges and advancements in stance sensing. we explore the techniques, models, and approaches that have proven most effective in this area, while examining the specific challenges encountered when modeling these complex aspects of language. in addition to stance detection, we build models for sentiment and sarcasm detection tasks, which are crucial aspects of natural language analysis. we begin our article with a detailed description of the corpus of data used, highlighting the different classes represented and the distribution of labels. we also outline the data preprocessing pipeline we have in place to ensure the quality and relevance of our data before using it in our models. in section 3 , we describe in detail the parameter tuning process we undertook to optimize the performance of our models. we explain the different steps of the process, including the pre-trained components used, as well as the hyperparameters adjusted to achieve the best possible performance. in section 4, we evaluate the performance of our models on the different components of our objectives. we present the evaluation metrics used and discuss the results obtained. in section 5 we discuss the results obtained and propose avenues for improvement for the performance of our models in the tasks of position detection, sarcasm and sentiment analysis. finally, we conclude our article by summarizing the main conclusions drawn from our study and proposing perspectives. ## data in this work, we use mawqif dataset @xcite which is an important resource for the study of various topics, including "covid-19 vaccination", "digital transformation" and "women's empowerment". it consists of 4121 entries across these three categories, with 1373 entries for covid-19 vaccination, 1348 for digital transformation and 1400 for women's empowerment. this dataset is structured as a multi-label dataset, which means that an entry can be associated with multiple labels. tags include "stance" which can be "favor", "against" or "none" "sentiment" which can be "positive", "negative" or "neutral" as well as "sarcasm" which can be "sarcastic " or "nonsarcastic". these different dimensions allow us to carry out an in-depth analysis of the attitudes, opinions and linguistic nuances expressed in the texts. figure 1 shows the comments associated with their annotations in the mawqif dataset. alturayeif et al., 2022. figure 2 shows the distribution of different labels in the training and testing parts of our dataset, highlighting an imbalance in the sarcasm label annotation. moreover, this dataset comprises several arabic dialects and subdialects, significantly complicating automated processing. to analyze the distribution of texts based on their dialect, we employed the pre-trained model camelbert-msa did nadi inoue et al., 2021. this model specializes in dialect identification, distinguishing between coarse-grained regional varieties and national varieties. it was developed by adapting the camelbert model originally designed for modern standard arabic (msa). figure 3 depicts the static distribution of dialects in our dataset, with modern standard arabic (msa) being the most dominant. ## system in this section, we fine-tuned four bert-based models for each the three tasks: sarcasm detection, stance detection, and sentiment analysis. the models used are as follows: we fine-tuned each model for up to 20 epochs, with a maximum sequence length of 128, a batch size of 32, an adamw optimizer with a learning rate of 2e-5, and a cosineannealinglr scheduler to dynamically adjust the learning rate during model training. we initially divided the mawqif dataset into two parts: a training set and a validation set. this split was performed using the train_test_split function from scikit-learn, with stratification based on the "stance," "sarcasm," and "sentiment" columns to preserve the class distribution between the training and validation sets. the proportions are as follows: 82% of the data for the training set and 18% for the validation set. this division ensures that model performance can be reliably assessed while maximizing the use of data for training. in addition to this split, we used a separate test set, loaded from a separate csv file, which contains data specifically reserved for the final evaluation of our model.the results reported in our study are based on the validation set for several reasons. initially, we utilized the validation set to fine-tune hyperparameters and conduct crossvalidation throughout the model training process. this approach enabled us to enhance the model's performance effectively while mitigating overfitting risks.secondly, we continuously monitored and assessed the model's performance throughout its development, allowing for iterative adjustments and preliminary performance evaluations. finally, the test set, sourced from a separate csv file, was strictly reserved for the final evaluation after the model underwent full training and validation. this method ensures that the reported results are based on completely independent data, providing a more objective and reliable measure of the model's performance. it's important to emphasize that this systematic approach was consistently applied to each task in our study-whether stance detection, sarcasm detection, or sentiment analysis. for every task, we followed the same protocol of data partitioning, hyperparameter optimization, and validation set evaluation before using the test set for the final assessment. this ensures a rigorous and standardized evaluation of our model's performance across various aspects of data analysis. ## results the arabert-twitter model demonstrated the best overall performance as well as per-target performance. tables 1 , 2 and 3 present the results of this model with the best configuration of parameters for stance, sarcasm and sentiment detection.following an evaluation for stance detection, the overall f1 score reached 78.08%. thus, the arabert-twitter model turns out to be more efficient in this task. regarding sarcasm detection, despite the imbalance of classes for this task and an attempt to increase it was made, the model displays high accuracy, suggesting its ability to properly classify the majority (non-sarcastic) class. however, this result can be attributed to a possible class bias where the model is more inclined to predict the majority class, as well as to the distribution of predictions where a few correct predictions for the minority class can contribute to the overall accuracy , so it is essential to note that accuracy alone only for this task provides part of the picture and it is important to look at other metrics, such as the f1 score, which in this case reaches 59.51%. an attempt to increase the corpus was undertaken, highlighting the major challenge of detecting sarcasm.finally, for sentiment detection, the f1 score reached 64.57%, demonstrating an intermediate performance of the model. these results highlight the need for further investigation, as in the area of sarcasm detection. ## discussion facing variable results despite using the same corpus of data for fine-tuning different bert-based models, with superior performance in stance detection, we are confronted with a set of challenges and opportunities to enhance the effectiveness of our models.an imaginable strategy to improve performance is to explore combined or ensemble models. by merging individual models for each task, we could harness the strengths of each to achieve a more consistent overall performance. this approach could enable better utilization of the information available in the data and more robust classification on new examples. furthermore, the adoption of multitask learning offers another intriguing avenue. by enabling the model to learn multiple tasks simultaneously, we could facilitate efficient knowledge sharing between them, potentially enhancing their respective performances. this would necessitate careful design of the model architecture and appropriate management of the weights assigned to each task to optimize results. by combining these approaches and tailoring them to our specific needs, we could hope to significantly enhance the performance of our models in stance detection, sarcasm detection and sentiment analysis tasks. this would require methodical work and rigorous evaluation, but the potential benefits in terms of precision and generalizability would certainly justify the effort. ## conclusion in conclusion, our study highlights the crucial importance of understanding linguistic nuances in stance detection, sarcasm detection, and sentiment analysis through natural language processing (nlp) . by closely examining recent advancements in these domains, we underscored the remarkable progress made in processing conversational data, particularly in adapting models for arabic language processing. additionally, we shed light on the persistent challenges in sarcasm detection and sentiment analysis, while identifying innovative approaches aimed at enhancing model accuracy. through the use of fine-tuning techniques on bert-based models, we have taken steps to maximize model performance in these essential tasks within the context of arabic dialects. our results emphasize the outstanding performance of the arabert-twitter model in stance detection, achieving an impressive overall f1 score of 78.08%. it is crucial to highlight the pivotal role of @xcite in continuously improving stance, sarcasm and sentiment detection techniques, particularly in adapting these approaches to the specificities of arabic dialects. these advancements represent a significant step towards a deeper and more precise understanding of human language, particularly in the realm of nlp applied to arabic dialects.