Advancing A rabic Sentiment Analysis: A r S en Benchmark and the Improved Fuzzy Deep Hybrid Network (2024)

Abstract: Sentiment analysis is pivotal in Natural Language Processing for understanding opinions and emotions in text. While advancements in Sentiment analysis for English are notable, Arabic Sentiment Analysis (ASA) lags, despite the growing Arabic online user base. Existing ASA benchmarks are often outdated and lack comprehensive evaluation capabilities for state-of-the-art models. To bridge this gap, we introduce ArSen, a meticulously annotated COVID-19-themed Arabic dataset, and the IFDHN, a novel model incorporating fuzzy logic for enhanced sentiment classification. ArSen provides a contemporary, robust benchmark, and IFDHN achieves state-of-the-art performance on ASA tasks. Comprehensive evaluations demonstrate the efficacy of IFDHN using the ArSen dataset, highlighting future research directions in ASA.

## introduction sentiment analysis (sa), also known as opinion mining, is a critical task in natural language processing (nlp) that involves detecting, extracting, and classifying opinions and emotions expressed in text @xcite . in recent years, the advent of social media platforms like twitter (x for now) has provided a rich data source for sa. building on this, sophisticated models such as roberta-lstm and keaht have emerged, further promoting the development of the sa field @xcite @xcite . despite these advancements in sentiment analysis, the complexity of the arabic language, com-bined with its significant differences from english, has led to a scarcity of studies and resources in arabic sentiment analysis (asa) @xcite . the widely used asa benchmarks, such as gold standard @xcite and semeval @xcite , are often outdated and small in scale (less than 10,000). to address this gap, we leveraged a large volume of arabic tweets generated during the covid-19 pandemic. during this pandemic, arabic-speaking users widely shared their emotions and experiences. this large-scale public sharing made it possible to construct a comprehensive and diverse dataset. therefore, we introduce arabic sentiment (arsen), a covid-19-themed arabic benchmark created through meticulous manual annotation by trained professionals. the arsen benchmark aims to address the previously mentioned challenges and provide asa research with a modern, comprehensive resource featuring accurate data annotations, thus advancing the field of asa within nlp. additionally, we propose a new model called the improved fuzzy deep hybrid network (ifdhn), designed specifically to enhance sentiment classification through the integration of fuzzy logic. fuzzy logic has been effectively applied in sentiment analysis to handle the ambiguity and nuances of language @xcite . our ifdhn model demonstrates state-of-the-art (sota) performance in asa tasks, validating the effectiveness of incorporating fuzzy logic to improve classification accuracy. our contributions are twofold: (1) we proposed arsen, a robust and contemporary benchmark for asa tasks, addressing the lack of up-to-date and high-quality benchmarks in this domain; (2) we introduced ifdhn, a novel model that integrates fuzzy logic to better handle ambiguous sentiments, improving overall classification performance. the paper is organized as follows: section 2 introduces the arsen benchmark, detailing its con-struction and significance. section 3 discusses the architecture and features of ifdhn model. section 4 presents comprehensive evaluations of the ifdhn model against leading sota models using the arsen dataset. finally, section 5 summarizes our findings and proposes directions for future research in asa. ## arsen benchmark to address the aforementioned shortcomings in asa, we introduce the arsen benchmark. firstly, our motivation for creating the arsen benchmark is discussed in section 2.1, where we outline the rationale for selecting covid-19-themed tweets to develop the benchmark. we then move on to describe the benchmark construction process in section 2.2, providing a thorough explanation of the data preprocessing and annotation steps involved. this section aims to provide a clear understanding of how arsen was developed and the rigorous methodologies employed to ensure its quality. ## proposed model researchers have long recognized the unique advantages of fuzzy logic in capturing the ambiguities and uncertainties of real-world data @xcite . @xcite introduced the concept of computing words using fuzzy logic. in this approach, sentiment polarity is determined by calculating fuzzy membership values ranging from 0.0 to 1.0. each word in the text is assigned a score within this range, reflecting the realistic scenario where sentiment is not always binary but often am-biguous and uncertain @xcite . in recent years, fuzzy logic has also drawn significant attention in the field of sa @xcite @xcite . moreover, the arsen dataset contains contextual information, so we would like to construct a multi-channel fuzzy model to test the arsen dataset. a recent study in the field of fake news detection provides an opportunity for this work. @xcite introduced the fdhn model, which uses fuzzy logic and multiple input types: news text, textual context, and numerical context. the text inputs are processed by textcnns, while numerical context is handled by cnn and bi-lstm layers, then processed by a fuzzy layer. the model's outputs are concatenated and integrated in the final layer, achieving sota performance metrics on the liar dataset (wang, 2017), which includes multiclass labels such as pants-fire, false, barely-true, half-true, mostly-true, and true. this use of fuzzy multi-class labels shows a strong similarity to our arsen benchmark. beyond this, they both require contextual information. therefore, we believe that the strengths of the fdhn model allow us to adequately analyze the arsen benchmark. in order to transfer fdhn to the asa task, we tailored and improved the architecture of fdhn to propose the ifdhn model, aiming to better utilize its fuzzy logic and context-dependent properties. through our experiments, we found that introducing textual context information, specifically the created_at and description features in the arsen dataset, was redundant and decreased the model's performance, leading us to remove these features. furthermore, we designed a separate textcnn to process tweet text and then fine-tuned the cnn-bilstm module for numerical context. although large language models (llms) like gpt-3.5/4 (openai, 2024) and llama-3 @xcite have achieved impressive results in many nlp tasks like question answering and text generation, they fall short in interpretability and computational efficiency for fuzzy classification tasks @xcite @xcite . for example, gpt-4 achieved only 28.1% accuracy on the liar dataset, whereas fdhn achieved 46.5% @xcite , and fdhn requires only about 3 seconds to train an epoch on a single a100 gpu, while llms generally require more than 4 a100 gpus to be fine-tuned for hours to train for downstream tasks. taking these considerations into account, we decided to use fdhn as the baseline model in this work. however, we also included the results of llama-3-8b for comparison. as illustrated in figure 1 , the ifdhn model comprises two primary channels: tweet text and numerical context. the tweet text is fed into a distinct textcnn, while the numerical context is processed by a combination of cnn and bi-lstm layers before being passed through a fuzzy layer. the model produces three outputs: output 1 is derived from the tweet text channel, output 2 is derived from the numerical context channel, and output 3 is the fuzzy layer-processed version of output 2. these three output representations are then concatenated and integrated in the final layer. in particular, an example data point used in our ifdhn model is shown in ## experimental results in this section, we present a comprehensive analysis of our experiments, which are divided into two main parts: a performance evaluation on the arsen dataset using the ifdhn model and other sota models (section 4.1). in addition, an ablation study was performed to investigate the impact of different features on the performance of the ifdhn model (section 4.2). performance evaluation metrics are detailed in appendix b. detailed information about our experimental setup, including the development environment and hyperparameter configurations, can be found in appendix c. ## conclusion in this paper, we introduced a novel arabic sentiment analysis benchmark focused on the covid-19 pandemic and presented the ifdhn model, tailored specifically for sentiment analysis within this context. our model demonstrated substantial performance improvements over other sota models. compared to the large language model llama-3-8b, our model achieved a 0.5% and 2.17% increase in accuracy on the validation and test sets, respectively, and a 2.17% increase in f1-micro on the test set. more notably, the ifdhn model reduced processing time by approximately 422 times compared to llama-3-8b, achieving a remarkable processing speed of just 0.44 seconds. this comprehensive evaluation highlights the ifdhn model's capability to effectively capture nuanced sentiments, making it a valuable tool for understanding public sentiment.