A Weak Supervised Transfer Learning Approach for Sentiment Analysis to the K uwaiti Dialect (2022)

Abstract: Developing a system for sentiment analysis is very challenging for the Arabic language due to the limitations in the available Arabic datasets. Many Arabic dialects are still not studied by researchers in Arabic sentiment analysis due to the complexity of annotators’ recruitment process during dataset creation. This paper covers the research gap in sentiment analysis for the Kuwaiti dialect by proposing a weak supervised approach to develop a large labeled dataset. Our dataset consists of over 16.6k tweets with 7,905 negatives, 7,902 positives, and 860 neutrals that spans several themes and time frames to remove any bias that might affect its content. The annotation agreement between our proposed system’s labels and human-annotated labels reports 93% for the pairwise percent agreement and 0.87 for Cohen’s kappa coefficient. Furthermore, we evaluate our dataset using multiple traditional machine learning classifiers and advanced deep learning language models to test its performance. The results report 89% accuracy when applied to the testing dataset using the ARBERT model.

## introduction datasets are the foundation of the most significant innovation in the field of natural language processing (nlp). the development of nlp algorithms and tools is dependent on the availability and quality of the datasets that serve their goals. while there are plenty of english language datasets, for some other natural languages, there are still minimal resources, such as the arabic language @xcite . the arabic language is considered among the low-resource languages for nlp, however, the number of people who speaks arabic exceeds 353.6 million foot_0 . the arabic language has multiple forms. the classical arabic language (cal) is the oldest form of arabic and is often used in islamic manuscripts (e.g., the quran) @xcite . modern standard arabic (msa) is the official language for arabic countries and it is used in official media resources, writing books, etc @xcite . the last and most dominant form of arabic is the arabic dialects, which are the native language form of daily communication. the arabic dialects differ based on geographical and social classes @xcite . moreover, arabic dialects are often used in online user-generated content such as on twitter, facebook, and instagram. this variation among arabic dialects makes it very challenging to develop tools that can process arabic social media content accurately. in this study, we develop a dataset based on an innovative method to reduce the number of human annotators and propose a text classification model for sentiment analysis specifically for the kuwaiti dialect. the kuwaiti dialect has not been comprehensively covered and studied in previous computational linguistic research. according to our knowledge, only @xcite investigates some linguistic tools for the kuwaiti dialect to develop an approach for unsupervised sentiment analysis, however, their dataset is not publicly available for researchers. this gap in research inspires us to further study the kuwaiti dialect and to create linguistic resources to support research in this area. this initial step in studying the kuwaiti dialect could also support the study of other underrepresented arabian gulf dialects that might share some vocabularies with the kuwaiti dialect, for example, it can help researchers in bahraini or qatari dialects. the key contributions of this study are threefold: this paper starts with some background information after the introduction that covers the kuwaiti dialect, sentiment analysis resources, the latest approaches and software frameworks in labeling large datasets, the weak supervised techniques, and the zero-shot models applied in the experiments. the methodology is discussed in detail in the third section, including dataset construction, dataset labeling, classification model, and performance evaluation. in the third section, we present the results, error analysis, and discuss them thoroughly. the paper concludes with a conclusion and proposes directions for future works. the paper also includes an ethics statement at the end and appenedices. ## results and discussion firstly, we evaluate the snorkel annotated dataset to check for the annotation agreement between the snorkle-labeled dataset and the gold-labeled dataset (human-labeled dataset). thus, we consider pairwise percent agreement and cohen's kappa coefficient metrics to evaluate annotation agreement. the result report 93% for the pairwise percent agreement and cohen's kappa coefficient is a near-perfect agreement with a value equals to 0.87. we also tested the performance of the snorkel probabilistic labeling model by applying the same steps illustrated in fig. 2 . snorkel framework already provides a function to evaluate its performance in case gold labels are present in the dataset. thus, we applied the steps to the gold-labeled testing dataset of @xmath0,534 tweets, and the final performance results of the labeling system were accuracy score of 93%, and f1-macro of 84%. the results of the bert models highlight an important finding. even though arabert includes in its pre-training dataset tweets and emoji, similar to our dataset, and marbert is developed using a large tweets dataset, they both were not performing as well as arbert. the arbert model reports 0.75 and 0.89 for the macro-averaged f1 and accuracy scores respectively on the test set. ## conclusions in this paper, we release the first open largescale dataset focused on sentiment analysis for the kuwaiti dialect using a semi-supervised approach. we created a semi-supervised model based on the snorkel framework to reduce the need for human annotators and boost the size of the labeled data rapidly and accurately. to test the applicability of the dataset, we evaluated various traditional machine learning classifier baselines, as well as advanced bert-based language model classifiers. the results showed that our approach generates high-performance scores in both macro-average f1 and accuracy results. we believe our approach will help foster research and development of nlp systems, which were previously little studied due to the challenges faced by human annotators.