SENIT at A ra F in NLP 2024: trust your model or combine two (2024)

Abstract: We describe our submitted system to the 2024 Shared Task on The Arabic Financial NLP (Malaysha et al., 2024). We tackled Subtask 1, namely Multi-dialect Intent Detection. We used state-of-the-art pretrained contextualized text representation models and fine-tuned them according to the downstream task at hand. We started by finetuning multilingual BERT and various Arabic variants, namely MARBERTV1, MARBERTV2, and CAMeLBERT. Then, we employed an ensembling technique to improve our classification performance combining MARBERTV2 and CAMeLBERT embeddings. The findings indicate that MARBERTV2 surpassed all the other models mentioned.

## introduction the arafinnlp @xcite initiative underscores the critical role of financial natural language processing (nlp) within the arab world's financial landscape. this importance is accentuated by the significant growth experienced in middle eastern stock markets, fueled by diverse sectors contributing to expansion. this growth, spanning multiple countries, reflects the dynamic nature of the region's financial markets, attracting global attention and investment. as these markets evolve, the demand for advanced arabic nlp tools becomes essential to address local linguistic intricacies and serve the global financial community engaging with these markets. the progression of arabic nlp capabilities in finance is vital for effectively analyzing and interpreting financial data. the arabic language includes many variants and dialects in addition to modern standard arabic (msa). while some dialects may share certain vocabulary, they still vary significantly between countries, each with its own unique characteristics, which impacts language processing tasks. for example, the sentence " " represents a query text in palestinian dialect @xcite and means "show me how to link the new card". the equivalent sentence in msa is: " ". for a speaker not used to the palestinian dialect this can be very confusing as " " is literally translated to "clas p" english. these sentences were extracted from arbanking77 @xcite , a msa and palestinian arabic dataset for intent detection in the banking domain. this dataset is being proposed in the 2024 shared task on the arabic financial nlp (arafinnlp) @xcite . in the intent detection task, extracting the necessary information to identify intent is challenging when dealing with multiple dialects. training a model on msa and palestinian data and expecting it to interpret a range of arabic dialects, such as gulf, levantine, and north african, adds to the complexity of the task. the contributions of this paper are summarized as follows: the structure of this paper is delineated as follows: section 2 explores related work approaches and datasets, section 3 furnishes a succinct exposition delineating the dataset employed in this study. section 4 elaborates on the systems utilized and the experimental configuration undertaken to construct models for multi-dialect intent detection. section 5 elucidates the results acquired from these endeavors. subsequently, section 6 engenders a comprehensive discussion. the paper culminates with section 7, offering conclusions and delineating prospective avenues for future research. ## literature review research in arabic nlp has seen significant advancements in recent years, with various studies exploring techniques and models tailored for different dialects and applications. pretrained language models, such as bert (jacob devlin and toutanova., 2019), have been adapted for arabic, leading to the development of models like marbert (muhammad abdul-mageed and nagoudi, 2021) and camelbert @xcite , which focus on modern standard arabic and various dialects. these models have shown improvements in tasks like sentiment analysis, named entity recognition, and intent detection. other notable works include the use of ensembling techniques and transfer learning to enhance model performance across different nlp tasks. the arafinnlp shared task builds on these advancements, providing a platform to further explore the potential of these models in a financial context. ## data the dataset employed in the context of the multidialect intent detection competition's subtask 1 originates from arbanking77. arbanking77 was obtained by translating the english banking77 dataset @xcite into msa and palestinian arabic. this dataset is being expanded in this shared task to include a set of arabic dialects in addition to palestinian. arbanking77 includes a substantial collection of 31,404 queries categorized into 77 distinct intent classes, encompassing a broad spectrum of banking-related inquiries and requests. the dataset provided for subtask-1 of the competition consists of 24,018 sentences in msa and palestinian dialect, labeled with 77 intent classes. the initial released training data included the validation dataset as well. the data distributions for subtask-1 are shown in figure 1. train validation total msa 10,733 1,230 11,963 pal 10,821 1,234 12,055 total 21,554 2,464 24,018 ## system description pretrained contextualized text representation models are highly effective for natural language comprehension. among these, bert (jacob devlin and toutanova., 2019) is the state-of-the-art model, surpassing its predecessors in nlp. arabic-specific studies have also been conducted. in our approach, we explored bert variations for arabic: camel-bert, marbertv1, and marbertv2. additionally, we combined camelbert and mar-bertv2 embeddings with a fully connected layer for fine-tuning. ## results ## conclusion we employed four language models to detect multidialect intents (camelbert, marbertv1, marbertv2, and a combination of camel-bert and marbertv2). the best results were obtained with marbertv2 for subtask 1, which was then selected for the final submission.