SMASH at A ra F in NLP 2024: Benchmarking A rabic BERT Models on the Intent Detection (2024)

Abstract: The recent growth in Middle Eastern stock markets has intensified the demand for specialized financial Arabic NLP models to serve this sector. This article presents the participation of Team SMASH of The University of Edinburgh in the Multi-dialect Intent Detection task (Subtask 1) of the Arabic Financial NLP (AraFinNLP) Shared Task 2024. The dataset used in the shared task is the ArBanking77 (Jarrar et al., 2023). We tackled this task as a classification problem and utilized several BERT and BART-based models to classify the queries efficiently. Our solution is based on implementing a two-step hierarchical classification model based on MARBERTv2. We fine-tuned the model by using the original queries. Our team, SMASH, was ranked 9th with a macro F1 score of 0.7866, indicating areas for further refinement and potential enhancement of the model’s performance.

## introduction the financial natural language processing (nlp) models have a crucial role in facilitating the daily operations offered by financial sectors worldwide. studies show that the middle eastern stock markets have a robust growth in recent years @xcite , yet arabic nlp in the financial sector still faces significant challenges @xcite . hence, advanced arabic nlp tools become essential to address local dialect differences and meet the needs of international financial communities dealing with these markets. despite the remarkable advancements in transformer-based english nlp models in recent years, the validation of robust financial arabic nlp models has remained notably deficient. this disparity persists in adequately addressing the linguistic nuances inherent in the diverse arabic dialects @xcite @xcite , thereby hindering their efficacy within the financial sector @xcite . this presents both challenges and opportunities for arabic nlp researchers. the modern standard arabic (msa) is the standard version of arabic that is usually used in official communications, news, and books. however, there are more than 27 arabic dialects used as spoken versions across the arabic world, and they have emerged in a written format recently @xcite . due to the complexity of the linguistic features that msa and these dialects have, nlp tools fail to support them properly @xcite @xcite . in pursuit of enhancing financial arabic nlp, arafinnlp shared task 2024 @xcite encompasses two pivotal subtasks: the dataset used in this shared task is arbank-ing77 @xcite , the arabic version of the english banking77 dataset @xcite . arbanking77 contains of different arabic dialects including the msa, moroccan, saudi, palestinian and tunisian arabic, and contains 39,315 annotated queries within the banking sector. this paper describes our participation in subtask 1 of the arafinnlp shared task 2024 @xcite , where we achieved 9th place. we compared the performance of different bert-based models in different setups, including direct finetuning on the 77 intents and a two-step classification approach in which we grouped the intents based on their categories. our official submission was based on the two-step classification approach utilizing the marbertv2 model, which achieved an f1 score of 0.9470 on the palestinian dialect validation set (pal val) but only 0.7866 on the shared task official results on the blind test set. ## literature review intent detection and classification is a form of text classification problem and it is widely investigated by researchers @xcite . most of the works focused on english, while arabic lagged behind with a few works such as @xcite @xcite . @xcite targeted intent classification in the context of dialogue systems. @xcite developed a chatbot that utilizes a transformer model trained for named entity recognition and entity classification in a multitask learning setup. @xcite created arbanking77 dataset that covers 77 intents in the banking domain. ## data the dataset used in the arafinnlp 2024 shared tasks is arbanking77 @xcite , a translated version of the english banking77 dataset into msa and palestinian arabic @xcite . arbanking77 dataset contains 39,315 annotated examples over 77 intents @xcite . for evaluation, the shared task organizers split the arbanking77 dataset into multiple sets, as shown in ## methodology this section provides an overview of the experimental setup and models used in the experiments. the experiments span two approaches: direct finetuning on all 77 classes (intents) and two-step finetuning after grouping the intents into groups. in both approaches, the models were fine-tuned on the training data for the task. ## conclusion this paper is an exploration study investigating the performance of several bert and bart-based models for the arafinnlp 2024 shared task. we evaluated the performance of these models in different setups, including the direct classification approach and the two-step classification approach. our experiments exhibited that the hierarchical classification approach generally improved the performance of the models except for arabart. however, further consideration may improve the performance, including using different models per group, which might result in an improvement in performance. such an approach is time consuming as it requires testing all possible permutations, which we may apply in future works. another potential improvement may be gained by experimenting with different grouping criteria.