F ired_from_ NLP at A ra F in NLP 2024: Dual-Phase- BERT - A Fine-Tuned Transformer-Based Model for Multi-Dialect Intent Detection in The Financial Domain for The A rabic Language (2024)

Abstract: In the financial industry, identifying user intent from text inputs is crucial for various tasks such as automated trading, sentiment analysis, and customer support. One important component of natural language processing (NLP) is intent detection, which is significant to the finance sector. Limited studies have been conducted in the field of finance using languages with limited resources like Arabic, despite notable works being done in high-resource languages like English. To advance Arabic NLP in the financial domain, the organizer of AraFinNLP 2024 has arranged a shared task for detecting banking intents from the queries in various Arabic dialects, introducing a novel dataset named ArBanking77 which includes a collection of banking queries categorized into 77 distinct intents classes. To accomplish this task, we have presented a hierarchical approach called Dual-Phase-BERT in which the detection of dialects is carried out first, followed by the detection of banking intents. Using the provided ArBanking77 dataset, we have trained and evaluated several conventional machine learning, and deep learning models along with some cutting-edge transformer-based models. Among these models, our proposed Dual-Phase-BERT model has ranked 7^{th} out of all competitors, scoring 0.801 on the scale of F1-score on the test set.

## introduction in the arabic-speaking world, where a multitude of dialects coexist alongside modern standard arabic (msa), the need for advanced nlp models capable of handling multi-dialectal variations is increasingly evident. one critical application of such models lies in the domain of intent detection, particularly in financial sectors. a significant amount of intent detection work has been done in high-resource languages, like english @xcite , @xcite , @xcite . however, there is a noticeable gap in significant studies addressing intent detection in the financial sector for languages with limited resources such as arabic. in response to this challenge, arafinnlp 2024 has arranged a shared task named task 1: multi-dialect intent detection @xcite , introducing a novel dataset called arbanking77 @xcite to classify the banking intents from queries in different arabic dialects. the key objective of the paper is to accurately understand user queries expressed in different dialects and categorize user intents accordingly. to achieve this objective, we have utilized multiple models, including three distinct deep learning architectures (lstm, bilstm), two transformer-based approaches (arabert, dual-phase bert), and three different machine learning algorithms (xgboost, random forest, svm). we have trained and assessed every model using the arbanking77 dataset @xcite provided for the arafinnlp 2024 task 1 @xcite , in order to do a comparative analysis. ultimately, our proposed dual-phase-bert model achieved the highest micro-f1 score of 0.801 on the test dataset among all evaluated models. the following are our research's main contributions: the implementation details are available in this github repository foot_0 . ## related work prior works on cross-dialect intent detection fall into three primary categories: machine learning, deep learning, and transformer-based methods. traditional machine learning approaches (al-tuama and nasrawi, 2022), @xcite have been applied for detecting intentions on augmented and without augmented data. machine learning algorithms such as multinomial naïve bayes, svm, logistic regression, and random forest have been used where random forest provided the best performance (al-tuama and nasrawi, 2022). in another previous work @xcite , class-specific (crime) intent detection in social media posts has been explored leveraging machine learning algorithms where logistic regression has shown superior performance in terms of accuracy. deep learning-based approaches perform better on cross-dialect intent detection due to their layered structure and less dependency on explicitly defining features. a deep learning neural network, bi-model based recurrent neural network @xcite , has been capitalized to detect intention and slot filling. in another work @xcite , for simultaneous intent detection and slot-filling, a unique bi-directional interconnected model has been presented. a fine-tuned transformer named joint-bert @xcite has been utilized for joint intent classification and slot filling. another approach @xcite uses a co-interactive transformer to detect intention and joint filling. @xcite highlights the sensitivity of the stateof-the-art multi-intention model to threshold setting. they have taken advantage of several layers within a transformer-based encoder to create multigrain representations. in @xcite , researchers enhance intent detection performance by utilizing datasets with different intent categories. ## dataset we have utilized the multi-dialect intent detection dataset arbanking77 @xcite provided for the shared task 1 of the arafinnlp 2024 @xcite . the dataset consists of arabic banking queries in the msa and the pal dialect, labeled one of the 77 categories (banking intents). the dataset that has been provided is highly unbalanced; samples under some categories, such as beneficiary disapproval and transaction adjustments twice, are significantly more than samples under other categories, such as virtual cards not working and contactless not working. ## methodology in this section, we have presented an explanation of our methodology to develop models for the detection of multi-dialect banking intents. figure 1 presents a summary of our methodology. ## experimental findings in this section, we have provided the experimental results of different conventional models alongside our proposed model on the test dataset. ## error analysis although our dual-phase-bert model has performed extremely well on the dev set with an f1 score of 0.984, the score has dropped to 0.801 in the test phase. the main reason behind this is the inclusion of some other arabic dialects such as levantine, north african, and gulf alongside msa and palestinian in the test set whereas the training set and the dev set contained queries only in msa and palestinian. besides that, the provided dataset is highly unbalanced. therefore, the model is slightly biased toward the classes that have more samples during the training period and therefore leads to some misclassifications. ## conclusion we have conducted a comparative performance analysis in this study by assessing a range of machine learning, deep learning, and transformerbased methods to detect banking intents from queries in different arabic dialects. the arbank-ing77 dataset that was given in the shared task has been used. our results indicate that our proposed dual-phase-bert model has achieved better results than all other models, achieving a remarkable f1 score of 0.801 on the test dataset. in the future, we will try different techniques to mitigate the data imbalance problem and explore more transformerbased model architectures.