S emantic C uet S ync at A ra F in NLP 2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers (2024)

Abstract: Intention detection is a crucial aspect of natural language understanding (NLU), focusing on identifying the primary objective underlying user input. In this work, we present a transformer-based method that excels in determining the intent of Arabic text within the banking domain. We explored several machine learning (ML), deep learning (DL), and transformer-based models on an Arabic banking dataset for intent detection. Our findings underscore the challenges that traditional ML and DL models face in understanding the nuances of various Arabic dialects, leading to subpar performance in intent detection. However, the transformer-based methods, designed to tackle such complexities, significantly outperformed the other models in classifying intent across different Arabic dialects. Notably, the AraBERTv2 model achieved the highest micro F1 score of 82.08% in ArBanking77 dataset, a testament to its effectiveness in this context. This achievement, which contributed to our work being ranked 5 ^{th} in the shared task, AraFinNLP2024, highlights the importance of developing models that can effectively handle the intricacies of Arabic language processing and intent detection.

## introduction textual intent detection is a natural language processing (nlp) process that involves identifying and understanding a text input's underlying purpose or objective. this task is essential for enabling machines to comprehend what a user wants to achieve with their message. the goal is to classify the text into predefined intent categories, such as asking a question, requesting, providing feedback, or initiating a transaction. intent detection plays a pivotal role in enhancing user experiences, automating processes, ensuring accuracy, scaling operations, providing business insights, and supporting multilingual environments. it is a cornerstone technology for creating intelligent and responsive systems in various domains. in the banking sector, intent detection is vital for improving customer service, enhancing operational efficiency, ensuring accuracy and security, gaining data-driven insights, supporting a multilingual customer base, and maintaining regulatory compliance. it helps banks deliver superior service while managing risks and optimizing operations. this research approach considered intent detection a classification problem, aiming to assign a predetermined intent label to the user's input. while intent detection in english has been extensively studied @xcite @xcite , the task is challenging for low-resource languages such as arabic and its dialects. very little research has been conducted on intent detection in arabic banking, making creating precise and efficient systems challenging. especially in particular fields like banking, the lack of research has led to fewer datasets, tools, and models customized in arabic. as a result, this gap restricts the development of efficient nlu systems in the financial sector for arabic-speaking users. the arafinnlp shared task aims to bridge this gap. under this shared task, we developed a transformer-based system to detect intent in various arabic dialects within the banking domain. the system was evaluated on the arbanking77 dataset, which consists of 77 classes that cover different arabic dialects, including palestinian arabic and modern standard arabic (msa). the main contributions of this work are: many studies have been accomplished on textual intent detection in recent years. yolchuyeva et al. (2020) proposed self-attention networks, including san and bilstm for intent detection. they used 300-dimensional word2vec and fast-text embeddings pre-trained on english wikipedia as word representations. their model achieved an f1 score of 96.81. zhang et al. (2020b) presented transformer-based solutions for english and multilingual datasets. this model also demonstrated vital zero and few-shot performance, reaching over 75% accuracy using only 100 training examples in all datasets. a dual channel model for intent detection was proposed by wang et al. (2022). it creates a dual-channel feature extraction technique by combining the high-level features obtained from the pooling operation and the capsule network. the final intent detection is achieved by utilizing the feature fusion of the two channels. their model achieved an f1 score of 88.5% and 89.76% in atis and smp2019-ecdt datasets, respectively. mezzi et al. (2022) suggested a system that uses the concepts of intent recognition to make mental health diagnoses of arabic-speaking patients based on bidirectional encoder representations from the transformers (bert) model. their system achieved an f1 score of 94%. shams et al. (2022) fine-tuned several bert models to detect intent for urdu. their analysis shows that on their two datasets, the fine-tuned models of mbert and roberta-urdusmall reach 96.38% and 93.30% accuracy, respectively. the intent detection problem can be treated as a classification problem as well. aldjanabi et al. (2021) proposed a multi-task learning technique developed on a pre-trained arabic language model for determining offensive and hate speech. the developed model surpassed past methods for offensive and hate speech detection tasks. jarrar et al. (2023a) proposed a solution for detecting intent in modern and dialectical arabic using a neural model based on arabert. their proposed model achieved f1 scores of 0.9209 and 0.8995 on msa and palestinian dialect, respectively. considering the current constraints of the task, this work applies a transformer-based method to identify intent in various arabic dialects used in the banking sec-tor. ## dataset and task description this work used arbanking77 @xcite , which originated from the banking77 dataset @xcite . subtask-1 @xcite concerns cross-dialect intent detection in the banking domain. this subtask mainly focused on creating systems that accurately classify consumer intents from queries in multiple arabic dialects. ## system overview this work exploited a variety of ml, dl, and transformer-based models to conduct multi-dialect intent detection in the arabic banking industry. figure 1 ## results and analysis results revealed that svm with tf-idf features earned the most elevated f1-score (61.69%) among the ml approaches. on the other hand, cnn outperformed the cnn+lstm (38.50%) and cnn+bilstm (42.04%) by obtaining the highest f1-score of 43.33%, which is approximately 18.36% lower than the best ml approach (svm). dl models utilizing fasttext word embeddings also performed relatively poorly, with cnn, cnn+lstm, and cnn+bilstm achieving f1-scores of 40.52%, 39.72%, and 37.82%, respectively. however, the transformer model outperforms the top-scoring ml and dl models. arabertv2 achieved the highest f1 score of 82.08%, whereas bert-banking77 and camelbert scored 68.56% and 73.00%, respectively. while ml and dl models performed well on the development set, their performance declined on the test sets. dl models significantly underperformed compared to ml models. arabertv2 is pre-trained with wikipedia dataset, encompassing a range of arabic dialects and demonstrating superior performance over all exploited models.