MA at A ra F in NLP 2024: BERT -based Ensemble for Cross-dialectal A rabic Intent Detection (2024)

Abstract: Intent detection, also called intent classification or recognition, is an NLP technique to comprehend the purpose behind user utterances. This paper focuses on Multi-dialect Arabic intent detection in banking, utilizing the ArBanking77 dataset. Our method employs an ensemble of fine-tuned BERT-based models, integrating contrastive loss for training. To enhance generalization to diverse Arabic dialects, we augment the ArBanking77 dataset, originally in Modern Standard Arabic (MSA) and Palestinian, with additional dialects such as Egyptian, Moroccan, and Saudi, among others. Our approach achieved an F1-score of 0.8771, ranking first in subtask-1 of the AraFinNLP shared task 2024.

## introduction intent detection aims at parsing the semantics of the user input to generate the best response. it is typically considered a classification task, where each utterance is associated with one, and sometimes multiple, intents. it is critical to task-oriented conversational systems in various domains, including the banking sector. while the intent detection task, has been widely investigated in the customer assistance domain @xcite , it is under-explored in the banking sector due to the limited availability of datasets, especially arabic datasets. in this paper, we present our submission to subtask-1 of shared task 1: arafinnlp2024 in the second arabic natural language processing conference @xcite . the task focuses on cross-dialectal arabic intent detection in the banking domain. the banking domain's first arabic intent detection dataset is arbanking77 @xcite , which is arabized from the banking77 * these authors contributed equally to this work. dataset @xcite . however, arbank-ing77 @xcite is limited to modern standard arabic (msa) and palestinian dialects. to help overcome this limitation, we first augment the arbanking77 dataset with additional dialects such as egyptian, moroccan, and saudi to improve the model's ability to generalize to various arabic dialects. in addition, for enhanced performance in crossdialectal intent detection, we propose an ensemble architecture of fine-tuned bert-based classification models. we utilize bert models pre-trained on dialectal arabic data, then we fine-tune these models on the augmented arbanking77, integrating contrastive loss in the training process. ## related work various previous works existed in intent detection in the banking domain in english. @xcite introduced the banking77 dataset and established baseline accuracy scores of 93.6% for full data by fine-tuning bert and using universal sentence encoders @xcite . @xcite later improved these results by correcting label errors in the dataset with confident learning and cosine similarity, achieving a 92.0% f1 score on a trimmed dataset. @xcite showed that pre-training intent representations could enhance performance in financial intent classification, attaining an 87.3% macro-f1 score on the banking77 dataset with prefix-tuning and fine-tuning of the last llm layer. while many previous works focused on english datasets, arabic has a limited number of labeled datasets, especially for dialectal and domainspecific tasks. one work that addressed the arabic intent detection problem is @xcite . they presented a model for intent detection in the mental health domain in tunisian arabic. their model classifies patient utterances into five differ-ent categories. they used bert as the encoder with five binary classifiers, one for each intent, and achieved an f1 score of 0.94. @xcite introduced the arsas dataset, comprising approximately 21,000 tweets that were manually labeled with speech-act and sentiment categories. they trained two models on the arsas dataset: a bidirectional long short-term memory (bilstm) network and a support vector machine (svm), achieving a macro f1 score of 0.615. @xcite introduced the arbanking77 dataset, significantly contributing to research in arabic intent detection in the banking domain. arbanking77 was used to fine-tune the bert-based model, achieving an f1-score of 0.9209 and 0.8995 on msa and palestinian dialects, respectively. ## data arbanking77 dataset @xcite has a total of 31,404 samples in msa and palestinian dialect, with 21,559 samples for training, and 2,464 samples for validation. each sample is a pair of the query text and its corresponding label, which is one of 77 intent classes. ## system this section presents the architecture and key components of our intent detection architecture for multi-dialect arabic intents. ## conclusion in this paper, we presented an ensemble architecture for intent classification in the banking domain, ranking first in subtask-1 of the arafinnlp shared task 2024. we also augmented the arbank-ing77 dataset with additional dialects, enhancing the model's generalizability to various dialects. additionally, we incorporated contrastive loss for enhanced text representations. future work can explore techniques to further improve generalization and classification performance.