A r B anking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical A rabic (2023)

Abstract: This paper presents the ArBanking77, a large Arabic dataset for intent detection in the banking domain. Our dataset was arabized and localized from the original English Banking77 dataset, which consists of 13,083 queries to ArBanking77 dataset with 31,404 queries in both Modern Standard Arabic (MSA) and Palestinian dialect, with each query classified into one of the 77 classes (intents). Furthermore, we present a neural model, based on AraBERT, fine-tuned on ArBanking77, which achieved an F1-score of 0.9209 and 0.8995 on MSA and Palestinian dialect, respectively. We performed extensive experimentation in which we simulated low-resource settings, where the model is trained on a subset of the data and augmented with noisy queries to simulate colloquial terms, mistakes and misspellings found in real NLP systems, especially live chat queries. The data and the models are publicly available at https://sina.birzeit.edu/arbanking77.

## introduction intent detection falls under natural language understanding (nlu) and it aims at parsing the semantics of the user input in order to generate the best response. intent representation is a mapping between the user request and the actions the chatbot triggers @xcite . intent detection is typically considered a classification task, where each utterance is associated with one, and sometimes multiple, intents (figure 1 ). intent detection can be a challenging problem. the utterances during the chat are usually short, providing only a brief context to rely on when predicting the intent and the label space can be very large requiring massive data annotation. in this paper, we present an arabic intent dataset and a bidirectional encoder representations from transformers (bert) based intent detection model. the arabic corpus presented in this paper is based on the banking77, an english question-intent corpus for banking @xcite . bank-ing77 includes 13,083 queries, each query classified into one of the 77 intents. we first arabized the english banking77 by providing an msa version to each of the 13,083 queries, resulting in 15,537 msa queries (some queries have more than one msa variation). the arabization was done semi-automatically, first we used google translate and then manually verified and revised each query. second, each query was manually re-written in the palestinian dialect, resulting in 15,867 queries, which makes the data linguistically more representative from various aspects including phonology, morphology, lexicon, and syntax @xcite . the final dataset contains 31,404 queries, which was used to train a bert-based model on intent detection task. the rest of the paper is organized as follows: section 2 reviews the related work, section 3 presents the arbanking77 corpus including data arabization and localization, section 4 presents the model architecture and training, section 5 presents the results for intent detection, section 6 presents our conclusion and section 7 states limitations. ## related work arabic has a limited number of available labeled datasets, especially for dialectal and domainspecific tasks @xcite . due to data scarcity in arabic language, research on arabic intent detection is almost non-existent. others have also stated the same, where conversational machine learning systems in arabic are limited due to deficiency of datasets @xcite and arabic conversational systems are lagging behind in applying the latest technology @xcite . one of the closest work to arabic intent detection is purposed in @xcite . the authors proposed intent detection model for the mental health domain in tunisian arabic. the idea is to classify the patient utterance or concern into five aspects: depression, suicide, panic disorder, social phobia and adjustment disorder. the data set was collected by simulating a real-life psychiatric interview where a 3d human avatar plays the doctor and asks the patient questions in tunisian arabic. the patient, in return, interacts with the avatar by answering the questions vocally, then the audio is transcribed to text. the authors used bert as the encoder and added five binary classifiers, one classifier for each intent, achieving 0.94 f1 score. @xcite classified question and nonquestion utterances in chatbots. decision trees were used to perform the classification and the model was integrated into arabchat @xcite to classify utterances before processing them. @xcite published a corpus in the levantine arabic dialect consisting of 873 sentences manually tagged with one of eight acts (greetings, goodbye, thanks, confirm, negate, ask/repeat, ask for alternative, and apology). the authors tried two features including term frequency-inverse document frequency (tf-idf) and n-gram. they also experimented with multiple classifiers and they concluded that support vector machine (svm) with 2-gram features performed the best at 0.86 accuracy. @xcite introduced a speech-act recognition and sentiment dataset (arsas). about 21k tweets were collected and manually labeled with two types of classes: speech-act and sentiment. speech-act labels include expression, assertion and question, while the sentiment labels are negative, positive, neutral and mixed. @xcite trained two models on the arsas dataset, a bidirectional long-short term memory (bilstm) and svm and achieved an accuracy of 0.875 and a macro f1 score of 0.615. @xcite proposed a contrastive based learning for out-of-domain data and tested the performance on multiple datasets including the banking data @xcite and they demonstrated improvement of the out-of-domain data without sacrificing performance on in-domain-data. other related languages for which intent detection was studied is urdu. in @xcite , the authors translated the air travel information system (atis) @xcite and aol datasets from english to urdu and performed intent detection using a combination of cnns, lstms and bilstms models. for atis, cnn performed the best at 0.924 accuracy, while for aol, bilstm achieved the highest performance at 0.831 accuracy. in later work the authors improved the accuracy to reach 0.9112 @xcite . atis was also used for intent detection in the indonesian language @xcite and the authors reported an accuracy of 0.9584 using a cnn-based model. @xcite utilized snips @xcite and atis to train a meta-learning approach with contrastive learning for intent detection and slot-filling. snips dataset covers multiple domains including restaurants, books, weather and music, making it more challenging than atis. the data is collected using snips personal assistant and contains 16k queries labeled with 7 intents. the reader may have already noticed that we could not find relevant work related to arabic intent detection recognition or any related work on labeled arabic intent datasets. in this paper, we attempt to address these two issues, arabic intent corpus and intent recognition. we present the ar-banking77, an arabic intent dataset, which was arabized and localized from the banking77 english dataset @xcite . arbanking77 was also augmented with thousands of additional msa and palestinian dialect queries, resulting in a final dataset of 31,404 queries and 77 intents. arbank-ing77 was used to fine-tune bert-based model, achieving an f1-score of 0.9209 and 0.8995 on msa and palestinian dialect, respectively. when deploying a fine-tuned intent detection model inside a chatbot system, other modules might be needed to better understand user queries, such as spell corrections @xcite , named entity recognition @xcite , word-sense disambiguation @xcite , synonymy expanding @xcite . ## the arbanking77 corpus the arbanking77 corpus is derived from the bank-ing77 dataset @xcite ) that consists of 13,083 queries and 77 classes @xcite and that is open under the (cc-by-4.0) license. banking77 was designed to focus on a fine-grained single domain, banking. each query is labeled with one of the 77 classes. example intents from the dataset include card arrival, personal identification number (pin) blocked, card linking, exchange rate and age limit. the number of queries per class ranges between 75 to 227, with an average of 170 queries per intent. the original banking77 dataset is divided into train and test dataset, their statistics are presented in ## intent detection model we fine-tuned a bert-based model on an intent detection task using the arbanking77 dataset. in this section, we will go over the model details. ## experiments and results we ran multiple experiments with different models and data configurations. in section 5.1, we evaluate zero-shot learning, section 5.2 benchmarks multiple pre-trained transformer models on arabic data, section 5.3 simulates low-resource settings and section 5.4 simulates different spelling errors that are commonly found in the arabic language. we report the model performance on the test set using macro f1, precision and recall scores. when training the models on the full dataset, we used the train, validation and test split listed in ## conclusion in this paper, we presented the arbanking77 dataset, consisting of queries in both msa and palestinian dialects in the banking domain. as far as we know, arbanking77 is the first arabic intent detection dataset in the banking domain. the dataset contains 31,404 queries and 77 intents. the data was then used to fine-tune a bert-based model for the intent detection task, resulting in an f1-score of 0.9209 for msa and 0.8995 for pal. we also simulated low-resource settings and found that the model is robust and with only 20% of the data, model performance on pal and msa dropped by only 6.32% and 4.51%, respectively. we noted that training data augmentation does not negatively affect the model performance on the clean msa and pal test sets. in fact, at the lowest resource settings (20%) the augmented model out-performed the non-augmented model on both msa and pal. we performed additional data augmentation to simulate errors, misspellings, and other mistakes that are common in real nlp systems. we observed the accuracy on pal-sim s suffers greatly when the model is trained on 20% of the non-augmented data. augmenting the training data closes the performance gap on pal-sim s by about 5%. this indicates that bert is susceptible to some errors, especially in dialectal arabic which has less consistent orthography than msa. it is also noticeable that the relative drop in accuracy between the 20% and 50% training sets is much larger than 50% and 100% case. this implies that the negative effect of the introduced errors in the dialectical arabic is inversely proportional to the amount of data used in the train set. finally, based on the low performance using zero-shot learning on msa and pal and a slight lexical overlap between them, we concluded that there is an urgent need to annotate msa and dialectical arabic.