A lexu NLP 24 at A ra F in NLP 2024: Multi-Dialect A rabic Intent Detection with Contrastive Learning in Banking Domain (2024)

Abstract: Arabic banking intent detection represents a challenging problem across multiple dialects. It imposes generalization difficulties due to the scarcity of Arabic language and its dialects resources compared to English. We propose a methodology that leverages contrastive training to overcome this limitation. We also augmented the data with several dialects using a translation model. Our experiments demonstrate the ability of our approach in capturing linguistic nuances across different Arabic dialects as well as accurately differentiating between banking intents across diverse linguistic landscapes. This would enhance multi-dialect banking services in the Arab world with limited Arabic language resources. Using our proposed method we achieved second place on subtask 1 leaderboard of the AraFinNLP2024 shared task with micro-F1 score of 0.8762 on the test split.

## introduction with the increase in virtual assistants' popularity, it is crucial to maintain a high level of quality and efficiency while fulfilling the objectives and serving the users. this highlights the importance of spoken and natural language understanding systems in identifying a users' goal from their utterances and provide assistance to achieve this goal. intent detection is one of the core and essential tasks of natural language understanding (nlu) @xcite . the goal is to extract syntactic and semantic information from utterances to be used in different downstream tasks such as conversation management @xcite , question answering @xcite , etc. this problem can further be classified based on the targeted semantic granularity. first, domain classification that identifies the topic the user is talking about such as banking, education, ...etc. the intent identification determines the specific outcome the user expects. at the lowest level comes the slot filling problem, where each word or span of words in the sentence is labelled based on the semantic information it provides relevant to the user's intent. intent detection and slot filling are usually tightly coupled and can be embedded jointly together especially in conversation pipelines @xcite . our research focuses on the banking domain in the arab world where users' commands need to be accurately classified into a set of pre-defined supported services. this would benefit customer services and automatic query execution, especially after the increase in online banking usage in arabicspeaking countries. while this increase is spanning across multiple countries, the need to handle different dialects has become more and more crucial. while arabic is recognized as the 4 th most used language on the internet @xcite . its dialects spanning, classical arabic (ca), modern standard arabic (msa), and dialectal arabic (da), imposes difficulties in acquiring data at scale that is enough for building robust and efficient models that understand linguistic nuances while covering several domains. in this paper, we present our submission to subtask-1 of the shared task arafinnlp2024. our main contribution lies in applying contrastive learning on a pre-trained bert-based model to enable it to capture linguistic differences and similarities. this will improve the separation between different banking intents while being robust against dialectal nuances. furthermore, we augmented the shared task's data through a secondary translation model, which will enable our model to understand dialects that are not available in the training split. ## related work in the deep learning era, different variants of cnns and rnns were heavily used in intent detection task @xcite @xcite . in 2017, the transformer architecture @xcite has emerged and quickly become the baseline of most of nlp problems. many studies used fine-tuned bert @xcite and its variations for sequence classification tasks such as sentiment analysis @xcite and intent detection @xcite . to overcome challenges presented specifically by the arabic language, multiple approaches are adapted. back-translation was used to solve the dialectal imbalance problem in many tasks such as @xcite . @xcite tries to solve the data scarcity problem by augmenting the existing data with synonymreplaced sentences or negated sentences with their appropriate labels. contrastive learning was used by @xcite to overcome the problem of overfitting when finetuning large language models on small-sized datasets. in the financial domain, finbert @xcite ) was developed by pre-training a bert model on financial text corpus. it was then used in many tasks such as stock prediction @xcite . efforts in the financial arabic domain are still lacking. this is due to the lack of large annotated datasets. hence many solutions applied in the english domain are still to be investigated and adapted for arabic domain. ## dataset we were provided with the arbanking77 dataset @xcite by the organizers of the shared task arafinnlp2024 @xcite . this dataset is a translated version of the banking77 dataset introduced in @xcite . the data includes msa and palestinian (pal) dialects only in the train and validation splits. the test split, on the other hand, contains 11,721 samples with added moroccan (mor), tunisian (tun), and saudi (sau) dialects. to overcome this issue, we augmented our training data with these three dialects by translating the msa part of the training data to these dialects using nllb (no language left behind) translation model @xcite which is good in translating low-resource languages. ## experiments we report our experiments with the macro-f1 score, that treats every class equally, over all dialects available in the validation split. we will also report the micro-f1 score on the shared task's test split. ## discussion arabic is a morphologically rich language that has a large variation of dialects. improved the performance. we can suggest using knowledge distillation to reduce model size while maintaining the high performance of the ensemble. the main limitation of this work is the dialect mismatch between the training and testing data and the absence of human annotated data. hence, our model performance would be bound to the quality of the machine translation model we used to augment the data as seen in section 5.6. ## conclusion this work investigated our work in subtask-1 of the arafinnlp shared task 2024 by applying of contrastive learning for intent classification using two distance metrics. we used marbert as a backbone encoder model for arabic dialects. then, we added a classification module for intent prediction. additionally, we explored the impact of enriching the data by translating msa training data, applying data cleaning pipeline and using different versions of the backbone model. lastly, outcomes of different model variants were combined in an ensemble model that achieved micro-f1 score on the competition's test set of 0.8762.