dz F in N lp at A ra F in NLP : Improving Intent Detection in Financial Conversational Agents (2024)

Abstract: In this paper, we present our dzFinNlp team’s contribution for intent detection in financial conversational agents, as part of the AraFinNLP shared task. We experimented with various models and feature configurations, including traditional machine learning methods like LinearSVC with TF-IDF, as well as deep learning models like Long Short-Term Memory (LSTM). Additionally, we explored the use of transformer-based models for this task. Our experiments show promising results, with our best model achieving a micro F1-score of 93.02% and 67.21% on the ArBanking77 dataset, in the development and test sets, respectively.

## introduction the arabic financial nlp (arafinnlp) shared task highlights the increasing importance of advanced natural language processing (nlp) tools tailored for the financial sector in the arab world. this initiative is particularly timely given the substantial growth of middle eastern stock markets, driven by diverse sectors across the region. this economic expansion underscores the need for sophisticated financial nlp systems capable of handling the unique linguistic and cultural nuances of arabic-speaking markets @xcite . arafinnlp presents two key subtasks aimed at enhancing financial arabic nlp capabilities: subtask-1, which focuses on multi-dialect intent detection, and subtask-2, which addresses crossdialect translation and intent preservation within the banking domain @xcite . these subtasks are crucial for interpreting complex and varied banking data across different arabic dialects, which is essential for improving customer service and automating query handling in financial institutions. the dataset central to these tasks, arbanking77, is derived from the translation of the english bank-ing77 dataset @xcite into modern standard arabic (msa) and palestinian arabic. this dataset is further expanded in the shared task to include a broader array of arabic dialects. with 31,404 queries categorized into 77 intent classes, arbanking77 provides a comprehensive foundation for training and evaluating nlp models on banking-specific communications in arabic @xcite . in recent years, the field of intent detection in conversational agents has seen significant advancements. traditional machine learning methods, such as linearsvc with tf-idf @xcite , have long been employed for their simplicity and effectiveness. however, the advent of deep learning techniques, particularly long short-term memory (lstm) networks @xcite and their bidirectional variants (bilstm) @xcite , has provided more nuanced understanding by capturing the sequential nature of text. more recently, transformer-based models, like @xcite , have set new benchmarks in nlp by leveraging self-attention mechanisms to understand contextual relationships within text, making them particularly effective for complex tasks like intent detection across varied dialects. our work in this shared task explores these diverse methodologies to enhance intent detection in financial conversational agents, particularly in the context of arabic dialects. we aim to contribute to the growing body of research in arabic nlp by demonstrating how these advanced techniques can be applied to effectively interpret and manage banking-related queries, ultimately fostering greater inclusivity and efficiency in financial services for arabic-speaking communities. the remaining sections of the paper are structured as follows: section 2 reviews related work in the field, while section 3 provides an overview of the dataset used in our study. section 4 details our proposed system architecture. section 5 presents our findings and discusses their significance. finally, section 6 concludes the paper by summarizing the key takeaways and contributions. ## related work the field of arabic nlp has seen extensive research and development, particularly in the areas of text classification and intent detection. traditional approaches like tf-idf have been widely used for feature extraction in various arabic text analysis tasks. however, with the advent of deep learning, more sophisticated methods have emerged, offering improved performance and deeper insights into textual data. in the context of feature extraction for arabic text analysis, standard approaches often rely on tf-idf. this aligns with our previous work in the madar'2019 shared task @xcite , which inspired the current approach. in that work, we employed a union of tf-idf features while experimenting with different n-gram analyzers for word segmentation (word, char, char_wb). for our first experiment, we specifically focused on unweighted tf-idf features. drawing inspiration from our past work and advancements in tf-idf feature extraction and weighted fusion, we explored alternative techniques in subsequent experiments. in the second experiment, we considered the introduction of a weighted union of tf-idf features. this builds upon the foundation laid in experiment 1, incorporating weighting techniques explored in our prior research (e.g., @xcite @xcite ). for the third experiment, we explored neural network architectures by using both long short-term memory (lstm), as we did in our previous work @xcite . these models are well-suited for capturing sequential information in text, which is crucial for understanding the nuances of stance in arabic text. finally, in the fourth experiment, we explored advanced pre-trained language models. specifically, we utilized sentence transformers @xcite to generate sentence embeddings, which capture the overall meaning of a sentence. these embeddings were then fed into neural network models for stance classification. ## description of the dataset the arbanking77 dataset, provided by the arafinnlp shared task organizers, is designed to facilitate the development of nlp models for intent detection in the banking domain across various arabic dialects @xcite . this dataset is a crucial resource for advancing the capabilities of financial conversational agents tailored to arabic-speaking regions. arbanking77 originates from the translation of the english banking77 dataset @xcite into modern standard arabic (msa) and palestinian arabic. for the shared task, this dataset has been expanded to include additional arabic dialects such as gulf, levantine, and north african arabic, reflecting the linguistic diversity across the arab world. the first subtask of the arafinnlp shared task focuses on multi-dialect intent detection, aiming to classify customer intents expressed in different arabic dialects. the dataset used for this subtask includes queries in palestinian arabic (pal), among other dialects. in the arbanking77 dataset, as summarized in in this study, we opted to concentrate on the palestinian arabic (pal) subset for training and validation purposes. this decision stems from our aim to specialize our model in a single dialect, enhancing its f1-score and understanding of the specific linguistic features present in palestinian arabic queries. the pal dataset is well-sized for this purpose, providing ample data to develop a nuanced model tailored to this dialect. by focusing on pal for model training, we ensure that the model is finely tuned to the dialect's unique characteristics. for evaluation, we tested the model on the multi-dialect dataset from the arafinnlp shared task, which includes queries in modern standard arabic (msa), gulf, levantine, and north african arabic. this approach allows us to assess the model's ability to generalize and handle diverse dialects, demonstrating its adaptability and potential for broader application across various arabicspeaking regions. ## proposed system we experimented with several models and feature configurations for intent detection. for traditional machine learning, we utilized linearsvc with tf-idf vectorization. exploring deep learning, we implemented lstm models using word embeddings. additionally, we experimented with transformerbased architectures, specifically leveraging xlm-roberta to harness contextual information from pre-trained language representations. our implementations were carried out using scikit-learn for model development and training. our exploration of feature extraction techniques began with investigating the union of term frequency-inverse document frequency (tf-idf) features using scikit-learn's featureunion module @xcite . in our first experiment, we examined the effectiveness of using the raw union of these features, encompassing different n-gram levels: word, character, and character ngrams with word boundaries (char_wb). n-grams refer to sequences of n words or characters that can capture short phrases or morphological variations within the arabic language. the second experiment built upon our initial investigation by incorporating weighted tf-idf features @xcite . this approach involved experimenting with weights ranging from 0.1 to 1.0, with a step of 0.1, for the trans-former_weights parameter in featureunion. these weights were chosen to emphasize the importance of capturing character-level and word-boundaryaware features in arabic customer queries, known for their linguistic complexity and dialectal variations. these two initial experiments allowed us to systematically evaluate the impact of different tf-idf feature extraction techniques on model performance in the context of arabic intent detection within the banking domain. the results of both experiments are summarized in building upon our exploration of feature extraction techniques, the third experiment focuses on neural network architectures, specifically long short-term memory (lstm) networks. as demonstrated in our prior work @xcite , lstm models excel at capturing long-range dependencies within sequences, which is crucial for understanding the nuanced intent in arabic customer queries within the banking domain. the lstm layer is configured with 100 units, determining the dimensionality of the internal representations and the number of lstm cells in the layer. an embedding dimension of 100 is chosen for word embeddings, defining how words are represented as dense vectors. input sequences are padded to a maximum length of 100 tokens to ensure uniformity. the model utilizes the categorical cross-entropy loss function and adam optimizer during training, aimed at minimizing classification errors and optimizing training efficiency. these hyperparameter choices are fundamental to enhancing the model's capability to discern subtle nuances in arabic text, thereby improving intent detection performance. the fourth experiment focused on leveraging the power of pre-trained language models (plms). these models are trained on massive amounts of text data and learn to represent words and sentences as vectors that capture their meaning and relationships. in this experiment, we employed sentence transformers, specifically the 'xlm-r-bert-basenli-stsb-mean-tokens' model, which is adept at generating rich sentence embeddings. these embeddings condense the overall meaning of a sentence into a vector format, capturing not just individual words but also their semantic relationships. we utilized these sentence embeddings as input to a logistic regression classifier for stance classification. hyperparameters such as the default settings of the logistic regression classifier, including regularization strength, solver, and multi-class handling, were chosen to optimize model performance. by harnessing the pre-trained knowledge embedded in sentence transformers, our aim was to enhance the model's ability to discern semantic nuances within arabic text, thereby improving f1-score in the intent detection tasks. the results of the third and fourth experiments are summarized in ## results and discussion after exploring various feature extraction and neural network architectures, we now turn to evaluating the performance of our intent detection system on unseen data. we utilize the test set provided by the arafinnlp shared task (?) for this purpose. this allows us to assess how well our model generalizes to new customer queries it hasn't encountered during training. in our baseline system (see building on this success, the second experiment (@xmath3, 7, 8, and 9 in in the third experiment focused on evaluating different lstm configurations for intent detection (see the fourth experiment examined the effectiveness of pre-trained transformer models for intent detection (see ## conclusion this work explored the effectiveness of various machine learning and deep learning approaches for arabic financial nlp tasks within the arafinnlp shared task, specifically participating in subtask1. we evaluated diverse models and feature configurations, gaining valuable insights into the complexities of analyzing arabic financial text data. our findings highlight the potential of both traditional and deep learning approaches in this domain. experiment 1 demonstrated the effectiveness of support vector machines (svm) with tf-idf features, achieving a high f1-score of 93.08%. this suggests the suitability of traditional machine learning techniques for specific arabic financial nlp tasks. experiment 2 focused on deep learning models (lstms and transformers). while lstms achieved competitive f1-scores (up to 79.84%), pre-trained transformer models yielded slightly lower results (around 75.76%). further investigation is needed to understand this performance difference and explore the potential of fine-tuning transformers for this specific task. overall, these findings underscore the promise of both traditional and deep learning approaches for arabic financial nlp. future work could explore hybrid approaches that integrate the strengths of both paradigms, potentially achieving even better performance. additionally, investigating the impact of fine-tuning pre-trained transformers specifically for the arabic financial domain is crucial to unlock their full potential in this task.