A ra F in NLP 2024: The First A rabic Financial NLP Shared Task (2024)

Abstract: The expanding financial markets of the Arab world require sophisticated Arabic NLP tools. To address this need within the banking domain, the Arabic Financial NLP (AraFinNLP) shared task proposes two subtasks: (i) Multi-dialect Intent Detection and (ii) Cross-dialect Translation and Intent Preservation. This shared task uses the updated ArBanking77 dataset, which includes about 39k parallel queries in MSA and four dialects. Each query is labeled with one or more of a common 77 intents in the banking domain. These resources aim to foster the development of robust financial Arabic NLP, particularly in the areas of machine translation and banking chat-bots.A total of 45 unique teams registered for this shared task, with 11 of them actively participated in the test phase. Specifically, 11 teams participated in Subtask 1, while only 1 team participated in Subtask 2. The winning team of Subtask 1 achieved F1 score of 0.8773, and the only team submitted in Subtask 2 achieved a 1.667 BLEU score.

## introduction financial natural language processing (finnlp) is revolutionising the financial sector, offering unmatched potential to enhance decision-making, manage risks, and drive operational efficiency @xcite . by leveraging advanced linguistic analysis @xcite and nlp algorithms @xcite , finnlp optimises processes and streamlines workflows, delivering a myriad of benefits @xcite . finnlp enables the extraction of key information, including events @xcite , relationships @xcite , and named entities @xcite , from diverse sources such as financial reports, news articles, invoices, and social media posts. practical applications of financial nlp include textual analysis in accounting and finance @xcite figure 1 : arabanking2024 datasets for intent detection 2019, 2020), analysis of financial transactions (jørgensen and igel, 2021), customer complaints @xcite , and text classification @xcite . nevertheless, as recently highlighted by @xcite ; el-haj et al. @xcite , the majority of financial nlp research is conducted in english. on the other hand, the arab world's financial landscape is experiencing robust growth, attracting global attention and investment across diverse sectors. this expansion underscores the critical role of financial nlp in understanding and interpreting the intricacies of arabic financial communications. as highlighted by @xcite , the dynamic nature of middle eastern stock markets reflects the region's evolving financial environment, necessitating advanced nlp tools tailored to local linguistic nuances. in this paper, we provide an overview of the arafinnlp-2024 shared task 1 , which represents a significant step forward in advancing the development of arabic nlp capabilities within the finance domain. we propose two subtasks aimed at addressing key challenges in the banking sector: (i) multi-dialect intent detection and (ii) crossdialect translation and intent preservation. for this shared task, we provided participants access to the arbanking77 foot_1 corpus @xcite , for training purposes, which contains about 31k queries in modern standard arabic (msa) and palestinian dialect, each labeled by at least one of the 77 intents in the banking domain. the data excluded any examples covering the moroccan, tunisian and saudi dialects. we discuss the test data preparation in section 4. the two subtasks are designed to tackle the complexities inherent in interpreting and managing diverse banking data prevalent in arabic-speaking regions, catering to the linguistic diversity across the arab world @xcite . by focusing on intent detection and dialectical translation, arafinnlp aims to enhance customer service, automate query handling, and facilitate seamless communication across various arabic dialects, thus fostering inclusivity and efficiency in financial services. the rest of the paper is organized as: section 2 presents the related work; section 3 describes the tasks; section 4 presents the dataset; section 5 reports on the performance of the participating systems; and, section 6 concludes. ## related work in the rapidly evolving field of nlp, integrating financial data has become a significant area of research, particularly in the context of multilingual and dialectal variations. recent studies in the financial nlp (finnlp) domain have investigated machine translation between msa and dialectal arabic @xcite . noteworthy contributions to finnlp have been made by @xcite , who examined the application of nlp techniques to analyse arabic financial texts. additionally, jarrar et al. (2023c) introduced the arbanking77 dataset, derived from the english banking77 dataset @xcite , which has been pivotal in advancing research in this area. the related works can be divided into four main categories: financial document extraction, lexical diversity and sentiment analysis, cross-lingual and dialectal studies, and related machine learning techniques. within the realm of finnlp, the process of financial document extraction emerges as a critical component in the financial industry's digital transformation. this intricate process entails the automated retrieval and meticulous analysis of pertinent data from a diverse array of financial documents, including but not limited to invoices, receipts, and financial statements. by harnessing cutting-edge methodologies such as semantic role-labeling schemes and integrating deep nlp techniques with knowledge graphs @xcite , researchers have significantly elevated the accuracy and efficiency of financial narrative processing @xcite @xcite . these advanced techniques not only streamline data extraction processes but also enable deeper insights into complex financial structures and trends, thereby empowering decision-makers with actionable intelligence to drive strategic initiatives, mitigate risks, and seize opportunities in a rapidly evolving financial landscape. through the seamless integration of technology and financial expertise, finnlp continues to revolutionise traditional workflows, facilitating agile decision-making and fostering innovation across the financial sector. in addition to document extraction, finnlp extensively explores the realms of lexical diversity and sentiment analysis. the assessment of lexical diversity delves into the richness and variety of vocabulary employed within financial texts. concurrently, sentiment analysis endeavours to decipher the emotional nuances embedded in financial communications. for instance, researchers have scrutinised sentiment patterns in arabic financial tweets, shedding light on the linguistic intricacies prevalent in such contexts @xcite . similarly, investigations into expressions of trust and doubt within financial reports underscore the emotional dimensions inherent in financial discourse @xcite . by elucidating these linguistic subtleties, finnlp not only enhances our understanding of financial communications, but also empowers stakeholders to navigate and interpret complex data with confidence and clarity. although msa and arabic dialects have been studied, mainly from morphological (and semantic) annotations and dialect identification perspectives @xcite @xcite , less attention was given to intent detection. in terms of cross-lingual and dialectal studies, important works examine how languages or dialects interact and translate across cultural and linguistic boundaries, focusing on understanding and processing variations in language usage and meaning. @xcite addresses the challenges of cross-lingual financial communications by focusing on analyzing and annotating english-chinese financial terms. the proposal of a multilingual financial narrative processing system by @xcite highlights the necessity for tools that function across diverse languages. @xcite used forward-looking sentence detection techniques to anticipate future statements in financial texts. these efforts highlight the varied methodologies in financial nlp, from structural data extraction to predictive analytics, while noting a gap in addressing arabic linguistic nuances. @xcite unveiled a transformerbased architecture to detect causality in financial texts, and (tsutsumi and utsuro, 2022) employed a bert-based model to discern stock price triggers from news, enhancing the understanding of nuanced financial changes. these studies advance automatic text processing, although they primarily focus on english datasets and traditional nlp tasks. another study by @xcite focused on the french financial sector, developing cofif plus, a corpus for summarizing french financial narratives, highlighting the need for tools that cater to non-english financial data. in the last year, nlp has seen significant advances in the understanding and generation of financial narratives across various languages @xcite introduced finarat5, a text-to-text model tailored for arabic financial text understanding and generation, emphasising the importance of language-specific models in financial contexts. while exploring the frontier of finnlp, it is essential to acknowledge the broader landscape of shared tasks and initiatives aimed at understanding arabic msa and dialects. these include notable endeavours such as arabicnlu for wordsense disambiguation @xcite , nadi for dialect identification @xcite , and wojoodner for named entity recognition @xcite ). notably, these tasks extend their focus to entities within the finance and banking domains @xcite , aligning closely with the objectives of finnlp. through collaborative efforts and interdisciplinary research, these shared tasks complement the advancements in finnlp, fostering a holistic understanding of linguistic nuances and enhancing the applicability of nlp techniques within financial contexts. as finnlp continues to evolve, leveraging insights from these shared tasks can further enrich its capabilities, ultimately driving innovation and efficacy in financial analysis, decision-making, and risk management. ## task description the arafinnlp shared task comprises two subtasks aimed at advancing financial arabic nlp: subtask 1 focuses on multi-dialect intent detection, while subtask 2 addresses cross-dialect translation and intent preservation within the banking domain. nonetheless, arafinnlp is the first shared task in arabic financial nlp, as well as the first arabic multi and cross dialects, where banking intents predicted and preserved across four dialects translated from msa and english. in this section we break into details of each sub-task. ## dataset the shared task provided a multi and cross dialectal arabic banking-related dataset, which consists of queries translated into msa and various arabic dialects, including palestinian, saudi, tunisian, and moroccan (figure 1 ). the queries are classified into intent classes selected from 77 intents, where each query could match to one or more intent. the initial data was obtained from arbanking77 dataset @xcite , available in msa and palestinian dialects. for this shared task, we augmented the arbanking77 dataset with three additional di-alects. we cooperated with teams of linguists each specialized in each dialect. moroccan dialect: the team first translated the original sentences from english and msa to moroccan darija using gpt-4 and meta's seamless-m4t-v2 models @xcite . a team of seven native moroccan darija speakers from various regions manually reviewed and corrected the translations for accuracy. they divided the dataset into seven parts, each checked by one annotator. the annotators were asked to refer to both msa and english queries when the translation was ambiguous. each translated sentence was carefully reviewed, resulting in approximately 67% of them being edited. lastly, two additional annotators conducted a final review of all the data to ensure consistency. saudi and tunisian dialects: one specialized linguist in each of these two dialects worked on translating data from msa to the saudi and tunisian dialects. as they are native speakers of these dialects, they ensured the translations captured all the linguistic nuances of the najdi dialect for saudi and local terms for tunisian, preserving the true meaning of the text. during the process, they carefully double-checked the translation to address any typos or ambiguity. ## results and discussion the arafinnlp shared subtasks attracted a good number of teams, with a diverse array of participants employing various techniques and methodologies tailored to the challenges of multi-dialectal intent detection and cross-dialect translation in the banking domain. leveraging a range of nlp models and approaches, including fine-tuning pretrained transformers and utilising deep learning models, participating teams navigated the complexities of arabic dialects and the nuances inherent in financial communications @xcite @xcite @xcite @xcite . with queries in msa and several arabic dialects, extractive techniques were predominantly employed. the arafinnlp shared task results ( the evaluation metrics for each subtask offer a comprehensive overview of system performance across various dialects and tasks, facilitating comparisons and informing future research directions in arabic nlp, particularly within the finance domain. ## conclusions and future work in conclusion, the arafinnlp shared task represents a significant effort in advancing financial arabic natural language processing by addressing critical challenges in multi-dialect intent detection and cross-dialect translation within the banking domain. the participation of diverse teams employing various methodologies has yielded valuable insights into the effectiveness of different approaches. these findings can inform future research, guiding the development of more robust and accurate nlp models for handling the complexities of arabic dialects in financial contexts. the results highlight the impressive performance of the ma team, whose use of fine-tuned bertbased models and contrastive loss for training, coupled with data augmentation techniques, proved particularly effective. this underscores the importance of both sophisticated model architectures and enriched training datasets in achieving high performance in nlp tasks involving diverse languages. looking ahead, future work in financial arabic nlp could explore several avenues for further improvement and innovation. enhancing the performance and adaptability of nlp models across a wider range of arabic dialects could lead to more inclusive and effective solutions for financial communication. additionally, the development of specialized resources and datasets tailored to specific financial domains and dialectal variations could facilitate more targeted and accurate analysis of financial texts. furthermore, ongoing advancements in nlp technologies, such as the integration of multi-modal inputs and the incorporation of domain-specific knowledge, offer promising oppor-tunities for improving the capabilities of financial arabic nlp systems. the arafinnlp shared task serves as a catalyst for advancing research and development in financial arabic nlp, paving the way for more sophisticated and versatile systems capable of addressing the diverse linguistic and communicative needs of arabic-speaking communities in the financial domain. by fostering collaboration and innovation, the shared task contributes to the broader goal of enhancing accessibility, efficiency, and inclusivity in financial services through the application of natural language processing technologies.