B abel B ot at A ra F in NLP 2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling (2024)

Abstract: This paper presents our results for the Arabic Financial NLP (AraFinNLP) shared task at the Second Arabic Natural Language Processing Conference (ArabicNLP 2024). We participated in the first sub-task, Multi-dialect Intent Detection, which focused on cross-dialect intent detection in the banking domain. Our approach involved fine-tuning an encoder-only T5 model, generating synthetic data, and model ensembling. Additionally, we conducted an in-depth analysis of the dataset, addressing annotation errors and problematic translations. Our model was ranked third in the shared task, achieving a F1-score of 0.871.

## introduction financial natural language processing (nlp) currently plays a crucial role in the arab world. this is particularly true in the light of the considerable growth observed in the stock markets in the middle east, and the diverse sectors contributing to this expansion (zmandar et al., 2021). given this evolution, it is paramount to develop arabic nlp tools that are able to address local linguistic nuances. despite its widespread use, arabic faces a scarcity of labelled data especially when it comes to dialects and domain-specific tasks (darwish et al., 2021). this scarcity poses a challenge to most tasks focusing on arabic, but more particularly for largely unexplored tasks such as intent detection (jarrar et al., 2023). the arabic financial nlp (arafinnlp) (malaysha et al., 2024) shared task is dedicated to enhancing arabic nlp capabilities within the financial domain while also addressing the linguistic diversity of the arab world. to this end, arafinnlp 2024 introduces two subtasks: in this paper, we present our participation in sub-task 1. our contribution involves fine-tuning pre-trained language models (namely, a t5 model) for the task at hand, along with generating synthetic data using (much) large(r) language models. in section 2 we describe the arbanking77 (jarrar et al., 2023) dataset, conduct an analysis of its annotation quality and introduce the synthetic data we created to improve the performance of our model. in section 3 we present our proposed model, and describe our experiments and results in section 4. finally, we summarise our main findings, and discuss possible future work in section 5. ## system description the main idea behind our system aligns with earlier methods as it involves fine-tuning a pre-trained language model (jarrar et al., 2023). nevertheless, we opted for a different pre-trained model and augmented the training data. we fine-tuned (the encoder of) a t5-based model (raffel et al., 2020) on the combination of the arbanking77 dataset and the synthetic data introduced in section 2. the t5 architecture is an encoder-decoder transformer model (i.e. sequenceto-sequence model), but it has been previously used for regression, classification, ranking, and sentence embedding tasks (ni et al., 2022; zhuang et al., 2023; do et al., 2024). to use t5 for classification, @xcite the length-based heuristic is only to exclude selfrepetition in the output of command r+. @xcite we prompted the command r+ model to translate the same number of queries into all dialects but the final numbers reported vary across dialects due to post-processing. as we explain in section 4.1, for our final submission we trained three separate t5-based classifiers and ensembled their output. our ensembling approach is rather straightforward as it boils down to collecting the labels from three models and finding the majority label. in case of a tie, we select the label with the highest prediction score. ## experiments and results in this section, we describe the series of experiments we conducted to develop our system. these experiments aimed to assess the usefulness of synthetic data and determine the extent to which a model can generalise to unseen dialects. we fine-tuned the t5 model, by fares (2024), using five different combinations of the gold and synthetic datasets and evaluated its performance on four distinct development sets. @xcite the results of these experiments are presented in ## conclusion in this paper we presented our participation in the arafinnlp shared task on the multi-dialect intent detection. we employed a strategy involving fine-tuning an encoder-only t5 model, generating synthetic data, and model ensembling to address the challenges of cross-dialect intent detection in the banking domain. our analysis of the dataset revealed various challenges, including annotation errors and problematic translations. experimental results demonstrate the effectiveness of our approach, particularly the significant performance improvement achieved by incorporating synthetic data. our final model secured third place in the shared task with a competitive micro-f1 score of 0.8709. while our approach yielded promising results, we believe further improvement of the synthetic data quality is needed. one path forward would be to use a metric like pointwise v-information (ethayarajh et al., 2022) to filter out synthetic queries with low pvi values (i.e. queries with little relevant information to their intents). lin et al. (2023) implemented a similar approach for intent classification under few-shot settings, that yielded good results.