Abed at KSAA - RD Shared Task: Enhancing A rabic Word Embedding with Modified BERT Multilingual (2023)

Abstract: This paper presents a novel approach to the Arabic Reverse Dictionary Shared Task at WANLP 2023 by leveraging the BERT Multilingual model and introducing modifications augmentation and using a multi attention head. The proposed method aims to enhance the performance of the model in understanding and generating word embeddings for Arabic definitions, both in monolingual and cross-lingual contexts. It achieved good results compared to benchmark and other models in the shared task 1 and 2.

## introduction the arabic reverse dictionary shared task at ara-bicnlp 2023 poses unique challenges in generating word embeddings from definitions, especially in a cross-lingual setting. while traditional models have shown promise, the complexity of the arabic language and its rich morphological structure necessitates advanced techniques. this paper introduces a modified bert multilingual model, incorporating changes augmentation and using a multi attention head, to address these challenges. this paper describes the system used for the arabic reverse dictionary shared task at arabicnlp 2023. the task was released for the the first based on the semeval 2022 shared task #1: comparing dictionaries and word embeddings (codwoe) @xcite but for the arabic language. competition results highlight two main trends: participants identified challenges such as subpar data quality, small training corpora, and mainstream natural language generation (nlg) metrics' limited relevance. teams have experimented with transformer, recurrent neural networks (rnn), and convolutional neural networks (cnn) models and found success with multi-task training. there's no single architecture that stands out as the best, with some evidence suggesting that transformers may not be ideal for this task @xcite . for future research, the focus should be on enhancing dataset size and quality and re-evaluating metrics. the competition has spotlighted a variety of natural language processing (nlp) models and approaches, underscoring the field's dynamic nature. for the arabic reverse dictionary shared task at arabicnlp 2023, our primary objective was to assess the competitiveness of our current bert multilingual cased implementation in the context of comparing dictionaries and embeddings. additionally, we endeavored to incorporate a data augmentation strategy to enhance our results. remarkably, our experiments with data augmentation yielded significant improvements in the development set results. the augmentation techniques employed were relatively basic, involving operations such as word addition, deletion, and swapping within sentences. due to the limited size of the available data, even with augmentation, our training was restricted to just two epochs. despite these constraints, our approach demonstrated competitive outcomes. ## methods this section explains the on data augmentation and model architecture part of the paper. the data augmentation section talks about using this method in nlp to make the model stronger and more adaptable by exposing it to a wider variety of language. different text changing techniques like swapping synonyms, adding or removing words, and switching word order are used to make the training data more varied, which helps the model perform better. the model subsection explains the design and training steps, the usage of bert multilingual model, andd highlights key parts like multihead attention, a linear layer, and the choice of loss function and optimizer. this structured approach reflects a systematic endeavor to enhance model performance and adaptability in handling text regression tasks across varying linguistic scenarios. ## results to validate the effectiveness of our proposed modifications, we conducted experiments on the provided dataset for the shared task. ## discussion and conclusion this paper presented a novel approach to the arabic reverse dictionary shared task using a modified bert multilingual model. the introduced modifications augmentation and using a multi attention head, have shown promise in enhancing the model's performance, paving the way for future research in this domain. our experiments demonstrate the potential of the bert multilingual model, even simple modifications such as data augmentation and using a multi attention head still provides good results but the improvements in sgns embeddings is less impressive.