KSAA - CAD Shared Task: Contemporary A rabic Dictionary for Reverse Dictionary and Word Sense Disambiguation (2024)

Abstract: This paper outlines the KSAA-CAD shared task, highlighting the Contemporary Arabic Language Dictionary within the scenario of developing a Reverse Dictionary (RD) system and enhancing Word Sense Disambiguation (WSD) capabilities. The first KSAA-RD (Al-Matham et al., 2023) highlighted significant gaps in the domain of RDs, which are designed to retrieve words by their meanings or definitions. This shared task comprises two tasks: RD and WSD. The RD task focuses on identifying word embeddings that most accurately match a given definition, termed a “gloss,” in Arabic. Conversely, the WSD task involves determining the specific meaning of a word in context, particularly when the word has multiple meanings. The winning team achieved the highest-ranking score of 0.0644 in RD using Electra embeddings. In this paper, we describe the methods employed by the participating teams and provide insights into the future direction of KSAA-CAD.

## introduction the ksaa-cad is considered an extension of the first ksaa-rd iteration @xcite . these two shared tasks utilized contemporary arabic language dictionaries (cad). while the first iteration was primarily focused on the reverse dictionary (rd) with two subtasks, arabic rd and cross-lingual rd (clrd), the current iteration extends its scope to include word sense disambiguation (wsd) task. the second rd task includes multiple definition from different 1 https://github.com/ksaa-nlp/ksaa-cad dictionaries, while the first one includes definitions from only one dictionary. rd enables users to find words by describing their meanings or definitions, rather than searching by exact word as in traditional dictionaries. users input a description or phrase, and the rd provides a list of corresponding words. on the other hand, the objective of the wsd task is to assist systems in determining the most appropriate sense of new terms and definitions, emphasizing the importance of context in understanding the intended sense. this shared task is conducted as a part of the second arabic natural language processing (arabicnlp) conference, held in conjunction with acl 2024. it featured the ksaa-cad (king salman global academy for arabic language). four research papers were submitted for the ksaa-cad shared task. the datasets and codes developed for the ksaa-cad shared task are publicly accessible in a github repository 1 , contributing to ongoing arabic nlp research efforts. this paper is structured into seven sections. section 2 delves into related work, section 3 presents the data for the shared task, and section 4 defines the ksaa-cad tasks. section 5 elucidates the performance evaluation metric. finally, section 6 discusses the baseline model and the results achieved by the participating teams. ## dataset this section examines the data employed in ksaa-cad for both rd and wsd tasks. ## task description this section provides a detailed overview of the two tasks: rd and wsd. the rd task aims to transform the arabic word definition or gloss into embeddings. meanwhile, the wsd task focuses on determining the intended meaning or sense of a word in the context of a given sentence or segment. ## evaluation the main objective of the rd task lies in the reconstruction of word embeddings. subsequent to the first ksaa-rd task, our methodology incorporated three approaches for evaluating vector similarity. these include the mean squared error (mse), cosine similarity measurement, and "id":"ar.78", "word":" ‫,"عين‬ "gloss":" ‫الماء‬ ‫نبع‬ ... ", "pos":"n", "electra":[0.7, 0.3, …], "arabert":[0.2, 0.8, …], "camelbert":[1.3, 0.5, …], } { "context_id":"context.301", "context":" ‫اللغة‬ ‫برمجان‬ ‫يأتي‬ ‫,"...العربية‬ "word": ‫,"اللغة"‬ "gloss_id":"gloss.205", "lemma_id":"ar.301", } { "lemma_id": "ar.301", "gloss_id":"gloss.205", "gloss":" ‫المشاعر‬ ‫لتبادل‬ ‫وسيلة‬ ُّ ‫ل‬ ُ ‫ك‬ ‫كاإلشارات‬ ‫واألفكار‬ ..." } { "lemma_id": "ar.301", "gloss_id":"gloss.211", "gloss":" ‫غة‬ ُّ ‫الل‬ ‫بدراسة‬ ‫تص‬ ْ ‫يخ‬ ‫علم‬ ‫إطار‬ ‫في‬ ‫َّة‬ ‫منهجي‬ ‫"...دراسة‬ } ranking evaluation metrics, as proposed in the codwoe semeval competition @xcite . the ranking metric can be described as follows: on the other hand, accuracy and mrr@2 (mean reciprocal rank at 2) are the primary metrics employed in the wsd task. these metrics are crucial as they provide a measure of how accurately the system identifies the correct sense of a word within its context. accuracy represents the proportion of correct predictions made by the model, while mrr@2 measures the average rank position of the correct answer, rewarding models that place the correct answer higher in the ranked list. ## shared task teams & results this section introduces baseline models, outline the participating teams, and provide an overview of the submitted systems along with their results. ## conclusion in this paper, we introduced the ksaa-cad shared task, a thorough effort focused on progressing rd systems and wsd in arabic. this work expands the initial version of ksaa-rd by incorporating new datasets and integrating modern embedding models such as electra, arabertv2, and camelbert-msa. notably, for all participant teams, electra achieved superior results compared to other embeddings using the same model. on the other hand, bertseg surpassed others in terms of cosine similarity performance. nevertheless, there is still potential for enhancement, particularly in utilizing other approaches such as llm and prompt engineering @xcite . in conclusion, the rd baseline surpasses all participant models in terms of rank. this achievement showcases how effectively contextualized word embeddings in handling arabic rd tasks. additionally, it clarifies that the arabic rd problem requires further investigation. asos team has conducted an analysis of the datasets that can be used in the future to aid in rd model training. in the wsd task, gloss-based methods showed potential in the baseline result but were not as effective as rd because of the complexity of arabic word meanings and the lack of annotated datasets. it is crucial to emphasize the team's low performance in the wsd task, which was 64% below the baseline. this highlights significant obstacles in the current methods of wsd in arabic, emphasizing the need for more comprehensive datasets, as well as more advanced models, to effectively address the complexities of arabic word meanings. the involvement and outcomes of different teams offered valuable perspectives on the techniques that can be used for future progress. groups that employed transformer-based models and creative data augmentation methods, like incorporating wikipedia embeddings and utilizing semi-encoder architectures, showcased significant enhancements in performance.