KSAA - RD Shared Task: A rabic Reverse Dictionary (2023)

Abstract: This paper outlines the KSAA-RD shared task, which aims to develop a Reverse Dictionary (RD) system for the Arabic language. RDs allow users to find words based on their meanings or definition. This shared task, KSAA-RD, includes two subtasks: Arabic RD and cross-lingual reverse dictionaries (CLRD). Given a definition (referred to as a “gloss”) in either Arabic or English, the teams compete to find the most similar word embeddings of their corresponding word. The winning team achieved 24.20 and 12.70 for RD and CLRD, respectively in terms of rank metric. In this paper, we describe the methods employed by the participating teams and offer an outlook for KSAA-RD.

## introduction a reverse dictionary (rd) is a type of dictionaries that allows users to find words based on their meanings or definitions. unlike a traditional dictionary, where users search for a word by its spelling, a rd allow users to enter a description of a word or a phrase, and the rd will generate a list of words that match that description. rds can be useful for writers, crossword puzzle enthusiasts, non-native language learners, and anyone looking to expand their vocabulary. specifically, rd addresses the tip-of-tongue (tot) phenomenon @xcite , which refers to the situation where a person is aware of a word they want to say but is unable to express it accurately @xcite . various approaches have been proposed in the literature to develop rds, including information retrieval (ir) system-based @xcite @xcite , graph-based @xcite @xcite , mental dictionary-based @xcite , vector space model-based semantic analysis @xcite , and neural language model-based approaches @xcite @xcite @xcite @xcite @xcite . however, to the best of our knowledge, there is no available arabic rd system that allows the user to find the best matching word for a gloss in a specific dictionary, while most of the arabic available digital dictionaries allow users to search for the definition by words @xcite . we ran this shared task as a part of the first arabic natural language processing (arabicnlp) conference collocated with emnlp 2023, featuring the ksaa-rd (king salman global academy for arabic language) with two subtasks: arabic rd (arabic to arabic) and cross-lingual reverse dictionary (clrd) (arabic to english). clrd task aims to assist translating systems in selecting the best arabic translation for new terms and definitions. the dataset for both tasks used arabic and english available dictionaries. also, we provide manually annotated mapped dictionary between arabic and english words to be used for supervised learning in the second task. ## related work various approaches have been proposed to develop rd systems, including information retrieval (ir) system-based approach, graphbased approach, mental dictionary-based approach, vector space model-based semantic analysis approach, and neural language modelbased approach. the four subsections provide a preview for each approach respectively. ## task description this section describes the two subtasks in detail: rd and clrd. the former converts arabic word definitions into arabic embeddings, while the latter converts english word definitions into arabic embeddings. ## data this section discusses the data used in the shared task. the dataset includes two main components: the dictionary data, which is presented in section 4.1, and the word embedding vectors, which is presented in section 4.2. section 4.3 describes further details of the dataset. ## evaluation the primary objective of our tasks is to find the most similar arabic embedding for an arabic or english definition. thus, we consider three different approaches to measure vector similarity. the first approach is a mean squared error (mse), which calculates the average squared difference between the generated reconstructed embedding and target embeddings. the second approach is using the cosine similarity measure, where a perfect reconstructed embedding would result in a cosine similarity of 1 with the target embeddings. the challenge with the cosine measure is that language models utilizing the transformer architecture can produce anisotropic output. hence, it is not reasonable to use it alone to anticipate that two random contextualized embeddings will be orthogonal @xcite . to complement the limitations of both mse and cosine measure, a third approach known as the ranking metric has been utilized. the ranking evaluation metrics proposed in the codwoe semeval competition @xcite . as shown in equation ( 1 ), the ranking metric is concerned with comparing and evaluating the proportion between the reconstructed embedding cosine 𝑝 ! and the target embedding cosine 𝑡 ! to the reconstruction embedding cosine 𝑝 ! with all other targets embedding 𝑡 " in the test set. the proportion of targets with a higher correlation is determined by identifying the number of cosine values greater than cos( 𝑝 ! , 𝑡 ! ) @xcite . the ranking metric can be described as: ## shared task teams & results in this section, we present our baseline model and, participating teams, and results and description of submitted systems. ## conclusion in this paper, we present the first arabic rd shared task, ksaa-rd, encompassing two subtasks: arabic rd and cross-lingual rd, clrd. the ksaa-rd task received 31 unique team registrations, resulting in 39 valid submissions and 4 submitted description papers. the outcomes from various teams underscore the persistent challenges posed by both rd and clrd tasks, emphasizing the need for continued research in the field of arabic rd tasks. our experience with ksaa-rd emphasize the significant impact of data manipulation techniques. furthermore, employing an ensemble of diverse transformer architectures proved superior to other methods, highlighting the importance of model diversity in enhancing performance. the arabic dictionary used in this shared task is limited, compared to the newly released dictionary of the arabic contemporary language: "alriyadh dictionary" (ksaa, 2023), which contains more than 120k terms compared to 58k and it is manually verified by groups of experts in the ksaa. other aspects of future work include exploring advanced embedding techniques that might be more suitable to the semantic notion of the problem. future work includes employing these techniques in a search engine and analyzing the user behavior of the search results. further investigation is needed to examine whether dictionary definitions (which are usually written in a formal style) are a good representation of the users' inquiries.